SR-IOV-Passthrough-For-Networking

=SR-IOV Networking in OpenStack= OpenStack Juno adds inbox support to request VM access to virtual network via SR-IOV NIC. With the introduction of SR-IOV based NICs, the traditional virtual bridge is no longer required. Each SR-IOV port is associated with a virtual function (VF). SR-IOV ports may be provided by Hardware-based Virtual Ethernet Bridging (HW VEB); or they may be extended to an upstream physical switch (IEEE 802.1br). There are two ways that SR-IOV port may be connected:
 * directly connected to its VF
 * connected with a macvtap device that resides on the host, which is then connected to the corresponding VF

Nova
Nova support for SR-IOV enables scheduling an instance with SR-IOV ports based on their network connectivity. The neutron ports' associated physical networks have to be considered in making the scheduling decision. The PCI Whitelist - which is specified on every compute node that has PCI passthrough devices - has been enhanced to allow tags to be associated with PCI devices. PCI devices available for SR-IOV networking should be tagged with physical_network label.

For SR-IOV networking, a pre-defined tag "physical_network" is used to define the physical network to which the devices are attached. A whitelist entry is defined as: ["vendor_id": "",] ["product_id": "",] ["address": "[[ ]:] ]:][ ][.[ " |   "devname": "Ethernet Interface Name (PF)",]      "physical_network":"name string of the physical network"

 can be an asterisk (*) or a valid vendor/product ID as displayed by the Linux utility lspci. The address uses the same syntax as in lspci. The devname can be a valid PCI device name. The only device names that are supported are those displayed by the Linux utility ifconfig -a and correspond to PF.

If the device defined by the address or devname corresponds to a SR-IOV PF, all VFs under the PF will match the entry.

Note that the 'product_id' of the VFs of a NIC is different to that of the PFs of the NIC - make sure you use the VF ID if you use this method of filtering your devices.

Multiple whitelist entries per host are supported.

Neutron
Neutron support for SR-IOV requires ML2 Plugin with SR-IOV supporting mechanism driver. Currently there is ML2 Mechanism Driver for SR-IOV capable NIC based switching (HW VEB). There are network adapters from different vendors that vary by supporting various functionality. If VF link state update is supported by vendor network adapter, the SR-IOV NIC L2 agent should be deployed to leverage this functionality. Note that in the nova boot API, users can specify either a port-ID or a net-ID. If a net-ID is specified, it is assumed that the user is requesting a normal virtual port (which is not an SR-IOV port).

=SR-IOV Configuration=

Nova Scheduler
Enabling PciPassthroughFilter modify /etc/nova/nova.conf scheduler_available_filters = nova.scheduler.filters.all_filters scheduler_default_filters = RetryFilter, AvailabilityZoneFilter, RamFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, PciPassthroughFilter

Neutron Server
Using ML2 Neutron plugin modify /etc/neutron/plugins/ml2/ml2_conf.ini:

[ml2] tenant_network_types = vlan type_drivers = vlan mechanism_drivers = openvswitch,sriovnicswitch [ml2_type_vlan] network_vlan_ranges = physnet1:2:100

Add supported PCI vendor VF devices, defined by vendor_id:product_id according to the PCI ID Repository in the /etc/neutron/plugins/ml2/ml2_conf_sriov.ini:

[ml2_sriov] supported_pci_vendor_devs = vendor_id:product_id

Example for Intel NIC that supports SR-IOV: supported_pci_vendor_devs = 8086:10ca

If SR-IOV network adapters support VF link state setting and admin state management is desired, make sure to add /etc/neutron/plugins/ml2/ml2_conf_sriov.ini    [ml2_sriov] section the following setting:

agent_required = True

Note:Usually the agent_required should be True unless you have Intel NIC which doesn't support admin state change.

Neutron server should be run with the two configuration files /etc/neutron/plugins/ml2/ml2_conf.ini and /etc/neutron/plugins/ml2/ml2_conf_sriov.ini neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini

enable SR-IOV on network adapter
By default SRIOV is disabled, you will have to enable SRIOV and create the VFs on each compute host that should support SRIOV functionality. Currently specific Intel and Mellanox cards are known to support SRIOV. Below resources contain information on how to enable and create the VFs:
 * Mellanox SR-IOV driver support
 * Intel SR-IOV driver support

nova-compute
Nova-compute needs to know which PCI devices are allowed to be passed through to the VMs. Also for SRIOV PCI devices it needs to know to which physical network the VF belongs. This is done through the pci_passthrough_whitelist parameter under the default section in /etc/nova/nova.conf. For example if we want to whitelist and tag the VFs by their PCI address we would use the following setting: pci_passthrough_whitelist = {"address":"*:0a:00.*","physical_network":"physnet1"} This associates any VF with address that includes ':0a:00.' in its address to the physical network physnet1.

Alternatively we can also use the interface name and say: "All VFs created from eth2 are allowed to be passed through and belong to physical network physnet1" in this case eth2 is the PF. For example: pci_passthrough_whitelist = {"devname": "eth2", "physical_network": "physnet1"}

After configuring the whitelist you have to restart nova-compute service.

When using devstack pci_passthrough_whitelist can be configured in local.conf file, for example: $NOVA_CONF [DEFAULT] pci_passthrough_whitelist = {"'"address"'":"'"*:02:00.*"'","'"physical_network"'":"'"physnet1"'"}

SR-IOV neutron agent
Only if the hardware supports it and you want to enable changing the port admin_state, you have to run the Neutron SR-IOV agent. Otherwise it's suggested to use agent_required = False.

Note:If you configured agent_required=True on the Neutron server, you must run the Agent on each compute node.

Modify /etc/neutron/plugins/ml2/ml2_conf_sriov.ini as follows: [securitygroup] firewall_driver = neutron.agent.firewall.NoopFirewallDriver [sriov_nic] physical_device_mappings = physnet1:eth1 exclude_devices =

Where:
 * physnet1 is the physical network
 * eth1 is the physical function (PF)
 * exclude_devices is empty so all the VFs associated with eth1 may be configured by the agent

Note:SR-IOV agent only work with NoopFirewallDriver when Security Groups are enabled, but you can still use other firewall_driver for other Agents by updating their conf with the requested firewall driver.

After modifying the configuration file, start the Neutron SR-IOV agent. Run: neutron-sriov-nic-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini

Exclude VFs
If you want to exclude some of the VFs so the agent does not configure them, you need to list them in the sriov_nic section:

Example: exclude_devices = eth1:0000:07:00.2; 0000:07:00.3, eth2:0000:05:00.1; 0000:05:00.2

FDB L2 Agent Extension
The FDB population is an L2 agent extension to OVS agent or Linux bridge. Its objective is to update the FDB table for existing instance using normal port, thus enabling communication between SR-IOV instances and normal instances. The use cases of the FDB population extension are:

1. Direct port and normal port instances reside on the same compute node.

2. Direct port instance using floating IP and network node are located on the same host. Additional information describing the problem can be found here: http://events.linuxfoundation.org/sites/events/files/slides/LinuxConJapan2014_makita_0.pdf

Note: This feature is supported from newton release.

To enable this extension edit the relevant l2 agent config file (ovs_agent.ini/linuxbridge_agent.ini)

•  in section [agent] add to extensions fdb: [agent] extensions = fdb •   add the FDB section and shared_physical_device_mappings parameter. This parameter maps each physical port to its physical network name, each physical network can be mapped to several ports: [FDB] shared_physical_device_mappings = physnet1:p1p1, physnet1:p1p2

Configure SR-IOV and PV on the same compute node
In case SR-IOV and PV are created on the same host the FDB L2 agent extension should be loaded to OVS agent or Linux bridge agent to enable communication between the two instances.

For more information on loading the FDB extension see the previous "FDB L2 Agent Extension" section.

VM creation flow with SR-IOV vNIC
neutron port-create  --binding:vnic-type 
 * Create one or more neutron ports. Run:

nova boot --flavor m1.large --image --nic port-id= --nic port-id=
 * Boot VM with one or more neutron ports. Run: