OpenStack - User contributions [en]

SR-IOV-Passthrough-For-Networking

2014-10-29T10:09:53Z

Itzikb: /* VM creation flow with SR-IOV vNIC */

=SR-IOV Networking in OpenStack Juno=
OpenStack Juno adds inbox support to request VM access to virtual network via SR-IOV NIC. With the introduction of SR-IOV based NICs, the traditional virtual bridge is no longer required. Each SR-IOV port is associated with a virtual function (VF). SR-IOV ports may be provided by Hardware-based Virtual Ethernet Bridging (HW VEB); or they may be extended to an upstream physical switch (IEEE 802.1br).
There are two ways that SR-IOV port may be connected:
* directly connected to its VF
* connected with a macvtap device that resides on the host, which is then connected to the corresponding VF

==Nova==
Nova support for SR-IOV enables scheduling an instance with SR-IOV ports based on their network connectivity. The neutron ports' associated physical networks have to be considered in making the scheduling decision.
PCI Whitelist has been enchanced to allow tags to be associated with PCI devices. PCI devices available for SR-IOV networking should be tagged with physical_network label.

For SR-IOV networking, a pre-defined tag "physical_network" is used to define the physical network to which the devices are attached. A whitelist entry is defined as:
["vendor_id": "<id>",] ["product_id": "<id>",]
["address": "[[[[<domain>]:]<bus>]:][<slot>][.[<function>]]" |
"devname": "Ethernet Interface Name",]
"physical_network":"name string of the physical network"

<id> can be an asterisk (*) or a valid vendor/product ID as displayed by the Linux utility lspci. The address uses the same syntax as in lspci. The devname can be a valid PCI device name. The only device names that are supported are those displayed by the Linux utility ifconfig -a and correspond to either a PF or a VF on a vNIC.

If the device defined by the address or devname corresponds to a SR-IOV PF, all VFs under the PF will match the entry.

Multiple whitelist entries per host are supported.

==Neutron==
Neutron support for SR-IOV requires ML2 Plugin with SR-IOV supporting mechanism driver.
Currently there is ML2 Mechanism Driver for SR-IOV capable NIC based switching (HW VEB).
There are network adapters from different vendors that vary by supporting various functionality.
If VF link state update is supported by vendor network adapter, the SR-IOV NIC L2 agent should be deployed to leverage this functionality .

==VM creation flow with SR-IOV vNIC==
* Create one or more neutron ports. Run:
neutron port-create <net-id> --binding:vnic-type <direct | macvtap | normal>

* Boot VM with one or more neutron ports. Run:
nova boot --flavor m1.large --image <image>
--nic port-id=<port1> --nic port-id=<port2> <vm name>

Note that in the nova boot API, users can specify either a port-ID or a net-ID. If a net-ID is specified, it is assumed that the user is requesting a normal virtual port (which is not an SR-IOV port).

=SR-IOV Configuration=

===Neutron Server===
Using ML2 Neutron plugin modify /etc/neutron/plugins/ml2/ml2_conf.ini:

[ml2]
tenant_network_types = vlan
type_drivers = vlan
mechanism_drivers = openvswitch,sriovnicswitch
[ml2_type_vlan]
network_vlan_ranges = physnet1:2:100

If SRIOV network adapters support VF link state setting and admin state management is desired, make sure to add /etc/neutron/plugins/ml2/ml2_conf_sriov.ini with following section:

[ml2_sriov]
agent_required = True

Neutron server should be run with the two configuration files /etc/neutron/plugins/ml2/ml2_conf.in and /etc/neutron/plugins/ml2/ml2_conf_sriov.ini
neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini

==Compute==
===nova-compute===
On each compute node you have to associate the VFs available to each physical network.
That is performed by configuring pci_passthrough_whitelist in /etc/nova/nova.conf. So, for example:
pci_passthrough_whitelist = {"address":"*:0a:00.*","physical_network":"physnet1"}
This associates any VF with address that includes ':0a:00.' in its address to the physical network physnet1.

After configuring the whitelist you have to restart nova-compute service.

===SR-IOV neutron agent===
If the hardware supports it and you want to enable changing the port admin_state, you have to run the Neutron SR-IOV agent. 

'''Note:'''If you configured agent_required=True on the Neutron server, you must run the Agent on each compute node.

In /etc/neutron/plugins/ml2/ml2_conf.ini make sure you have the following:
[securitygroup]
firewall_driver = neutron.agent.firewall.NoopFirewallDriver

Modify /etc/neutron/plugins/ml2/ml2_conf_sriov.ini as follows:

[sriov_nic]
physical_device_mappings = physnet1:eth1
exclude_devices =

Where:
* physnet1 is the physical network
* eth1 is the physical function (PF)
* exclude_devices is empty so all the VFs associated with eth1 may be configured by the agent

After modifying the configuration file, start the Neutron SR-IOV agent. Run:
neutron-sriov-nic-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini

====Exclude VFs====
If you want to exclude some of the VFs so the agent does not configure them, you need to list them in the sriov_nic section: 

'''Example:''' exclude_devices = eth1:0000:07:00.2; 0000:07:00.3, eth2:0000:05:00.1; 0000:05:00.2

=References=

[http://community.mellanox.com/docs/DOC-1484 Openstack ML2 SR-IOV driver support]

Juno-SR-IOV

2014-09-09T14:24:25Z

Itzikb: /* Neutron Server */

=This Page is under construction=
Once it will be ready, the content might move to another location.
=SR-IOV in OpenStack Juno=
OpenStack Juno adds inbox support for SR-IOV ....
==Nova==
Nova support for SR-IOV is ....

==Neutron==
Neutron support for SR-IOV is ...
embedded NIC is ...

=SR-IOV Configuration=

===Neutron Server===
Using ML2 Neutron plugin modify /etc/neutron/plugins/ml2/ml2_conf.ini:

[ml2]
tenant_network_types = vlan
type_drivers = vlan
mechanism_drivers = openvswitch,sriovnicswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100

Make sure /etc/neutron/plugins/ml2/ml2_conf_sriov.ini has the following section:

[ml2_sriov]
agent_required = True

Neutron server should be run with the two configuration files /etc/neutron/plugins/ml2/ml2_conf.in and /etc/neutron/plugins/ml2/ml2_conf_sriov.ini
neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini

==Compute==
===Nova===
On each compute you have to associate the Virtual Functions available to each Physical Network.
You do it by configuring pci_passthrough_whitelist in /etc/nova/noca.conf

For example:
pci_passthrough_whitelist = {"address":"*:0a:00.*","physical_network":"physnet1"}
This will associate any VF with address that includes ':0a:00.' in it's address to the Physical network physnet1

After configuring the white list you have to restart nova-compute service.

===Neutron===
If the hardware supports and and you want to enable changing the port admin_state you have to run the Neutron SR-IOV agent. 

'''Note:'''If you configured agent_required=True on the Neutron server you must run the Agent on each compute node.

In /etc/neutron/plugins/ml2/ml2_conf.ini make sure you have the following:
[securitygroup]
firewall_driver = neutron.agent.firewall.NoopFirewallDriver

Modify /etc/neutron/plugins/ml2/ml2_conf_sriov.ini as follows

[sriov_nic]
physical_device_mappings = physnet1:eth1
exclude_devices =

physnet1 is the physical network
eth1 if the Physical Function (PF)
exclude_devices is empty so All the VFs associated with eth1 are allowed to be configured by the agent.

After modifying the configuration file start the Neutron SR-IOV agent:
neutron-sriov-nic-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini

====Exclude VFs====
If you want to exclude some of the VFs so the agent won't configure them you need to list them in the sriov_nic section: 

'''Example:''' exclude_devices = eth1:0000:07:00.2; 0000:07:00.3, eth2:0000:05:00.1; 0000:05:00.2

Juno-SR-IOV

2014-09-09T14:24:11Z

Itzikb: /* Neutron */

=This Page is under construction=
Once it will be ready, the content might move to another location.
=SR-IOV in OpenStack Juno=
OpenStack Juno adds inbox support for SR-IOV ....
==Nova==
Nova support for SR-IOV is ....

==Neutron==
Neutron support for SR-IOV is ...
embedded NIC is ...

=SR-IOV Configuration=

===Neutron Server===
Using ML2 Neutron plugin modify /etc/neutron/plugins/ml2/ml2_conf.ini:

[ml2]
tenant_network_types = vlan
type_drivers = vlan
mechanism_drivers = openvswitch,sriovnicswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[securitygroup]
firewall_driver = neutron.agent.firewall.NoopFirewallDriver

Make sure /etc/neutron/plugins/ml2/ml2_conf_sriov.ini has the following section:

[ml2_sriov]
agent_required = True

Neutron server should be run with the two configuration files /etc/neutron/plugins/ml2/ml2_conf.in and /etc/neutron/plugins/ml2/ml2_conf_sriov.ini
neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini

==Compute==
===Nova===
On each compute you have to associate the Virtual Functions available to each Physical Network.
You do it by configuring pci_passthrough_whitelist in /etc/nova/noca.conf

For example:
pci_passthrough_whitelist = {"address":"*:0a:00.*","physical_network":"physnet1"}
This will associate any VF with address that includes ':0a:00.' in it's address to the Physical network physnet1

After configuring the white list you have to restart nova-compute service.

===Neutron===
If the hardware supports and and you want to enable changing the port admin_state you have to run the Neutron SR-IOV agent. 

'''Note:'''If you configured agent_required=True on the Neutron server you must run the Agent on each compute node.

In /etc/neutron/plugins/ml2/ml2_conf.ini make sure you have the following:
[securitygroup]
firewall_driver = neutron.agent.firewall.NoopFirewallDriver

Modify /etc/neutron/plugins/ml2/ml2_conf_sriov.ini as follows

[sriov_nic]
physical_device_mappings = physnet1:eth1
exclude_devices =

physnet1 is the physical network
eth1 if the Physical Function (PF)
exclude_devices is empty so All the VFs associated with eth1 are allowed to be configured by the agent.

After modifying the configuration file start the Neutron SR-IOV agent:
neutron-sriov-nic-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini

====Exclude VFs====
If you want to exclude some of the VFs so the agent won't configure them you need to list them in the sriov_nic section: 

'''Example:''' exclude_devices = eth1:0000:07:00.2; 0000:07:00.3, eth2:0000:05:00.1; 0000:05:00.2

Juno-SR-IOV

2014-09-09T14:20:05Z

Itzikb: /* Exclude VFs */

=This Page is under construction=
Once it will be ready, the content might move to another location.
=SR-IOV in OpenStack Juno=
OpenStack Juno adds inbox support for SR-IOV ....
==Nova==
Nova support for SR-IOV is ....

==Neutron==
Neutron support for SR-IOV is ...
embedded NIC is ...

=SR-IOV Configuration=

===Neutron Server===
Using ML2 Neutron plugin modify /etc/neutron/plugins/ml2/ml2_conf.ini:

[ml2]
tenant_network_types = vlan
type_drivers = vlan
mechanism_drivers = openvswitch,sriovnicswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[securitygroup]
firewall_driver = neutron.agent.firewall.NoopFirewallDriver

Make sure /etc/neutron/plugins/ml2/ml2_conf_sriov.ini has the following section:

[ml2_sriov]
agent_required = True

Neutron server should be run with the two configuration files /etc/neutron/plugins/ml2/ml2_conf.in and /etc/neutron/plugins/ml2/ml2_conf_sriov.ini
neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini

==Compute==
===Nova===
On each compute you have to associate the Virtual Functions available to each Physical Network.
You do it by configuring pci_passthrough_whitelist in /etc/nova/noca.conf

For example:
pci_passthrough_whitelist = {"address":"*:0a:00.*","physical_network":"physnet1"}
This will associate any VF with address that includes ':0a:00.' in it's address to the Physical network physnet1

After configuring the white list you have to restart nova-compute service.

===Neutron===
If the hardware supports and and you want to enable changing the port admin_state you have to run the Neutron SR-IOV agent. 

'''Note:'''If you configured agent_required=True on the Neutron server you must run the Agent on each compute node.

Modify /etc/neutron/plugins/ml2/ml2_conf_sriov.ini as follows

[sriov_nic]
physical_device_mappings = physnet1:eth1
exclude_devices =

physnet1 is the physical network
eth1 if the Physical Function (PF)
exclude_devices is empty so All the VFs associated with eth1 are allowed to be configured by the agent.

After modifying the configuration file start the Neutron SR-IOV agent:
neutron-sriov-nic-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini

====Exclude VFs====
If you want to exclude some of the VFs so the agent won't configure them you need to list them in the sriov_nic section: 

'''Example:''' exclude_devices = eth1:0000:07:00.2; 0000:07:00.3, eth2:0000:05:00.1; 0000:05:00.2

Juno-SR-IOV

2014-09-09T14:18:13Z

Itzikb: /* Exclude VFs */

=This Page is under construction=
Once it will be ready, the content might move to another location.
=SR-IOV in OpenStack Juno=
OpenStack Juno adds inbox support for SR-IOV ....
==Nova==
Nova support for SR-IOV is ....

==Neutron==
Neutron support for SR-IOV is ...
embedded NIC is ...

=SR-IOV Configuration=

===Neutron Server===
Using ML2 Neutron plugin modify /etc/neutron/plugins/ml2/ml2_conf.ini:

[ml2]
tenant_network_types = vlan
type_drivers = vlan
mechanism_drivers = openvswitch,sriovnicswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[securitygroup]
firewall_driver = neutron.agent.firewall.NoopFirewallDriver

Make sure /etc/neutron/plugins/ml2/ml2_conf_sriov.ini has the following section:

[ml2_sriov]
agent_required = True

Neutron server should be run with the two configuration files /etc/neutron/plugins/ml2/ml2_conf.in and /etc/neutron/plugins/ml2/ml2_conf_sriov.ini
neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini

==Compute==
===Nova===
On each compute you have to associate the Virtual Functions available to each Physical Network.
You do it by configuring pci_passthrough_whitelist in /etc/nova/noca.conf

For example:
pci_passthrough_whitelist = {"address":"*:0a:00.*","physical_network":"physnet1"}
This will associate any VF with address that includes ':0a:00.' in it's address to the Physical network physnet1

After configuring the white list you have to restart nova-compute service.

===Neutron===
If the hardware supports and and you want to enable changing the port admin_state you have to run the Neutron SR-IOV agent. 

'''Note:'''If you configured agent_required=True on the Neutron server you must run the Agent on each compute node.

Modify /etc/neutron/plugins/ml2/ml2_conf_sriov.ini as follows

[sriov_nic]
physical_device_mappings = physnet1:eth1
exclude_devices =

physnet1 is the physical network
eth1 if the Physical Function (PF)
exclude_devices is empty so All the VFs associated with eth1 are allowed to be configured by the agent.

After modifying the configuration file start the Neutron SR-IOV agent:
neutron-sriov-nic-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini

====Exclude VFs====
If you want to exclude some of the VFs so the agent won't configure them you need to list them in the sriov_nic section: 

'''Example:''' exclude_devices = eth1:0000:07:00.2; 0000:07:00.3

Juno-SR-IOV

2014-09-09T14:14:27Z

Itzikb: /* Exclude VFs */

=This Page is under construction=
Once it will be ready, the content might move to another location.
=SR-IOV in OpenStack Juno=
OpenStack Juno adds inbox support for SR-IOV ....
==Nova==
Nova support for SR-IOV is ....

==Neutron==
Neutron support for SR-IOV is ...
embedded NIC is ...

=SR-IOV Configuration=

===Neutron Server===
Using ML2 Neutron plugin modify /etc/neutron/plugins/ml2/ml2_conf.ini:

[ml2]
tenant_network_types = vlan
type_drivers = vlan
mechanism_drivers = openvswitch,sriovnicswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[securitygroup]
firewall_driver = neutron.agent.firewall.NoopFirewallDriver

Make sure /etc/neutron/plugins/ml2/ml2_conf_sriov.ini has the following section:

[ml2_sriov]
agent_required = True

Neutron server should be run with the two configuration files /etc/neutron/plugins/ml2/ml2_conf.in and /etc/neutron/plugins/ml2/ml2_conf_sriov.ini
neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini

==Compute==
===Nova===
On each compute you have to associate the Virtual Functions available to each Physical Network.
You do it by configuring pci_passthrough_whitelist in /etc/nova/noca.conf

For example:
pci_passthrough_whitelist = {"address":"*:0a:00.*","physical_network":"physnet1"}
This will associate any VF with address that includes ':0a:00.' in it's address to the Physical network physnet1

After configuring the white list you have to restart nova-compute service.

===Neutron===
If the hardware supports and and you want to enable changing the port admin_state you have to run the Neutron SR-IOV agent. 

'''Note:'''If you configured agent_required=True on the Neutron server you must run the Agent on each compute node.

Modify /etc/neutron/plugins/ml2/ml2_conf_sriov.ini as follows

[sriov_nic]
physical_device_mappings = physnet1:eth1
exclude_devices =

physnet1 is the physical network
eth1 if the Physical Function (PF)
exclude_devices is empty so All the VFs associated with eth1 are allowed to be configured by the agent.

After modifying the configuration file start the Neutron SR-IOV agent:
neutron-sriov-nic-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini

====Exclude VFs====
If you want to exclude some of the VFs so the agent won't configure them you need to put list them in the sriov_nic section: 

'''Example:''' exclude_devices = eth1:0000:07:00.2; 0000:07:00.3

Juno-SR-IOV

2014-09-09T14:13:51Z

Itzikb: /* Neutron */

=This Page is under construction=
Once it will be ready, the content might move to another location.
=SR-IOV in OpenStack Juno=
OpenStack Juno adds inbox support for SR-IOV ....
==Nova==
Nova support for SR-IOV is ....

==Neutron==
Neutron support for SR-IOV is ...
embedded NIC is ...

=SR-IOV Configuration=

===Neutron Server===
Using ML2 Neutron plugin modify /etc/neutron/plugins/ml2/ml2_conf.ini:

[ml2]
tenant_network_types = vlan
type_drivers = vlan
mechanism_drivers = openvswitch,sriovnicswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[securitygroup]
firewall_driver = neutron.agent.firewall.NoopFirewallDriver

Make sure /etc/neutron/plugins/ml2/ml2_conf_sriov.ini has the following section:

[ml2_sriov]
agent_required = True

Neutron server should be run with the two configuration files /etc/neutron/plugins/ml2/ml2_conf.in and /etc/neutron/plugins/ml2/ml2_conf_sriov.ini
neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini

==Compute==
===Nova===
On each compute you have to associate the Virtual Functions available to each Physical Network.
You do it by configuring pci_passthrough_whitelist in /etc/nova/noca.conf

For example:
pci_passthrough_whitelist = {"address":"*:0a:00.*","physical_network":"physnet1"}
This will associate any VF with address that includes ':0a:00.' in it's address to the Physical network physnet1

After configuring the white list you have to restart nova-compute service.

===Neutron===
If the hardware supports and and you want to enable changing the port admin_state you have to run the Neutron SR-IOV agent. 

'''Note:'''If you configured agent_required=True on the Neutron server you must run the Agent on each compute node.

Modify /etc/neutron/plugins/ml2/ml2_conf_sriov.ini as follows

[sriov_nic]
physical_device_mappings = physnet1:eth1
exclude_devices =

physnet1 is the physical network
eth1 if the Physical Function (PF)
exclude_devices is empty so All the VFs associated with eth1 are allowed to be configured by the agent.

After modifying the configuration file start the Neutron SR-IOV agent:
neutron-sriov-nic-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini

====Exclude VFs====
If you want to exclude some of the VFs so the agent won't configure them you need to put list them in the sriov_nic section:
'''Example:'''exclude_devices = eth1:0000:07:00.2; 0000:07:00.3

Juno-SR-IOV

2014-09-09T14:12:36Z

Itzikb: /* Neutron */

=This Page is under construction=
Once it will be ready, the content might move to another location.
=SR-IOV in OpenStack Juno=
OpenStack Juno adds inbox support for SR-IOV ....
==Nova==
Nova support for SR-IOV is ....

==Neutron==
Neutron support for SR-IOV is ...
embedded NIC is ...

=SR-IOV Configuration=

===Neutron Server===
Using ML2 Neutron plugin modify /etc/neutron/plugins/ml2/ml2_conf.ini:

[ml2]
tenant_network_types = vlan
type_drivers = vlan
mechanism_drivers = openvswitch,sriovnicswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[securitygroup]
firewall_driver = neutron.agent.firewall.NoopFirewallDriver

Make sure /etc/neutron/plugins/ml2/ml2_conf_sriov.ini has the following section:

[ml2_sriov]
agent_required = True

Neutron server should be run with the two configuration files /etc/neutron/plugins/ml2/ml2_conf.in and /etc/neutron/plugins/ml2/ml2_conf_sriov.ini
neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini

==Compute==
===Nova===
On each compute you have to associate the Virtual Functions available to each Physical Network.
You do it by configuring pci_passthrough_whitelist in /etc/nova/noca.conf

For example:
pci_passthrough_whitelist = {"address":"*:0a:00.*","physical_network":"physnet1"}
This will associate any VF with address that includes ':0a:00.' in it's address to the Physical network physnet1

After configuring the white list you have to restart nova-compute service.

===Neutron===
If the hardware supports and and you want to enable changing the port admin_state you have to run the Neutron SR-IOV agent. 

'''Note:'''If you configured agent_required=True on the Neutron server you must run the Agent on each compute node.

Modify /etc/neutron/plugins/ml2/ml2_conf_sriov.ini as follows

[sriov_nic]
physical_device_mappings = physnet1:eth1
exclude_devices =

physnet1 is the physical network
eth1 if the Physical Function (PF)
exclude_devices is empty so All the VFs associated with eth1 are allowed to be configured by the agent.

After modifying the configuration file start the Neutron SR-IOV agent.

====Exclude VFs====
If you want to exclude some of the VFs so the agent won't configure them you need to put list them in the sriov_nic section:
'''Example:'''exclude_devices = eth1:0000:07:00.2; 0000:07:00.3

Juno-SR-IOV

2014-09-09T11:05:26Z

Itzikb: /* Neutron */

=This Page is under construction=
Once it will be ready, the content might move to another location.
=SR-IOV in OpenStack Juno=
OpenStack Juno adds inbox support for SR-IOV ....
==Nova==
Nova support for SR-IOV is ....

==Neutron==
Neutron support for SR-IOV is ...
embedded NIC is ...

=SR-IOV Configuration=

===Neutron Server===
Using ML2 Neutron plugin modify /etc/neutron/plugins/ml2/ml2_conf.ini:

[ml2]
tenant_network_types = vlan
type_drivers = vlan
mechanism_drivers = openvswitch,sriovnicswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[securitygroup]
firewall_driver = neutron.agent.firewall.NoopFirewallDriver

Make sure /etc/neutron/plugins/ml2/ml2_conf_sriov.ini has the following section:

[ml2_sriov]
agent_required = True

Neutron server should be run with the two configuration files /etc/neutron/plugins/ml2/ml2_conf.in and /etc/neutron/plugins/ml2/ml2_conf_sriov.ini
neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini

==Compute==
===Nova===
On each compute you have to associate the Virtual Functions available to each Physical Network.
You do it by configuring pci_passthrough_whitelist in /etc/nova/noca.conf

For example:
pci_passthrough_whitelist = {"address":"*:0a:00.*","physical_network":"physnet1"}
This will associate any VF with address that includes ':0a:00.' in it's address to the Physical network physnet1

After configuring the white list you have to restart nova-compute service.

===Neutron===
If the hardware supports and and you want to enable changing the port admin_state you have to run the Neutron SR-IOV agent. 

'''Note:'''If you configured agent_required=True on the Neutron server you must run the Agent on each compute node.

Modify /etc/neutron/plugins/ml2/ml2_conf_sriov.ini as follows

[sriov_nic]
physical_device_mappings = physnet1:eth1
exclude_devices =

Where physnet1 is the physical network, eth1 if the Physical Function (PF).
exclude_devices is empty so All the VFs associated with eth1 are allowed to be configured by the agent.

Juno-SR-IOV

2014-09-09T09:11:44Z

Itzikb: /* Neutron Server */

Juno-SR-IOV

2014-09-09T09:03:10Z

Itzikb: /* Nova */

Juno-SR-IOV

2014-09-09T09:02:11Z

Itzikb: /* Neutron */

Juno-SR-IOV

2014-09-07T07:30:18Z

Itzikb: /* SR-IOV Configuration */

Juno-SR-IOV

2014-09-07T07:25:12Z

Itzikb: /* SR-IOV Configuration */

Mellanox-Neutron-Icehouse-Redhat

2014-09-01T14:27:05Z

Itzikb: /* OpenSM configuration - Without UFM */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

You can do it by using packstack:

1.Create an answer file
#packstack --gen-answer-file=GEN_ANSWER_FILE

2. Change the following in GEN_ANSWER_FILE
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_VLAN_RANGES=default:2:10
CONFIG_NEUTRON_L2_AGENT=openvswitch

3. Run packstack
#packstack --answer-file=GEN_ANSWER_FILE

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = openvswitch,mlnx
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. In /etc/nova/nova.conf

Check that the compute driver is libvirt:
[DEFAULT]
compute_driver=libvirt.LibvirtDriver

Change the VIF driver:
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized mode make sure vif driver is as follows:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc_support_old_agents''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#service openstack-nova-compute restart

2. Start eswitch Daemon
#service eswitchd start

3. Start the Neutron agent
#service neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/ml2/ml2_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#service neutron-openvswitch-agent start
#service neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#service neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#service opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==
We are using the linuxbridge mechanism driver so we can use the DHCP Server with the Linux Bridge interface driver.

Edit /etc/neutron/plugins/ml2/ml2_conf.ini as follows (The VLAN range is an example)

[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = linuxbridge,mlnx
[ml2_type_vlan]
network_vlan_ranges=default:2:10
[securitygroup]
enable_security_group = True
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

== Compute Nodes ==

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#service openibd restart
eswitchd should be started and then Neutron agent should be started
#service eswitchd restart
#service neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#service openstack-nova-compute restart

== Network Node ==

Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:

E_IPOIB_LOAD=yes
And restart openibd:

#service openibd restart
Please refer to eIPoIB configuration in Mellanox OFED User Manual
Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example: Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")

#ethtool -i <interface>
driver: eth_ipoib
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Restart neutron-linuxbridge-agent and neutron-dhcp-agent

#service neutron-linuxbridge-agent restart

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Start dhcp server
#service neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-09-01T14:26:17Z

Itzikb: /* Network Node */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

You can do it by using packstack:

1.Create an answer file
#packstack --gen-answer-file=GEN_ANSWER_FILE

2. Change the following in GEN_ANSWER_FILE
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_VLAN_RANGES=default:2:10
CONFIG_NEUTRON_L2_AGENT=openvswitch

3. Run packstack
#packstack --answer-file=GEN_ANSWER_FILE

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = openvswitch,mlnx
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. In /etc/nova/nova.conf

Check that the compute driver is libvirt:
[DEFAULT]
compute_driver=libvirt.LibvirtDriver

Change the VIF driver:
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized mode make sure vif driver is as follows:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc_support_old_agents''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#service openstack-nova-compute restart

2. Start eswitch Daemon
#service eswitchd start

3. Start the Neutron agent
#service neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/ml2/ml2_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#service neutron-openvswitch-agent start
#service neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#service neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==
We are using the linuxbridge mechanism driver so we can use the DHCP Server with the Linux Bridge interface driver.

Edit /etc/neutron/plugins/ml2/ml2_conf.ini as follows (The VLAN range is an example)

[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = linuxbridge,mlnx
[ml2_type_vlan]
network_vlan_ranges=default:2:10
[securitygroup]
enable_security_group = True
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

== Compute Nodes ==

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#service openibd restart
eswitchd should be started and then Neutron agent should be started
#service eswitchd restart
#service neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#service openstack-nova-compute restart

== Network Node ==

Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:

E_IPOIB_LOAD=yes
And restart openibd:

#service openibd restart
Please refer to eIPoIB configuration in Mellanox OFED User Manual
Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example: Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")

#ethtool -i <interface>
driver: eth_ipoib
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Restart neutron-linuxbridge-agent and neutron-dhcp-agent

#service neutron-linuxbridge-agent restart

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Start dhcp server
#service neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-09-01T14:25:29Z

Itzikb: /* Start Services */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

You can do it by using packstack:

1.Create an answer file
#packstack --gen-answer-file=GEN_ANSWER_FILE

2. Change the following in GEN_ANSWER_FILE
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_VLAN_RANGES=default:2:10
CONFIG_NEUTRON_L2_AGENT=openvswitch

3. Run packstack
#packstack --answer-file=GEN_ANSWER_FILE

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = openvswitch,mlnx
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. In /etc/nova/nova.conf

Check that the compute driver is libvirt:
[DEFAULT]
compute_driver=libvirt.LibvirtDriver

Change the VIF driver:
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized mode make sure vif driver is as follows:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc_support_old_agents''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#service openstack-nova-compute restart

2. Start eswitch Daemon
#service eswitchd start

3. Start the Neutron agent
#service neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/ml2/ml2_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-openvswitch-agent start
#/etc/init.d/neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#/etc/init.d/neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==
We are using the linuxbridge mechanism driver so we can use the DHCP Server with the Linux Bridge interface driver.

Edit /etc/neutron/plugins/ml2/ml2_conf.ini as follows (The VLAN range is an example)

[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = linuxbridge,mlnx
[ml2_type_vlan]
network_vlan_ranges=default:2:10
[securitygroup]
enable_security_group = True
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

== Compute Nodes ==

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#service openibd restart
eswitchd should be started and then Neutron agent should be started
#service eswitchd restart
#service neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#service openstack-nova-compute restart

== Network Node ==

Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:

E_IPOIB_LOAD=yes
And restart openibd:

#service openibd restart
Please refer to eIPoIB configuration in Mellanox OFED User Manual
Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example: Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")

#ethtool -i <interface>
driver: eth_ipoib
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Restart neutron-linuxbridge-agent and neutron-dhcp-agent

#service neutron-linuxbridge-agent restart

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Start dhcp server
#service neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-09-01T14:24:25Z

Itzikb: /* Network Node */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

You can do it by using packstack:

1.Create an answer file
#packstack --gen-answer-file=GEN_ANSWER_FILE

2. Change the following in GEN_ANSWER_FILE
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_VLAN_RANGES=default:2:10
CONFIG_NEUTRON_L2_AGENT=openvswitch

3. Run packstack
#packstack --answer-file=GEN_ANSWER_FILE

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = openvswitch,mlnx
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. In /etc/nova/nova.conf

Check that the compute driver is libvirt:
[DEFAULT]
compute_driver=libvirt.LibvirtDriver

Change the VIF driver:
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized mode make sure vif driver is as follows:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc_support_old_agents''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/ml2/ml2_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-openvswitch-agent start
#/etc/init.d/neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#/etc/init.d/neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==
We are using the linuxbridge mechanism driver so we can use the DHCP Server with the Linux Bridge interface driver.

Edit /etc/neutron/plugins/ml2/ml2_conf.ini as follows (The VLAN range is an example)

[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = linuxbridge,mlnx
[ml2_type_vlan]
network_vlan_ranges=default:2:10
[securitygroup]
enable_security_group = True
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

== Compute Nodes ==

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#service openibd restart
eswitchd should be started and then Neutron agent should be started
#service eswitchd restart
#service neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#service openstack-nova-compute restart

== Network Node ==

Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:

E_IPOIB_LOAD=yes
And restart openibd:

#service openibd restart
Please refer to eIPoIB configuration in Mellanox OFED User Manual
Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example: Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")

#ethtool -i <interface>
driver: eth_ipoib
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Restart neutron-linuxbridge-agent and neutron-dhcp-agent

#service neutron-linuxbridge-agent restart

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Start dhcp server
#service neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-09-01T14:09:53Z

Itzikb: /* SR-IOV */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

You can do it by using packstack:

1.Create an answer file
#packstack --gen-answer-file=GEN_ANSWER_FILE

2. Change the following in GEN_ANSWER_FILE
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_VLAN_RANGES=default:2:10
CONFIG_NEUTRON_L2_AGENT=openvswitch

3. Run packstack
#packstack --answer-file=GEN_ANSWER_FILE

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = openvswitch,mlnx
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. In /etc/nova/nova.conf

Check that the compute driver is libvirt:
[DEFAULT]
compute_driver=libvirt.LibvirtDriver

Change the VIF driver:
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized mode make sure vif driver is as follows:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc_support_old_agents''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/ml2/ml2_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-openvswitch-agent start
#/etc/init.d/neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#/etc/init.d/neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==
We are using the linuxbridge mechanism driver so we can use the DHCP Server with the Linux Bridge interface driver.

Edit /etc/neutron/plugins/ml2/ml2_conf.ini as follows (The VLAN range is an example)

[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = linuxbridge,mlnx
[ml2_type_vlan]
network_vlan_ranges=default:2:10
[securitygroup]
enable_security_group = True
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

== Compute Nodes ==

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#service openibd restart
eswitchd should be started and then Neutron agent should be started
#service eswitchd restart
#service neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#service openstack-nova-compute restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

In case you use MLNX-OFED older than 2.1, Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
#/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
#/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-09-01T14:05:52Z

Itzikb: /* Para-Virtualized */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

You can do it by using packstack:

1.Create an answer file
#packstack --gen-answer-file=GEN_ANSWER_FILE

2. Change the following in GEN_ANSWER_FILE
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_VLAN_RANGES=default:2:10
CONFIG_NEUTRON_L2_AGENT=openvswitch

3. Run packstack
#packstack --answer-file=GEN_ANSWER_FILE

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = openvswitch,mlnx
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. In /etc/nova/nova.conf

Check that the compute driver is libvirt:
[DEFAULT]
compute_driver=libvirt.LibvirtDriver

Change the VIF driver:
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized mode make sure vif driver is as follows:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc_support_old_agents''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/ml2/ml2_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-openvswitch-agent start
#/etc/init.d/neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#/etc/init.d/neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==
We are using the linuxbridge mechanism driver so we can use the DHCP Server with the Linux Bridge interface driver.

Edit /etc/neutron/plugins/ml2/ml2_conf.ini as follows (The VLAN range is an example)

[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = linuxbridge,mlnx
[ml2_type_vlan]
network_vlan_ranges=default:2:10
[securitygroup]
enable_security_group = True
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
#/etc/init.d/eswitchd restart
#/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

In case you use MLNX-OFED older than 2.1, Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
#/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
#/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-09-01T14:04:41Z

Itzikb: /* Neutron Server Node */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

You can do it by using packstack:

1.Create an answer file
#packstack --gen-answer-file=GEN_ANSWER_FILE

2. Change the following in GEN_ANSWER_FILE
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_VLAN_RANGES=default:2:10
CONFIG_NEUTRON_L2_AGENT=openvswitch

3. Run packstack
#packstack --answer-file=GEN_ANSWER_FILE

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = openvswitch,mlnx
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. In /etc/nova/nova.conf

Check that the compute driver is libvirt:
[DEFAULT]
compute_driver=libvirt.LibvirtDriver

Change the VIF driver:
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized mode make sure vif driver is as follows:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc_support_old_agents''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/ml2/ml2_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-openvswitch-agent start
#/etc/init.d/neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#/etc/init.d/neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==
We are using the linuxbridge mechanism driver so we can use the DHCP Server with the Linux Bridge interface driver.

Edit /etc/neutron/plugins/ml2/ml2_conf.ini as follows (The VLAN range is an example)

[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = linuxbridge,mlnx
[ml2_type_vlan]
network_vlan_ranges=default:2:10
[securitygroup]
enable_security_group = True
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
#/etc/init.d/eswitchd restart
#/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

In case you use MLNX-OFED older than 2.1, Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
#/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
#/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-09-01T13:47:06Z

Itzikb: /* Start Services */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

You can do it by using packstack:

1.Create an answer file
#packstack --gen-answer-file=GEN_ANSWER_FILE

2. Change the following in GEN_ANSWER_FILE
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_VLAN_RANGES=default:2:10
CONFIG_NEUTRON_L2_AGENT=openvswitch

3. Run packstack
#packstack --answer-file=GEN_ANSWER_FILE

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = openvswitch,mlnx
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. In /etc/nova/nova.conf

Check that the compute driver is libvirt:
[DEFAULT]
compute_driver=libvirt.LibvirtDriver

Change the VIF driver:
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized mode make sure vif driver is as follows:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc_support_old_agents''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/ml2/ml2_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-openvswitch-agent start
#/etc/init.d/neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#/etc/init.d/neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
#/etc/init.d/eswitchd restart
#/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

In case you use MLNX-OFED older than 2.1, Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
#/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
#/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-07-22T07:23:39Z

Itzikb: /* Configuration */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

You can do it by using packstack:

1.Create an answer file
#packstack --gen-answer-file=GEN_ANSWER_FILE

2. Change the following in GEN_ANSWER_FILE
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_VLAN_RANGES=default:2:10
CONFIG_NEUTRON_L2_AGENT=openvswitch

3. Run packstack
#packstack --answer-file=GEN_ANSWER_FILE

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = openvswitch,mlnx
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. In /etc/nova/nova.conf

Check that the compute driver is libvirt:
[DEFAULT]
compute_driver=libvirt.LibvirtDriver

Change the VIF driver:
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized mode make sure vif driver is as follows:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc_support_old_agents''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/ml2/ml2_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-openvswitch-agent start
#/etc/init.d/neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#/etc/init.d/neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
#/etc/init.d/eswitchd restart
#/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

In case you use MLNX-OFED older than 2.1, Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
#/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
#/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-07-10T11:57:41Z

Itzikb: /* Network Node */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

You can do it by using packstack:

1.Create an answer file
#packstack --gen-answer-file=GEN_ANSWER_FILE

2. Change the following in GEN_ANSWER_FILE
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_VLAN_RANGES=default:2:10
CONFIG_NEUTRON_L2_AGENT=openvswitch

3. Run packstack
#packstack --answer-file=GEN_ANSWER_FILE

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = openvswitch,mlnx
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

[DEFAULT]
compute_driver=nova.virt.libvirt.driver.LibvirtDriver
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc_support_old_agents''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/ml2/ml2_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-openvswitch-agent start
#/etc/init.d/neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#/etc/init.d/neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
#/etc/init.d/eswitchd restart
#/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

In case you use MLNX-OFED older than 2.1, Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
#/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
#/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-07-03T11:43:31Z

Itzikb: /* Installation */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

You can do it by using packstack:

1.Create an answer file
#packstack --gen-answer-file=GEN_ANSWER_FILE

2. Change the following in GEN_ANSWER_FILE
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_VLAN_RANGES=default:2:10
CONFIG_NEUTRON_L2_AGENT=openvswitch

3. Run packstack
#packstack --answer-file=GEN_ANSWER_FILE

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = openvswitch,mlnx
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

[DEFAULT]
compute_driver=nova.virt.libvirt.driver.LibvirtDriver
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc_support_old_agents''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-openvswitch-agent start
#/etc/init.d/neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#/etc/init.d/neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
#/etc/init.d/eswitchd restart
#/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

In case you use MLNX-OFED older than 2.1, Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
#/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
#/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-06-26T12:18:24Z

Itzikb: /* Configuration */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

You can do it by using packstack:

1.Create an answer file
#packstack --gen-answer-file=GEN_ANSWER_FILE

2. Change the following in GEN_ANSWER_FILE
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_VLAN_RANGES=default:2:10
CONFIG_NEUTRON_L2_AGENT=openvswitch

3. Run packstack
#packstack --answer-file=GEN_ANSWER_FILE

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = mlnx,openvswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

[DEFAULT]
compute_driver=nova.virt.libvirt.driver.LibvirtDriver
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc_support_old_agents''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-openvswitch-agent start
#/etc/init.d/neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#/etc/init.d/neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
#/etc/init.d/eswitchd restart
#/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

In case you use MLNX-OFED older than 2.1, Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
#/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
#/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Meetings/Passthrough

2014-06-24T11:45:29Z

Itzikb: /* Agenda on Jun 24th, 2014 */

Next meeting will be on Wed May 7th at 1300 UTC on #openstack-meeting-alt

== Apologies for Absence ==
* 2014-06-24 Irena Berezovsky (irenab)

=== Agenda on May 26th, 2014 ===
* Nova SR-IOV networking
** [https://review.openstack.org/#/c/86606/ nova spec review]
** patch submission soon
** tempest tests
* Bugs
** [https://review.openstack.org/#/c/82206/ Init PCI manager after compute node initialization]
** [https://review.openstack.org/#/c/81954/ makes sure correct PCI device allocation]
* Other works/features
** Live migration
** Attach SR-IOV port to an existing instance
** possibly changing the names of existing vnic-types
** associate a default vnic-type with a neutron network
** admin controlled knobs
** HA
** others

=== Agenda on March 25th, 2014 ===
* PCI SR-IOV Networking use case
Refer to https://docs.google.com/document/d/1zgMaXqrCnad01-jQH7Mkmf6amlghw9RMScGLBrKslmw/edit

=== Recap of Discussions ===
==== Current PCI Passthrough ====

* How to Use Wiki: https://wiki.openstack.org/wiki/Pci_passthrough

* PCI whitelist: defines all the PCI passthrough devices that are available on a compute node. It's currently based on <vendor_id> <product_id>
* PCI stats group: defines the keys based on which a PCI device is accounted. It's currently based on the keys: <vendor_id> <product_id> <extra_info or simply PCI address of a physical function>.
* PCI Alias: specifies a list of PCI device requirements. A PCI requirement specifies a dictionary with keys: <alias name> <vendor_id> <product_id> <device_type>. Multiple requirements can be specified per PCI Alias with the same <device_type> (which may not work yet due to bugs). <alias name> is required. Others are optional.
* Nova Server Flavor: PCI requirements can be added into a nova flavor as it's extra-spec in the syntax "pci_passthrough:alias"="<PCI Alias Name>:<count>{,<PCI Alias Name>:<count>}"
* PCI Passthrough filter for nova scheduler: this filter works based on the PCI stats group and PCI aliases referenced in the nova flavor. If mutliple aliases exist in the flavor, all of the them have to be satified. If multiple PCI requirements exist in one PCI alias, only one of the requirements defined in the alias needs to be satisfied. Suppose a PCI requirement is represented as R, and a PCI alias as (R1 OR R2 OR ...):count. Further assuming two PCI aliases, with the first one having two PCI requirements, with the second having one PCI requirement. Logically speaking, in order to choose a host as candidate, it must satisfy
(R11 OR R12):count AND R21:count

Given a PCI requirement, it's used to match against PCI stats groups until one or multiple matches are found to satisfy the count. Note that matching is based on the keys existing in the PCI requirement. Therefore it's possible for a PCI requirement to be matching multiple PCI stats groups.

==== Motivation for enhancement ====

* PCI whitelist only uses <vendor_id> and <product_id>. This doesn't work if a compute node has multiple vNICs from the same vendor that are not used in the same way.
* PCI requirement only uses <vendor_id> and <product_id>. This doesn't work when the vNIC ports are connected to different physical networks . In other words, PCI devices from the same vendor may not be treated equally when it comes to choose vNIC ports for a VM.

==== What we have discussed so far to satisfy the above requirements ====

===== PCI Group =====

* Refer to https://docs.google.com/document/d/1EMwDg9J8zOxzvTnQJ9HwZdiotaVstFWKIuKrPse6JOs/edit#heading=h.30de7p6sgoxp. The doc becomes messy and almost unreadable. But refer to https://wiki.openstack.org/wiki/Meetings/Passthrough#Agenda_on_Jan._8th.2C_2014 for the main ideas behind it.
* PCI Group: a collection of PCI devices that share the same functions or belong to the same subsystem in a cloud
* PCI whitelist: each entry is defined as a tuple of <device-filter, pci-group-name>. The device-filter uses vendor_id, product_id and PCI addresses to specify a collection of PCI devices.
* PCI stats group: each PCI group is a PCI stats group
* PCI Alias: alias is no longer needed
* Nova Server Flavor: Use G to represent a PCI group. An exampel of PCI requirements can be specified as:
(G1 OR G2):count AND G3:count
It defines that the VM needs "count" of PCI devices that are either from G1 or G2, and 'count" of PCI devices from G3.

It supports only one well-defined tag called PCI group, which is considered too limited by some of the folks.
It simplifies the existing implementation without losing any of its capabilities (or use cases the current implementation can support).

===== PCI Flavor =====

* Refer to https://docs.google.com/document/d/1vadqmurlnlvZ5bv3BlUbFeXRS_wh-dsgi5plSjimWjU/edit

* PCI whitelist: it's called PCI information in the above wiki
* PCI stats group: not clearly defined
* PCI flavor Attributes: refer to the wiki
* PCI flavor API: refer to the wiki

This can be considered a generalized version of PCI Group as it supports arbitrary number of arbitrary tags. But system behaviors (PCI stats groups and scheduling, for example) are not clearly specified. Use cases to justify it is yet to be established.

===== Host Aggregate =====

It was brought up during the discussion that the existing host aggregate may be used to support the new requirement. A couple of issues with host aggregate were discussed:
* it doesn't support dynamic host join to a host aggregate
* scheduling with host aggregate is not stats based.

===== Nic Type/Flavor =====
Refer to http://lists.openstack.org/pipermail/openstack-dev/2014-January/023981.html and http://lists.openstack.org/pipermail/openstack-dev/2013-December/022737.html

===== A compromised implementation =====
to speed up the development and facilitate integration testing with neutron.
Refer to https://blueprints.launchpad.net/nova/+spec/pci-passthrough-sriov. Also refer to the submitted patch.

=== BPs & Patches ===
==== Neutron Side ====
* RKukura (binding:vif_details and binding:profile)
** https://blueprints.launchpad.net/neutron/+spec/vif-details
** https://blueprints.launchpad.net/neutron/+spec/ml2-binding-profile
* Sadasu (Cisco MD)
** https://blueprints.launchpad.net/neutron/+spec/ml2-ucs-manager-mechanism-driver
* Irenab (vnic_type and mlnx MD)
** https://blueprints.launchpad.net/neutron/+spec/ml2-sriov-nic-switch
** https://blueprints.launchpad.net/neutron/+spec/mlnx-ml2-support
** https://blueprints.launchpad.net/neutron/+spec/ml2-request-vnic-type
* Baoli (for anything else that's needed for SRIOV in neutron)
** https://blueprints.launchpad.net/neutron/+spec/pci-passthrough-sriov

==== Nova Side ====
* Baoli
** https://blueprints.launchpad.net/nova/+spec/pci-passthrough-sriov
* Yongli
** https://blueprints.launchpad.net/nova/+spec/pci-extra-info
* Yunhong
** https://blueprints.launchpad.net/nova/+spec/pci-extra-info-icehouse

==== Itzik's patch to add physical network in VIF ====
https://review.openstack.org/#/c/59093/

===Agenda on Feb. 13th, 2014===
* Discuss the Neutron SR-IOV's requirements on the nova generic PCI support, and their availability
** in the PCI information definition:
*** the pci device filter expression (I don't see it's given a name in https://wiki.openstack.org/wiki/PCI_passthrough_SRIOV_support_Icehouse) should support (which may be done as an enhancement).
**** specifying a physical function in the form of domain:bus:slot.func
**** specifying a physical function in the form of a ethernet interface name.
*** tagging with an attribute called "net-group" or simply "physical_network". The tag name "phyical_network" now may make more sense based on the consensus reached on Feb. 12th.
** PCI stats based on the tag "physical_network"
** An API to retrieve a tag's value on a per PCI device basis, something like: get_pci_device_tag_value(pci_dev, tag_name)
** An API to create a PCI request for scheduling purpose
** An API to retrieve a PCI device that is linked to the original PCI request.
** The existing pci_manager.get_instance_pci_devs(instance) shouldn't return PCI devices that are allocated as a result of the aforementioned PCI requests.

===Feb. 12th, 2014 Recap===
Consensus reached during this meeting:
* The binding:profile dictionary will now have these keys defined: 'physical_network', 'pci_vendor_info', 'pci_slot'.
** physical_network: its value is the physical network name that has been chosen for the nic (neutron port) to attach to. ML2 MD will use this information:
*** In the non-SR-IOV case with the agent-based MDs, the ML2 plugin's port binding code iterates over the registered MDs to try to bind, calling bind_port() on each. Within the AgentMechanismDriverBase.bind_port() implementation, it iterates over the network segments, calling check_segment_for_agent() on the derived class for each segment. The 1st segment that is tried for which the agent on the node identified by binding:host_id has a mapping for the segment's physical_network is used.
*** in the non-SR_IOV case, need to add a line of code in the existing agent-based MDs check_segment_for_agent() to make sure vnic_type == 'virtio', so it won't bind when SR-IOV is required. Irenab's vnic BP will take care of this.
*** in the SR-IOV case, the physical network names that a neutron port can potentially attach to will be used for scheduling, and the change to the scheduling filter in the case of multiprovidernet extension is TBD.
*** in the SR-IOV case, SR-IOV MDs will need to iterate over the segments looking for the 1st one that has network_type of 'vlan' and that has the physical_network specified in binding:profile:physical_network; the SR-IOV MDs can include the segment's segmentation_id within the binding:vif_details so that VIF driver can put that into the libvirt XML
** pci_vendor_info: its value is a string with the format "vendor_id:product_id". Both vendor_id and product_id corresponds to the PCI device's vendor_id and product_id
** pci_slot: its value is a string with the format "domain:bus:slot.func" that correspond's to the PCI device's slot as named on a linux system.
*The binding:vif_details will have keys depending on the neutron port's vif type
** profileid: this key will be used to support the vif type VIF_TYPE_802_1QBH
** vlan_id: this key will be used to support the vif type VIF_TYPE_HW_VEB [irenab - neutron uses 'segmentation_id'. Let's pass it in the binding:vif_details]
* interface config and resulting interface XML will be generated based on both vnic_type and vif_type. The vnic_type, if not present as a key in the top level port dictionary, defaults to 'virtio'

===Agenda on Jan 28th, 2014===
[[nova-neutron-sriov]]
* Openstack Icehouse schedule: https://wiki.openstack.org/wiki/Icehouse_Release_Schedule
* Work Items for the initial release:
** Nova: generic PCI-passthrough [Nova folks to add details]
*** pci_information support, add tag/extra information to pci devices.
**** pci_information = { { 'device_id': "8086", 'vendor_id': "000[1-2]" }, { 'e.group' :'gpu' } }
*** pci flavor define attr can be used in the pci flavor and how pci stats report it's pool
**** pci_flavor_attrs = ['e.group']
*** pci schduler to support corresponding extra information.
** Nova: SRIOV
*** Dependencies on Nova Generic PCI-Passthrough
**** Support of PCI attribute sriov_group in the PCI passthrough device list
**** Support of PCI stats based on sriov_group
*** overal change breakdown
**** Nova server and scheduler changes:
***** requested network and SRIOV request spec management
**** Nova compute
***** SRIOV request spec--vif association
***** neutronv2 API in nova that supports interaction with neutron. Particularly speaking:
****** to support the enhanced port binding [irenab: vnic_type, pci slot record]
****** to support an enhanced vif dictionary (e.g., vlan id is missing from the dictionary)
***** libvirt driver to support sriov
***** vif driver to support sriov, especially to generate config for sriov device and to generate interface/network xml
***** to support live migration assuming per interface network xml
** Neutron:
**** enhance neutron port-create to support vnic-type and port profile [ irenab: pci slot record]
**** to support the enhanced port binding and as a result API changes in the main plugin
**** to enhance the neutron client in support of the enhanced interaction between nova and neutron [Might not be needed]
**** to enhance various plugins, especially ml2 plugin to support the enhanced port binding (particularly vnic-type)

===Agenda on Jan 27th, 2014===
* Main focus is on SRIOV
* References:
**https://docs.google.com/document/d/1RfxfXBNB0mD_kH9SamwqPI8ZM-jg797ky_Fze7SakRc/edit?pli=1
**https://docs.google.com/document/d/1vadqmurlnlvZ5bv3BlUbFeXRS_wh-dsgi5plSjimWjU/edit?pli=1#
* Some ideas that came out of discussions
** Assuming there is an attribute (based on pci flavor terminology) for sriov, which name is yet to be determined. Call it sriov_group temporarily for reference in below
** Assuming PCI stats will be supported based on this attribute
** Assuming for each physical network that's supported in the cloud, there is a corresponding sriov_group. To make it simple, say if there are two physical networks: physnet1 and physnet2, then there are two sriov_groups named physnet1 and physnet2
** sriov devices in the pci passthrough device list (the pci whitelist) will be tagged by "sriov_group: physnet1" or "sriov_group: physnet2" based on which physical network the device belongs to.
** Initially, we don't have to change nova CLI/API. Instead, the neutron port-create will be enhanced to support the following new arguments:
*** --vnic-type = “vnic” | “direct” | “macvtap”. OPTIONAL with default as "vnic". The actually names of the three possible vnic types for further discussion
*** port-profile=port-profile-name for support of IEEE802.1BR. OPTIONAL.
*** pci-flavor=pci-flavor-name. Support of this argument is debatable. If sriov devices can be used outside of the physical network, or it's desirable to further partition a physical network, it might be necessary to introduce this argument. In that case, pci flavors need to be explicitly created once it's available. This argument may not be needed initially.
** to create a VM with sriov NICs
*** Configuration on the controller node: pci attribute sriov_group
*** configurations on a compute node: physical networks as required by the neutron plugin, pci passthrough device list that are properly tagged with sriov_group.
*** neutron net-create that may be provided with the provider network name, or otherwise uses default physical network name as configured in the neutron plugin
*** neutron port-create --vnic-type direct <net-uuid>
*** nova boot --flavor m1.large --image <image-uuid> --nic port-id=<port-uuid-from-port-create>

* Possible Work Items
** Existing BPs that are relevant based on the above
*** https://blueprints.launchpad.net/neutron/+spec/ml2-request-vnic-type
*** https://blueprints.launchpad.net/neutron/+spec/pci-passthrough-sriov
*** https://blueprints.launchpad.net/nova/+spec/pci-passthrough-sriov
** The above BPs may be redefined or new BPs can be created based on the actual work items.
** Roughly speaking, there are the following work items
*** nova side: a proof of concept patch is captured in here: https://docs.google.com/document/d/196pcKK0iQBJwQfCP0MRaXndjnO-RGfgW1zfeo1YcY4A/edit?pli=1 and https://review.openstack.org/67500
**** Nova server and scheduler changes:
***** although no changes to --nic options for now, the semantics of the port-id parameter is enhanced to support sriov. Basically, if it's a sriov port, request_specs for scheduling should be created and managed.
***** how do we address this requirement that a compute node may support SRIOV port only, a new filter or enhancement to the existing pci passthrough filter? [irenab - May use host aggregate ]
**** Nova compute
***** Nova compute manager to associate a requested network with a request spec, thus corresponds a requested network with a PCI device that is allocated for it
***** neutronv2 API in nova that supports interaction with neutron. Particularly speaking:
****** to support the enhanced port binding
****** to support an enhanced vif dictionary (e.g., vlan id is missing from the dictionary)
***** libvirt driver to support sriov
***** vif driver to support sriov, especially to generate config for sriov device and to generate interface xml
***** to support live migration assuming per interface network xml
*** neutron side:
**** to support the enhanced port binding in the main plugin
**** to enhance the neutron client in support of the enhanced interaction between nova and neutron [irenab - if we use binding:profile, there should be no changes to existing support]
**** to enhance various plugins, especially ml2 plugin to support the enhanced port binding (particularly vnic-type)

===Agenda on Jan 15th, 2014===
* Ian's proposal for flavor and backend tagging: https://docs.google.com/document/d/1vadqmurlnlvZ5bv3BlUbFeXRS_wh-dsgi5plSjimWjU/edit#
* Review this against known use cases
* Document any use cases not in that document
=== Agenda on Jan. 14th 2014===
* PCI group versus PCI flavor: let's sort out what exactly they are, APIs around them, and pros and cons of each.
* Please check [https://wiki.openstack.org/wiki/Meetings/Passthrough#POC_Implementation]
* Division of works

=== POC Implementation ===
See [https://docs.google.com/document/d/196pcKK0iQBJwQfCP0MRaXndjnO-RGfgW1zfeo1YcY4A/edit?usp=sharing POC implementation]

=== Definitions ===

A specific PCI attachment (could by a virtual function) is described by:
* vendor_id
* product_id
* address

There is a whitelist (at the moment):
* which devices on a specific hypervisor host can be exposed

There is an alias (at the moment):
* groupings of PCI devices

==== The user view of system ====

For GPU passthrough, we need things like:
* user request a "large" GPU
* could be from various vendors or product versions

For network, we need things like:
* user requests a nic for a specific neutron network
* they want to say if it's virtual (the default type) or passthrough (super fast, slow, etc)
* this includes groups by address (virtual function, etc) so it's specific to a particular _group_ of neutron network, each with specific configurations (e.g. VLAN id, a NIC attached to a specific provider network)
* or it involves a NIC that can be programmatically made to attach to a specific neutron network

==== The user view of requesting things ====

For GPU passthrough:
* user requests a flavor extra specs *imply* which possible PCI devices can be connected
* nova boot --image some_image --flavor flavor_that_has_big_GPU_attached some_name

The admin would expose a flavor that gives you, for example, one large GPU and one small GPU:
* nova flavor-key m1.large set "pci_passthrough:alias"=" large_GPU:1,small_GPU:1"
* TODO - this may change in the future

For SRIOV:
* in the most basic case, the user may be given direct access to a network card, just like we do with GPU, but this is less useful than...
* user requests neutron nics, on specific neutron networks, but connected in a specific way (i.e. high speed SRIOV vs virtual)
* note that some of the nics may be virtual, some may be passthrough, and some might be a different type of passthrough
* nova boot --flavor m1.large --image <image_id> --nic net-id=<net-id>,nic-type=<slow | fast | foobar> <vm-name>
* (where slow is a virtual connection, fast is a PCI passthrough, and foobar is some other type of PCI passthrough)
* consider several nics, of different types: nova boot --flavor m1.large --image <image_id> --nic net-id=<net-id-1> --nic net-id=<net-id-2>,nic-type=fast --nic net-id=<net-id-3>,nic-type=faster <vm-name>
* when hot-plugging hot-unplugging, we also need to specify vnic-type in a similar way
* also, this should work nova boot --flavor m1.large --image <image_id> --nic port-id=<port-id>, given
* quantum port-create --fixed-ip subnet_id=<subnet-id>,ip_address=192.168.57.101 <net-id> --nic-type=<slow | fast | foobar>

TODO: need agreement, but one idea for admin...
* pci_alias_1='{"name":"Cisco.VIC", devices:[{"vendor_id":"1137","product_id":"0071", address:"*", "attach-type":"macvtap"}],"nic-type":"fast"}'
* pci_alias_2='{"name":"Fast",devices:[{"vendor_id":"1137","product_id":"0071", address:"*","attach-type":"direct"}, {"vendor_id":"123","product_id":"0081", address:"*","attach-type":"macvtap"}],"nic-type":"faster", }'

==== New Proposal for admin view ====

Whitelist:
* only certain devices exposed to Nova
* just a list of addresses that are allowed (including wildcards)
* by default, nothing is allowed
* this is assumed to be (mostly) static for the lifetime of the machine
* contained in nova.conf

PCI flavors:
* specify groups of PCI devices, to be used in Neutron port descriptions or Server flavor extra specs
* configured using host aggregates API:
** a combination of whitelist, alias and group
** raw device passthrough (grouped by device_id and product_id)
** network device passthrough (grouped by device address also)
** note: there might be several options for each (GPU v3 and GPU v4 in a single flavor)
* only servers in the aggregate will be considered by the scheduler for each PCI flavor
* these are shared across the whole child cell (or if no cells, whole nova deploy)

Scheduler updates:
* on periodic update, report current status of devices
* if any devices are in the whitelist, look up host aggregates to check what device types to report
* report the number of free devices per PCI flavor
* device usage tracked by resource manager as normal, looking at all devices in whitelist

On attach of PCI device:
* scheduler picks host it things has a free device
* check with resource manage in usual way
* assign device to VM
* ignoring migration for now

On attach of VIF device (through boot or otherwise):
* TBD... very sketchy plan...
* ideally neutron port contains associated PCI flavor/alias, or its assumed to be a virtual port
* neutron supplies the usual information, VLAN-id, etc
* neutron and nova negotiate which VIF driver to use, in usual way, given extra info about nic-type from PCI alias settings, etc
* VIF driver given a hypervisor agnostic lib to attach the PCI device, extracted from Nova attach PCI device code
* VIF driver is free to configure the specific PCI device before attaching it using the callback into the Nova driver (or modify Nova code to extend the create API)

=== Agenda on Jan. 8th, 2014 ===

:Let's go over the key concepts and use cases. In the use cases, neutron or neutron plugin specific configurations are not mentioned.

==== Key Concepts ====
'''Note that paragraphs with ''Itablic'' text were added by Iawells as comments.'''
* PCI Groups
:# A PCI group is a collection of PCI devices that share the same functions or belong to the same subsystem in a cloud.

''In fact, two proposals exist for PCI group definition - via API, with the implication that they're stored centrally in the database, and via config, specifically a (match expression -> PCI group name) in the compute node configuration. A competing proposal is PCI aliases, which work on the current assumption that all PCI device data is returned to the database and PCI devices can be selected by doing matching at schedule time and thus a name -> match expression mapping is all that need be saved. Thus the internal question of "should all device information be returned to the controller" drives some of the design options.''

:# it's worth mentioning that using an API to define PCI groups make them owned by the tenant who creates them.

* Pre-defined PCI Groups
:For each PCI device class that openstack supports, a PCI group is defined and associated with the PCI devices belonging to that device class. For example, for the PCI device class ''net'', there is a predefined PCI group named ''net''

* User-defined PCI Groups
:User can define PCI groups using a Nova API.

* PCI Passthrough List (whitelist)
:# Specified on a compute node to define all the PCI passthrough devices and their associated PCI groups that are available on the node.
:# blacklist (exclude list) may be added later if deemed necessary.

* vnic_type:
:# virtio: a virtual port that is attached to a virtual switch
:# direct: SRIOV without macvtap
:# macvtap: SRIOV with macvtap
''This configuration item is not essential to PCI passthrough. It's also a Neutron configuration item.''

* nova boot: new parameters in --nic option
:# vnic-type=“vnic” | “direct” | “macvtap”
:# pci-group=pci-group-name
:# port-profile=port-profile-name ''This property is not related directly to use of PCI passthrough for networks. It is a requirement of 802.1BR-based systems.''

* neutron port-create: new arguments
:# --vnic-type “vnic” | “direct” | “macvtap”
:# --pci-group pci-group-name
:# port-profile port-profile-name

* Nova SRIOV Configuration
:# vnic_type = <vnic-type>: specified in controller node to indicate the default vnic-type that VMs will be booted with. default value is "vnic"
:# sriov_auto_all = <on | off>: specified in compute nodes to indicate that all sriov capable ports are added into the ‘net’ PCI group.
:# sriov_only = <on | off>: specified in compute nodes to indicate that nova can only place VMs with sriov vnics onto these nodes. Default value is on for nodes with SRIOV ports.
:# sriov_pci_group = <pci-group-name>: specified in compute nodes in which all of its SRIOV ports belong to a single pci group.
''The SRIOV cofiguration items are enhancements to the base proposal that make it much easier to configure compute hosts where it is known that all VFs will be available to cloud users.''

==== Use Cases ====
''These use cases do not include non-network passthrough cases.''
* SRIOV based cloud
:# All the compute nodes are identical and all the NICs are SRIOV based
:# All the NICs are connected to the same physical network

:In this cloud, the admin only needs to specify vnic_type=direct on the controller and sriov_auto_all=on on the compute nodes in the nova configuration file. In addition, the new arguments introduced in the nova boot command are not required.

* A cloud with mixed Vnics
:# On compute nodes with sriov ports only, set sriov_auto_all = on
:# On compute nodes without sriov ports, no change is required

:In such a cloud, when booting a VM with sriov vnic, the nova boot command would look like:

nova boot --flavor m1.large --image <image_id>
--nic net-id=<net-id>,vnic-type=direct <vm-name>

:This will require some minimum change in the existing applications.

* A Cloud that requires multiple SRIOV PCI groups
:# create all the pci-groups in the cloud by invoking a Nova API
:# on compute nodes that support a single pci group and in which all of the SRIOV ports belong to this group, set sriov_auto_all=on, sriov_pci_group=<group_name>
:# on compute nodes that support multiple pci groups, define the pci-passthrough-list

:In such a cloud, when booting a VM with sriov macvtap, the nova boot command would look like:
nova boot --flavor m1.large --image <image_id>
--nic net-id=<net-id>,vnic-type=macvtap,pci-group=<group-name> <vm-name>

* Introducing new compute nodes with SRIOV into an existing cloud
:Depending on the cloud and the compute node being introduced:
:# it could be as simple as adding sriov_auto_all=on into the nova config file
:# it could be setting sriov_auto_all=on and pci_group=<group_name>
:# it could be defining the pci-passthrough-list.

* NIC hot plug

=== Evolving Design Doc ===

https://docs.google.com/document/d/1EMwDg9J8zOxzvTnQJ9HwZdiotaVstFWKIuKrPse6JOs/edit?usp=sharing

Ian typed up a complete proposal in two sections in that document, which is pulled out here: https://docs.google.com/document/d/1svN89UXKbFoka0EF6MFUP6OwdNvhY4OkdjEZN-rD-0Q/edit# - this proposal takes the 'PCI groups via compute node config' approach and makes no attempt at proposing APIs.

=== Previous Meetings ===
http://eavesdrop.openstack.org/meetings/pci_passthrough_meeting/2013/pci_passthrough_meeting.2013-12-24-14.02.log.html
http://eavesdrop.openstack.org/meetings/pci_passthrough/

:'''Meeting log on Dec. 17th, 2013'''
[[Meetings/Passthrough/dec-17th-2013.log]]

:'''Meeting log on June. 17th, 2014'''
http://eavesdrop.openstack.org/meetings/passthrough/2014/passthrough.2014-06-17-13.07.log.html

Meetings/Passthrough

2014-06-24T11:41:51Z

Itzikb: /* Agenda on May 26th, 2014 */

Next meeting will be on Wed May 7th at 1300 UTC on #openstack-meeting-alt

=== Agenda on Jun 24th, 2014 ===

=== Agenda on May 26th, 2014 ===
* Nova SR-IOV networking
** [https://review.openstack.org/#/c/86606/ nova spec review]
** patch submission soon
** tempest tests
* Bugs
** [https://review.openstack.org/#/c/82206/ Init PCI manager after compute node initialization]
** [https://review.openstack.org/#/c/81954/ makes sure correct PCI device allocation]
* Other works/features
** Live migration
** Attach SR-IOV port to an existing instance
** possibly changing the names of existing vnic-types
** associate a default vnic-type with a neutron network
** admin controlled knobs
** HA
** others

=== Agenda on March 25th, 2014 ===
* PCI SR-IOV Networking use case
Refer to https://docs.google.com/document/d/1zgMaXqrCnad01-jQH7Mkmf6amlghw9RMScGLBrKslmw/edit

=== Recap of Discussions ===
==== Current PCI Passthrough ====

* How to Use Wiki: https://wiki.openstack.org/wiki/Pci_passthrough

* PCI whitelist: defines all the PCI passthrough devices that are available on a compute node. It's currently based on <vendor_id> <product_id>
* PCI stats group: defines the keys based on which a PCI device is accounted. It's currently based on the keys: <vendor_id> <product_id> <extra_info or simply PCI address of a physical function>.
* PCI Alias: specifies a list of PCI device requirements. A PCI requirement specifies a dictionary with keys: <alias name> <vendor_id> <product_id> <device_type>. Multiple requirements can be specified per PCI Alias with the same <device_type> (which may not work yet due to bugs). <alias name> is required. Others are optional.
* Nova Server Flavor: PCI requirements can be added into a nova flavor as it's extra-spec in the syntax "pci_passthrough:alias"="<PCI Alias Name>:<count>{,<PCI Alias Name>:<count>}"
* PCI Passthrough filter for nova scheduler: this filter works based on the PCI stats group and PCI aliases referenced in the nova flavor. If mutliple aliases exist in the flavor, all of the them have to be satified. If multiple PCI requirements exist in one PCI alias, only one of the requirements defined in the alias needs to be satisfied. Suppose a PCI requirement is represented as R, and a PCI alias as (R1 OR R2 OR ...):count. Further assuming two PCI aliases, with the first one having two PCI requirements, with the second having one PCI requirement. Logically speaking, in order to choose a host as candidate, it must satisfy
(R11 OR R12):count AND R21:count

Given a PCI requirement, it's used to match against PCI stats groups until one or multiple matches are found to satisfy the count. Note that matching is based on the keys existing in the PCI requirement. Therefore it's possible for a PCI requirement to be matching multiple PCI stats groups.

==== Motivation for enhancement ====

* PCI whitelist only uses <vendor_id> and <product_id>. This doesn't work if a compute node has multiple vNICs from the same vendor that are not used in the same way.
* PCI requirement only uses <vendor_id> and <product_id>. This doesn't work when the vNIC ports are connected to different physical networks . In other words, PCI devices from the same vendor may not be treated equally when it comes to choose vNIC ports for a VM.

==== What we have discussed so far to satisfy the above requirements ====

===== PCI Group =====

* Refer to https://docs.google.com/document/d/1EMwDg9J8zOxzvTnQJ9HwZdiotaVstFWKIuKrPse6JOs/edit#heading=h.30de7p6sgoxp. The doc becomes messy and almost unreadable. But refer to https://wiki.openstack.org/wiki/Meetings/Passthrough#Agenda_on_Jan._8th.2C_2014 for the main ideas behind it.
* PCI Group: a collection of PCI devices that share the same functions or belong to the same subsystem in a cloud
* PCI whitelist: each entry is defined as a tuple of <device-filter, pci-group-name>. The device-filter uses vendor_id, product_id and PCI addresses to specify a collection of PCI devices.
* PCI stats group: each PCI group is a PCI stats group
* PCI Alias: alias is no longer needed
* Nova Server Flavor: Use G to represent a PCI group. An exampel of PCI requirements can be specified as:
(G1 OR G2):count AND G3:count
It defines that the VM needs "count" of PCI devices that are either from G1 or G2, and 'count" of PCI devices from G3.

It supports only one well-defined tag called PCI group, which is considered too limited by some of the folks.
It simplifies the existing implementation without losing any of its capabilities (or use cases the current implementation can support).

===== PCI Flavor =====

* Refer to https://docs.google.com/document/d/1vadqmurlnlvZ5bv3BlUbFeXRS_wh-dsgi5plSjimWjU/edit

* PCI whitelist: it's called PCI information in the above wiki
* PCI stats group: not clearly defined
* PCI flavor Attributes: refer to the wiki
* PCI flavor API: refer to the wiki

This can be considered a generalized version of PCI Group as it supports arbitrary number of arbitrary tags. But system behaviors (PCI stats groups and scheduling, for example) are not clearly specified. Use cases to justify it is yet to be established.

===== Host Aggregate =====

It was brought up during the discussion that the existing host aggregate may be used to support the new requirement. A couple of issues with host aggregate were discussed:
* it doesn't support dynamic host join to a host aggregate
* scheduling with host aggregate is not stats based.

===== Nic Type/Flavor =====
Refer to http://lists.openstack.org/pipermail/openstack-dev/2014-January/023981.html and http://lists.openstack.org/pipermail/openstack-dev/2013-December/022737.html

===== A compromised implementation =====
to speed up the development and facilitate integration testing with neutron.
Refer to https://blueprints.launchpad.net/nova/+spec/pci-passthrough-sriov. Also refer to the submitted patch.

=== BPs & Patches ===
==== Neutron Side ====
* RKukura (binding:vif_details and binding:profile)
** https://blueprints.launchpad.net/neutron/+spec/vif-details
** https://blueprints.launchpad.net/neutron/+spec/ml2-binding-profile
* Sadasu (Cisco MD)
** https://blueprints.launchpad.net/neutron/+spec/ml2-ucs-manager-mechanism-driver
* Irenab (vnic_type and mlnx MD)
** https://blueprints.launchpad.net/neutron/+spec/ml2-sriov-nic-switch
** https://blueprints.launchpad.net/neutron/+spec/mlnx-ml2-support
** https://blueprints.launchpad.net/neutron/+spec/ml2-request-vnic-type
* Baoli (for anything else that's needed for SRIOV in neutron)
** https://blueprints.launchpad.net/neutron/+spec/pci-passthrough-sriov

==== Nova Side ====
* Baoli
** https://blueprints.launchpad.net/nova/+spec/pci-passthrough-sriov
* Yongli
** https://blueprints.launchpad.net/nova/+spec/pci-extra-info
* Yunhong
** https://blueprints.launchpad.net/nova/+spec/pci-extra-info-icehouse

==== Itzik's patch to add physical network in VIF ====
https://review.openstack.org/#/c/59093/

===Agenda on Feb. 13th, 2014===
* Discuss the Neutron SR-IOV's requirements on the nova generic PCI support, and their availability
** in the PCI information definition:
*** the pci device filter expression (I don't see it's given a name in https://wiki.openstack.org/wiki/PCI_passthrough_SRIOV_support_Icehouse) should support (which may be done as an enhancement).
**** specifying a physical function in the form of domain:bus:slot.func
**** specifying a physical function in the form of a ethernet interface name.
*** tagging with an attribute called "net-group" or simply "physical_network". The tag name "phyical_network" now may make more sense based on the consensus reached on Feb. 12th.
** PCI stats based on the tag "physical_network"
** An API to retrieve a tag's value on a per PCI device basis, something like: get_pci_device_tag_value(pci_dev, tag_name)
** An API to create a PCI request for scheduling purpose
** An API to retrieve a PCI device that is linked to the original PCI request.
** The existing pci_manager.get_instance_pci_devs(instance) shouldn't return PCI devices that are allocated as a result of the aforementioned PCI requests.

===Feb. 12th, 2014 Recap===
Consensus reached during this meeting:
* The binding:profile dictionary will now have these keys defined: 'physical_network', 'pci_vendor_info', 'pci_slot'.
** physical_network: its value is the physical network name that has been chosen for the nic (neutron port) to attach to. ML2 MD will use this information:
*** In the non-SR-IOV case with the agent-based MDs, the ML2 plugin's port binding code iterates over the registered MDs to try to bind, calling bind_port() on each. Within the AgentMechanismDriverBase.bind_port() implementation, it iterates over the network segments, calling check_segment_for_agent() on the derived class for each segment. The 1st segment that is tried for which the agent on the node identified by binding:host_id has a mapping for the segment's physical_network is used.
*** in the non-SR_IOV case, need to add a line of code in the existing agent-based MDs check_segment_for_agent() to make sure vnic_type == 'virtio', so it won't bind when SR-IOV is required. Irenab's vnic BP will take care of this.
*** in the SR-IOV case, the physical network names that a neutron port can potentially attach to will be used for scheduling, and the change to the scheduling filter in the case of multiprovidernet extension is TBD.
*** in the SR-IOV case, SR-IOV MDs will need to iterate over the segments looking for the 1st one that has network_type of 'vlan' and that has the physical_network specified in binding:profile:physical_network; the SR-IOV MDs can include the segment's segmentation_id within the binding:vif_details so that VIF driver can put that into the libvirt XML
** pci_vendor_info: its value is a string with the format "vendor_id:product_id". Both vendor_id and product_id corresponds to the PCI device's vendor_id and product_id
** pci_slot: its value is a string with the format "domain:bus:slot.func" that correspond's to the PCI device's slot as named on a linux system.
*The binding:vif_details will have keys depending on the neutron port's vif type
** profileid: this key will be used to support the vif type VIF_TYPE_802_1QBH
** vlan_id: this key will be used to support the vif type VIF_TYPE_HW_VEB [irenab - neutron uses 'segmentation_id'. Let's pass it in the binding:vif_details]
* interface config and resulting interface XML will be generated based on both vnic_type and vif_type. The vnic_type, if not present as a key in the top level port dictionary, defaults to 'virtio'

===Agenda on Jan 28th, 2014===
[[nova-neutron-sriov]]
* Openstack Icehouse schedule: https://wiki.openstack.org/wiki/Icehouse_Release_Schedule
* Work Items for the initial release:
** Nova: generic PCI-passthrough [Nova folks to add details]
*** pci_information support, add tag/extra information to pci devices.
**** pci_information = { { 'device_id': "8086", 'vendor_id': "000[1-2]" }, { 'e.group' :'gpu' } }
*** pci flavor define attr can be used in the pci flavor and how pci stats report it's pool
**** pci_flavor_attrs = ['e.group']
*** pci schduler to support corresponding extra information.
** Nova: SRIOV
*** Dependencies on Nova Generic PCI-Passthrough
**** Support of PCI attribute sriov_group in the PCI passthrough device list
**** Support of PCI stats based on sriov_group
*** overal change breakdown
**** Nova server and scheduler changes:
***** requested network and SRIOV request spec management
**** Nova compute
***** SRIOV request spec--vif association
***** neutronv2 API in nova that supports interaction with neutron. Particularly speaking:
****** to support the enhanced port binding [irenab: vnic_type, pci slot record]
****** to support an enhanced vif dictionary (e.g., vlan id is missing from the dictionary)
***** libvirt driver to support sriov
***** vif driver to support sriov, especially to generate config for sriov device and to generate interface/network xml
***** to support live migration assuming per interface network xml
** Neutron:
**** enhance neutron port-create to support vnic-type and port profile [ irenab: pci slot record]
**** to support the enhanced port binding and as a result API changes in the main plugin
**** to enhance the neutron client in support of the enhanced interaction between nova and neutron [Might not be needed]
**** to enhance various plugins, especially ml2 plugin to support the enhanced port binding (particularly vnic-type)

===Agenda on Jan 27th, 2014===
* Main focus is on SRIOV
* References:
**https://docs.google.com/document/d/1RfxfXBNB0mD_kH9SamwqPI8ZM-jg797ky_Fze7SakRc/edit?pli=1
**https://docs.google.com/document/d/1vadqmurlnlvZ5bv3BlUbFeXRS_wh-dsgi5plSjimWjU/edit?pli=1#
* Some ideas that came out of discussions
** Assuming there is an attribute (based on pci flavor terminology) for sriov, which name is yet to be determined. Call it sriov_group temporarily for reference in below
** Assuming PCI stats will be supported based on this attribute
** Assuming for each physical network that's supported in the cloud, there is a corresponding sriov_group. To make it simple, say if there are two physical networks: physnet1 and physnet2, then there are two sriov_groups named physnet1 and physnet2
** sriov devices in the pci passthrough device list (the pci whitelist) will be tagged by "sriov_group: physnet1" or "sriov_group: physnet2" based on which physical network the device belongs to.
** Initially, we don't have to change nova CLI/API. Instead, the neutron port-create will be enhanced to support the following new arguments:
*** --vnic-type = “vnic” | “direct” | “macvtap”. OPTIONAL with default as "vnic". The actually names of the three possible vnic types for further discussion
*** port-profile=port-profile-name for support of IEEE802.1BR. OPTIONAL.
*** pci-flavor=pci-flavor-name. Support of this argument is debatable. If sriov devices can be used outside of the physical network, or it's desirable to further partition a physical network, it might be necessary to introduce this argument. In that case, pci flavors need to be explicitly created once it's available. This argument may not be needed initially.
** to create a VM with sriov NICs
*** Configuration on the controller node: pci attribute sriov_group
*** configurations on a compute node: physical networks as required by the neutron plugin, pci passthrough device list that are properly tagged with sriov_group.
*** neutron net-create that may be provided with the provider network name, or otherwise uses default physical network name as configured in the neutron plugin
*** neutron port-create --vnic-type direct <net-uuid>
*** nova boot --flavor m1.large --image <image-uuid> --nic port-id=<port-uuid-from-port-create>

* Possible Work Items
** Existing BPs that are relevant based on the above
*** https://blueprints.launchpad.net/neutron/+spec/ml2-request-vnic-type
*** https://blueprints.launchpad.net/neutron/+spec/pci-passthrough-sriov
*** https://blueprints.launchpad.net/nova/+spec/pci-passthrough-sriov
** The above BPs may be redefined or new BPs can be created based on the actual work items.
** Roughly speaking, there are the following work items
*** nova side: a proof of concept patch is captured in here: https://docs.google.com/document/d/196pcKK0iQBJwQfCP0MRaXndjnO-RGfgW1zfeo1YcY4A/edit?pli=1 and https://review.openstack.org/67500
**** Nova server and scheduler changes:
***** although no changes to --nic options for now, the semantics of the port-id parameter is enhanced to support sriov. Basically, if it's a sriov port, request_specs for scheduling should be created and managed.
***** how do we address this requirement that a compute node may support SRIOV port only, a new filter or enhancement to the existing pci passthrough filter? [irenab - May use host aggregate ]
**** Nova compute
***** Nova compute manager to associate a requested network with a request spec, thus corresponds a requested network with a PCI device that is allocated for it
***** neutronv2 API in nova that supports interaction with neutron. Particularly speaking:
****** to support the enhanced port binding
****** to support an enhanced vif dictionary (e.g., vlan id is missing from the dictionary)
***** libvirt driver to support sriov
***** vif driver to support sriov, especially to generate config for sriov device and to generate interface xml
***** to support live migration assuming per interface network xml
*** neutron side:
**** to support the enhanced port binding in the main plugin
**** to enhance the neutron client in support of the enhanced interaction between nova and neutron [irenab - if we use binding:profile, there should be no changes to existing support]
**** to enhance various plugins, especially ml2 plugin to support the enhanced port binding (particularly vnic-type)

===Agenda on Jan 15th, 2014===
* Ian's proposal for flavor and backend tagging: https://docs.google.com/document/d/1vadqmurlnlvZ5bv3BlUbFeXRS_wh-dsgi5plSjimWjU/edit#
* Review this against known use cases
* Document any use cases not in that document
=== Agenda on Jan. 14th 2014===
* PCI group versus PCI flavor: let's sort out what exactly they are, APIs around them, and pros and cons of each.
* Please check [https://wiki.openstack.org/wiki/Meetings/Passthrough#POC_Implementation]
* Division of works

=== POC Implementation ===
See [https://docs.google.com/document/d/196pcKK0iQBJwQfCP0MRaXndjnO-RGfgW1zfeo1YcY4A/edit?usp=sharing POC implementation]

=== Definitions ===

A specific PCI attachment (could by a virtual function) is described by:
* vendor_id
* product_id
* address

There is a whitelist (at the moment):
* which devices on a specific hypervisor host can be exposed

There is an alias (at the moment):
* groupings of PCI devices

==== The user view of system ====

For GPU passthrough, we need things like:
* user request a "large" GPU
* could be from various vendors or product versions

For network, we need things like:
* user requests a nic for a specific neutron network
* they want to say if it's virtual (the default type) or passthrough (super fast, slow, etc)
* this includes groups by address (virtual function, etc) so it's specific to a particular _group_ of neutron network, each with specific configurations (e.g. VLAN id, a NIC attached to a specific provider network)
* or it involves a NIC that can be programmatically made to attach to a specific neutron network

==== The user view of requesting things ====

For GPU passthrough:
* user requests a flavor extra specs *imply* which possible PCI devices can be connected
* nova boot --image some_image --flavor flavor_that_has_big_GPU_attached some_name

The admin would expose a flavor that gives you, for example, one large GPU and one small GPU:
* nova flavor-key m1.large set "pci_passthrough:alias"=" large_GPU:1,small_GPU:1"
* TODO - this may change in the future

For SRIOV:
* in the most basic case, the user may be given direct access to a network card, just like we do with GPU, but this is less useful than...
* user requests neutron nics, on specific neutron networks, but connected in a specific way (i.e. high speed SRIOV vs virtual)
* note that some of the nics may be virtual, some may be passthrough, and some might be a different type of passthrough
* nova boot --flavor m1.large --image <image_id> --nic net-id=<net-id>,nic-type=<slow | fast | foobar> <vm-name>
* (where slow is a virtual connection, fast is a PCI passthrough, and foobar is some other type of PCI passthrough)
* consider several nics, of different types: nova boot --flavor m1.large --image <image_id> --nic net-id=<net-id-1> --nic net-id=<net-id-2>,nic-type=fast --nic net-id=<net-id-3>,nic-type=faster <vm-name>
* when hot-plugging hot-unplugging, we also need to specify vnic-type in a similar way
* also, this should work nova boot --flavor m1.large --image <image_id> --nic port-id=<port-id>, given
* quantum port-create --fixed-ip subnet_id=<subnet-id>,ip_address=192.168.57.101 <net-id> --nic-type=<slow | fast | foobar>

TODO: need agreement, but one idea for admin...
* pci_alias_1='{"name":"Cisco.VIC", devices:[{"vendor_id":"1137","product_id":"0071", address:"*", "attach-type":"macvtap"}],"nic-type":"fast"}'
* pci_alias_2='{"name":"Fast",devices:[{"vendor_id":"1137","product_id":"0071", address:"*","attach-type":"direct"}, {"vendor_id":"123","product_id":"0081", address:"*","attach-type":"macvtap"}],"nic-type":"faster", }'

==== New Proposal for admin view ====

Whitelist:
* only certain devices exposed to Nova
* just a list of addresses that are allowed (including wildcards)
* by default, nothing is allowed
* this is assumed to be (mostly) static for the lifetime of the machine
* contained in nova.conf

PCI flavors:
* specify groups of PCI devices, to be used in Neutron port descriptions or Server flavor extra specs
* configured using host aggregates API:
** a combination of whitelist, alias and group
** raw device passthrough (grouped by device_id and product_id)
** network device passthrough (grouped by device address also)
** note: there might be several options for each (GPU v3 and GPU v4 in a single flavor)
* only servers in the aggregate will be considered by the scheduler for each PCI flavor
* these are shared across the whole child cell (or if no cells, whole nova deploy)

Scheduler updates:
* on periodic update, report current status of devices
* if any devices are in the whitelist, look up host aggregates to check what device types to report
* report the number of free devices per PCI flavor
* device usage tracked by resource manager as normal, looking at all devices in whitelist

On attach of PCI device:
* scheduler picks host it things has a free device
* check with resource manage in usual way
* assign device to VM
* ignoring migration for now

On attach of VIF device (through boot or otherwise):
* TBD... very sketchy plan...
* ideally neutron port contains associated PCI flavor/alias, or its assumed to be a virtual port
* neutron supplies the usual information, VLAN-id, etc
* neutron and nova negotiate which VIF driver to use, in usual way, given extra info about nic-type from PCI alias settings, etc
* VIF driver given a hypervisor agnostic lib to attach the PCI device, extracted from Nova attach PCI device code
* VIF driver is free to configure the specific PCI device before attaching it using the callback into the Nova driver (or modify Nova code to extend the create API)

=== Agenda on Jan. 8th, 2014 ===

:Let's go over the key concepts and use cases. In the use cases, neutron or neutron plugin specific configurations are not mentioned.

==== Key Concepts ====
'''Note that paragraphs with ''Itablic'' text were added by Iawells as comments.'''
* PCI Groups
:# A PCI group is a collection of PCI devices that share the same functions or belong to the same subsystem in a cloud.

''In fact, two proposals exist for PCI group definition - via API, with the implication that they're stored centrally in the database, and via config, specifically a (match expression -> PCI group name) in the compute node configuration. A competing proposal is PCI aliases, which work on the current assumption that all PCI device data is returned to the database and PCI devices can be selected by doing matching at schedule time and thus a name -> match expression mapping is all that need be saved. Thus the internal question of "should all device information be returned to the controller" drives some of the design options.''

:# it's worth mentioning that using an API to define PCI groups make them owned by the tenant who creates them.

* Pre-defined PCI Groups
:For each PCI device class that openstack supports, a PCI group is defined and associated with the PCI devices belonging to that device class. For example, for the PCI device class ''net'', there is a predefined PCI group named ''net''

* User-defined PCI Groups
:User can define PCI groups using a Nova API.

* PCI Passthrough List (whitelist)
:# Specified on a compute node to define all the PCI passthrough devices and their associated PCI groups that are available on the node.
:# blacklist (exclude list) may be added later if deemed necessary.

* vnic_type:
:# virtio: a virtual port that is attached to a virtual switch
:# direct: SRIOV without macvtap
:# macvtap: SRIOV with macvtap
''This configuration item is not essential to PCI passthrough. It's also a Neutron configuration item.''

* nova boot: new parameters in --nic option
:# vnic-type=“vnic” | “direct” | “macvtap”
:# pci-group=pci-group-name
:# port-profile=port-profile-name ''This property is not related directly to use of PCI passthrough for networks. It is a requirement of 802.1BR-based systems.''

* neutron port-create: new arguments
:# --vnic-type “vnic” | “direct” | “macvtap”
:# --pci-group pci-group-name
:# port-profile port-profile-name

* Nova SRIOV Configuration
:# vnic_type = <vnic-type>: specified in controller node to indicate the default vnic-type that VMs will be booted with. default value is "vnic"
:# sriov_auto_all = <on | off>: specified in compute nodes to indicate that all sriov capable ports are added into the ‘net’ PCI group.
:# sriov_only = <on | off>: specified in compute nodes to indicate that nova can only place VMs with sriov vnics onto these nodes. Default value is on for nodes with SRIOV ports.
:# sriov_pci_group = <pci-group-name>: specified in compute nodes in which all of its SRIOV ports belong to a single pci group.
''The SRIOV cofiguration items are enhancements to the base proposal that make it much easier to configure compute hosts where it is known that all VFs will be available to cloud users.''

==== Use Cases ====
''These use cases do not include non-network passthrough cases.''
* SRIOV based cloud
:# All the compute nodes are identical and all the NICs are SRIOV based
:# All the NICs are connected to the same physical network

:In this cloud, the admin only needs to specify vnic_type=direct on the controller and sriov_auto_all=on on the compute nodes in the nova configuration file. In addition, the new arguments introduced in the nova boot command are not required.

* A cloud with mixed Vnics
:# On compute nodes with sriov ports only, set sriov_auto_all = on
:# On compute nodes without sriov ports, no change is required

:In such a cloud, when booting a VM with sriov vnic, the nova boot command would look like:

nova boot --flavor m1.large --image <image_id>
--nic net-id=<net-id>,vnic-type=direct <vm-name>

:This will require some minimum change in the existing applications.

* A Cloud that requires multiple SRIOV PCI groups
:# create all the pci-groups in the cloud by invoking a Nova API
:# on compute nodes that support a single pci group and in which all of the SRIOV ports belong to this group, set sriov_auto_all=on, sriov_pci_group=<group_name>
:# on compute nodes that support multiple pci groups, define the pci-passthrough-list

:In such a cloud, when booting a VM with sriov macvtap, the nova boot command would look like:
nova boot --flavor m1.large --image <image_id>
--nic net-id=<net-id>,vnic-type=macvtap,pci-group=<group-name> <vm-name>

* Introducing new compute nodes with SRIOV into an existing cloud
:Depending on the cloud and the compute node being introduced:
:# it could be as simple as adding sriov_auto_all=on into the nova config file
:# it could be setting sriov_auto_all=on and pci_group=<group_name>
:# it could be defining the pci-passthrough-list.

* NIC hot plug

=== Evolving Design Doc ===

https://docs.google.com/document/d/1EMwDg9J8zOxzvTnQJ9HwZdiotaVstFWKIuKrPse6JOs/edit?usp=sharing

Ian typed up a complete proposal in two sections in that document, which is pulled out here: https://docs.google.com/document/d/1svN89UXKbFoka0EF6MFUP6OwdNvhY4OkdjEZN-rD-0Q/edit# - this proposal takes the 'PCI groups via compute node config' approach and makes no attempt at proposing APIs.

=== Previous Meetings ===
http://eavesdrop.openstack.org/meetings/pci_passthrough_meeting/2013/pci_passthrough_meeting.2013-12-24-14.02.log.html
http://eavesdrop.openstack.org/meetings/pci_passthrough/

:'''Meeting log on Dec. 17th, 2013'''
[[Meetings/Passthrough/dec-17th-2013.log]]

:'''Meeting log on June. 17th, 2014'''
http://eavesdrop.openstack.org/meetings/passthrough/2014/passthrough.2014-06-17-13.07.log.html

Meetings/Passthrough

2014-06-24T11:32:13Z

Itzikb: /* Agenda on May 26th, 2014 */

Next meeting will be on Wed May 7th at 1300 UTC on #openstack-meeting-alt

=== Agenda on May 26th, 2014 ===
* Nova SR-IOV networking
** [https://review.openstack.org/#/c/86606/ nova spec review]
** patch submission soon
** tempest tests
* Bugs
** [https://review.openstack.org/#/c/82206/ Init PCI manager after compute node initialization]
** [https://review.openstack.org/#/c/81954/ makes sure correct PCI device allocation]
* Other works/features
** Live migration
** Attach SR-IOV port to an existing instance
** possibly changing the names of existing vnic-types
** associate a default vnic-type with a neutron network
** admin controlled knobs
** HA
** others

=== Agenda on March 25th, 2014 ===
* PCI SR-IOV Networking use case
Refer to https://docs.google.com/document/d/1zgMaXqrCnad01-jQH7Mkmf6amlghw9RMScGLBrKslmw/edit

=== Recap of Discussions ===
==== Current PCI Passthrough ====

* How to Use Wiki: https://wiki.openstack.org/wiki/Pci_passthrough

* PCI whitelist: defines all the PCI passthrough devices that are available on a compute node. It's currently based on <vendor_id> <product_id>
* PCI stats group: defines the keys based on which a PCI device is accounted. It's currently based on the keys: <vendor_id> <product_id> <extra_info or simply PCI address of a physical function>.
* PCI Alias: specifies a list of PCI device requirements. A PCI requirement specifies a dictionary with keys: <alias name> <vendor_id> <product_id> <device_type>. Multiple requirements can be specified per PCI Alias with the same <device_type> (which may not work yet due to bugs). <alias name> is required. Others are optional.
* Nova Server Flavor: PCI requirements can be added into a nova flavor as it's extra-spec in the syntax "pci_passthrough:alias"="<PCI Alias Name>:<count>{,<PCI Alias Name>:<count>}"
* PCI Passthrough filter for nova scheduler: this filter works based on the PCI stats group and PCI aliases referenced in the nova flavor. If mutliple aliases exist in the flavor, all of the them have to be satified. If multiple PCI requirements exist in one PCI alias, only one of the requirements defined in the alias needs to be satisfied. Suppose a PCI requirement is represented as R, and a PCI alias as (R1 OR R2 OR ...):count. Further assuming two PCI aliases, with the first one having two PCI requirements, with the second having one PCI requirement. Logically speaking, in order to choose a host as candidate, it must satisfy
(R11 OR R12):count AND R21:count

Given a PCI requirement, it's used to match against PCI stats groups until one or multiple matches are found to satisfy the count. Note that matching is based on the keys existing in the PCI requirement. Therefore it's possible for a PCI requirement to be matching multiple PCI stats groups.

==== Motivation for enhancement ====

* PCI whitelist only uses <vendor_id> and <product_id>. This doesn't work if a compute node has multiple vNICs from the same vendor that are not used in the same way.
* PCI requirement only uses <vendor_id> and <product_id>. This doesn't work when the vNIC ports are connected to different physical networks . In other words, PCI devices from the same vendor may not be treated equally when it comes to choose vNIC ports for a VM.

==== What we have discussed so far to satisfy the above requirements ====

===== PCI Group =====

* Refer to https://docs.google.com/document/d/1EMwDg9J8zOxzvTnQJ9HwZdiotaVstFWKIuKrPse6JOs/edit#heading=h.30de7p6sgoxp. The doc becomes messy and almost unreadable. But refer to https://wiki.openstack.org/wiki/Meetings/Passthrough#Agenda_on_Jan._8th.2C_2014 for the main ideas behind it.
* PCI Group: a collection of PCI devices that share the same functions or belong to the same subsystem in a cloud
* PCI whitelist: each entry is defined as a tuple of <device-filter, pci-group-name>. The device-filter uses vendor_id, product_id and PCI addresses to specify a collection of PCI devices.
* PCI stats group: each PCI group is a PCI stats group
* PCI Alias: alias is no longer needed
* Nova Server Flavor: Use G to represent a PCI group. An exampel of PCI requirements can be specified as:
(G1 OR G2):count AND G3:count
It defines that the VM needs "count" of PCI devices that are either from G1 or G2, and 'count" of PCI devices from G3.

It supports only one well-defined tag called PCI group, which is considered too limited by some of the folks.
It simplifies the existing implementation without losing any of its capabilities (or use cases the current implementation can support).

===== PCI Flavor =====

* Refer to https://docs.google.com/document/d/1vadqmurlnlvZ5bv3BlUbFeXRS_wh-dsgi5plSjimWjU/edit

* PCI whitelist: it's called PCI information in the above wiki
* PCI stats group: not clearly defined
* PCI flavor Attributes: refer to the wiki
* PCI flavor API: refer to the wiki

This can be considered a generalized version of PCI Group as it supports arbitrary number of arbitrary tags. But system behaviors (PCI stats groups and scheduling, for example) are not clearly specified. Use cases to justify it is yet to be established.

===== Host Aggregate =====

It was brought up during the discussion that the existing host aggregate may be used to support the new requirement. A couple of issues with host aggregate were discussed:
* it doesn't support dynamic host join to a host aggregate
* scheduling with host aggregate is not stats based.

===== Nic Type/Flavor =====
Refer to http://lists.openstack.org/pipermail/openstack-dev/2014-January/023981.html and http://lists.openstack.org/pipermail/openstack-dev/2013-December/022737.html

===== A compromised implementation =====
to speed up the development and facilitate integration testing with neutron.
Refer to https://blueprints.launchpad.net/nova/+spec/pci-passthrough-sriov. Also refer to the submitted patch.

=== BPs & Patches ===
==== Neutron Side ====
* RKukura (binding:vif_details and binding:profile)
** https://blueprints.launchpad.net/neutron/+spec/vif-details
** https://blueprints.launchpad.net/neutron/+spec/ml2-binding-profile
* Sadasu (Cisco MD)
** https://blueprints.launchpad.net/neutron/+spec/ml2-ucs-manager-mechanism-driver
* Irenab (vnic_type and mlnx MD)
** https://blueprints.launchpad.net/neutron/+spec/ml2-sriov-nic-switch
** https://blueprints.launchpad.net/neutron/+spec/mlnx-ml2-support
** https://blueprints.launchpad.net/neutron/+spec/ml2-request-vnic-type
* Baoli (for anything else that's needed for SRIOV in neutron)
** https://blueprints.launchpad.net/neutron/+spec/pci-passthrough-sriov

==== Nova Side ====
* Baoli
** https://blueprints.launchpad.net/nova/+spec/pci-passthrough-sriov
* Yongli
** https://blueprints.launchpad.net/nova/+spec/pci-extra-info
* Yunhong
** https://blueprints.launchpad.net/nova/+spec/pci-extra-info-icehouse

==== Itzik's patch to add physical network in VIF ====
https://review.openstack.org/#/c/59093/

===Agenda on Feb. 13th, 2014===
* Discuss the Neutron SR-IOV's requirements on the nova generic PCI support, and their availability
** in the PCI information definition:
*** the pci device filter expression (I don't see it's given a name in https://wiki.openstack.org/wiki/PCI_passthrough_SRIOV_support_Icehouse) should support (which may be done as an enhancement).
**** specifying a physical function in the form of domain:bus:slot.func
**** specifying a physical function in the form of a ethernet interface name.
*** tagging with an attribute called "net-group" or simply "physical_network". The tag name "phyical_network" now may make more sense based on the consensus reached on Feb. 12th.
** PCI stats based on the tag "physical_network"
** An API to retrieve a tag's value on a per PCI device basis, something like: get_pci_device_tag_value(pci_dev, tag_name)
** An API to create a PCI request for scheduling purpose
** An API to retrieve a PCI device that is linked to the original PCI request.
** The existing pci_manager.get_instance_pci_devs(instance) shouldn't return PCI devices that are allocated as a result of the aforementioned PCI requests.

===Feb. 12th, 2014 Recap===
Consensus reached during this meeting:
* The binding:profile dictionary will now have these keys defined: 'physical_network', 'pci_vendor_info', 'pci_slot'.
** physical_network: its value is the physical network name that has been chosen for the nic (neutron port) to attach to. ML2 MD will use this information:
*** In the non-SR-IOV case with the agent-based MDs, the ML2 plugin's port binding code iterates over the registered MDs to try to bind, calling bind_port() on each. Within the AgentMechanismDriverBase.bind_port() implementation, it iterates over the network segments, calling check_segment_for_agent() on the derived class for each segment. The 1st segment that is tried for which the agent on the node identified by binding:host_id has a mapping for the segment's physical_network is used.
*** in the non-SR_IOV case, need to add a line of code in the existing agent-based MDs check_segment_for_agent() to make sure vnic_type == 'virtio', so it won't bind when SR-IOV is required. Irenab's vnic BP will take care of this.
*** in the SR-IOV case, the physical network names that a neutron port can potentially attach to will be used for scheduling, and the change to the scheduling filter in the case of multiprovidernet extension is TBD.
*** in the SR-IOV case, SR-IOV MDs will need to iterate over the segments looking for the 1st one that has network_type of 'vlan' and that has the physical_network specified in binding:profile:physical_network; the SR-IOV MDs can include the segment's segmentation_id within the binding:vif_details so that VIF driver can put that into the libvirt XML
** pci_vendor_info: its value is a string with the format "vendor_id:product_id". Both vendor_id and product_id corresponds to the PCI device's vendor_id and product_id
** pci_slot: its value is a string with the format "domain:bus:slot.func" that correspond's to the PCI device's slot as named on a linux system.
*The binding:vif_details will have keys depending on the neutron port's vif type
** profileid: this key will be used to support the vif type VIF_TYPE_802_1QBH
** vlan_id: this key will be used to support the vif type VIF_TYPE_HW_VEB [irenab - neutron uses 'segmentation_id'. Let's pass it in the binding:vif_details]
* interface config and resulting interface XML will be generated based on both vnic_type and vif_type. The vnic_type, if not present as a key in the top level port dictionary, defaults to 'virtio'

===Agenda on Jan 28th, 2014===
[[nova-neutron-sriov]]
* Openstack Icehouse schedule: https://wiki.openstack.org/wiki/Icehouse_Release_Schedule
* Work Items for the initial release:
** Nova: generic PCI-passthrough [Nova folks to add details]
*** pci_information support, add tag/extra information to pci devices.
**** pci_information = { { 'device_id': "8086", 'vendor_id': "000[1-2]" }, { 'e.group' :'gpu' } }
*** pci flavor define attr can be used in the pci flavor and how pci stats report it's pool
**** pci_flavor_attrs = ['e.group']
*** pci schduler to support corresponding extra information.
** Nova: SRIOV
*** Dependencies on Nova Generic PCI-Passthrough
**** Support of PCI attribute sriov_group in the PCI passthrough device list
**** Support of PCI stats based on sriov_group
*** overal change breakdown
**** Nova server and scheduler changes:
***** requested network and SRIOV request spec management
**** Nova compute
***** SRIOV request spec--vif association
***** neutronv2 API in nova that supports interaction with neutron. Particularly speaking:
****** to support the enhanced port binding [irenab: vnic_type, pci slot record]
****** to support an enhanced vif dictionary (e.g., vlan id is missing from the dictionary)
***** libvirt driver to support sriov
***** vif driver to support sriov, especially to generate config for sriov device and to generate interface/network xml
***** to support live migration assuming per interface network xml
** Neutron:
**** enhance neutron port-create to support vnic-type and port profile [ irenab: pci slot record]
**** to support the enhanced port binding and as a result API changes in the main plugin
**** to enhance the neutron client in support of the enhanced interaction between nova and neutron [Might not be needed]
**** to enhance various plugins, especially ml2 plugin to support the enhanced port binding (particularly vnic-type)

===Agenda on Jan 27th, 2014===
* Main focus is on SRIOV
* References:
**https://docs.google.com/document/d/1RfxfXBNB0mD_kH9SamwqPI8ZM-jg797ky_Fze7SakRc/edit?pli=1
**https://docs.google.com/document/d/1vadqmurlnlvZ5bv3BlUbFeXRS_wh-dsgi5plSjimWjU/edit?pli=1#
* Some ideas that came out of discussions
** Assuming there is an attribute (based on pci flavor terminology) for sriov, which name is yet to be determined. Call it sriov_group temporarily for reference in below
** Assuming PCI stats will be supported based on this attribute
** Assuming for each physical network that's supported in the cloud, there is a corresponding sriov_group. To make it simple, say if there are two physical networks: physnet1 and physnet2, then there are two sriov_groups named physnet1 and physnet2
** sriov devices in the pci passthrough device list (the pci whitelist) will be tagged by "sriov_group: physnet1" or "sriov_group: physnet2" based on which physical network the device belongs to.
** Initially, we don't have to change nova CLI/API. Instead, the neutron port-create will be enhanced to support the following new arguments:
*** --vnic-type = “vnic” | “direct” | “macvtap”. OPTIONAL with default as "vnic". The actually names of the three possible vnic types for further discussion
*** port-profile=port-profile-name for support of IEEE802.1BR. OPTIONAL.
*** pci-flavor=pci-flavor-name. Support of this argument is debatable. If sriov devices can be used outside of the physical network, or it's desirable to further partition a physical network, it might be necessary to introduce this argument. In that case, pci flavors need to be explicitly created once it's available. This argument may not be needed initially.
** to create a VM with sriov NICs
*** Configuration on the controller node: pci attribute sriov_group
*** configurations on a compute node: physical networks as required by the neutron plugin, pci passthrough device list that are properly tagged with sriov_group.
*** neutron net-create that may be provided with the provider network name, or otherwise uses default physical network name as configured in the neutron plugin
*** neutron port-create --vnic-type direct <net-uuid>
*** nova boot --flavor m1.large --image <image-uuid> --nic port-id=<port-uuid-from-port-create>

* Possible Work Items
** Existing BPs that are relevant based on the above
*** https://blueprints.launchpad.net/neutron/+spec/ml2-request-vnic-type
*** https://blueprints.launchpad.net/neutron/+spec/pci-passthrough-sriov
*** https://blueprints.launchpad.net/nova/+spec/pci-passthrough-sriov
** The above BPs may be redefined or new BPs can be created based on the actual work items.
** Roughly speaking, there are the following work items
*** nova side: a proof of concept patch is captured in here: https://docs.google.com/document/d/196pcKK0iQBJwQfCP0MRaXndjnO-RGfgW1zfeo1YcY4A/edit?pli=1 and https://review.openstack.org/67500
**** Nova server and scheduler changes:
***** although no changes to --nic options for now, the semantics of the port-id parameter is enhanced to support sriov. Basically, if it's a sriov port, request_specs for scheduling should be created and managed.
***** how do we address this requirement that a compute node may support SRIOV port only, a new filter or enhancement to the existing pci passthrough filter? [irenab - May use host aggregate ]
**** Nova compute
***** Nova compute manager to associate a requested network with a request spec, thus corresponds a requested network with a PCI device that is allocated for it
***** neutronv2 API in nova that supports interaction with neutron. Particularly speaking:
****** to support the enhanced port binding
****** to support an enhanced vif dictionary (e.g., vlan id is missing from the dictionary)
***** libvirt driver to support sriov
***** vif driver to support sriov, especially to generate config for sriov device and to generate interface xml
***** to support live migration assuming per interface network xml
*** neutron side:
**** to support the enhanced port binding in the main plugin
**** to enhance the neutron client in support of the enhanced interaction between nova and neutron [irenab - if we use binding:profile, there should be no changes to existing support]
**** to enhance various plugins, especially ml2 plugin to support the enhanced port binding (particularly vnic-type)

===Agenda on Jan 15th, 2014===
* Ian's proposal for flavor and backend tagging: https://docs.google.com/document/d/1vadqmurlnlvZ5bv3BlUbFeXRS_wh-dsgi5plSjimWjU/edit#
* Review this against known use cases
* Document any use cases not in that document
=== Agenda on Jan. 14th 2014===
* PCI group versus PCI flavor: let's sort out what exactly they are, APIs around them, and pros and cons of each.
* Please check [https://wiki.openstack.org/wiki/Meetings/Passthrough#POC_Implementation]
* Division of works

=== POC Implementation ===
See [https://docs.google.com/document/d/196pcKK0iQBJwQfCP0MRaXndjnO-RGfgW1zfeo1YcY4A/edit?usp=sharing POC implementation]

=== Definitions ===

A specific PCI attachment (could by a virtual function) is described by:
* vendor_id
* product_id
* address

There is a whitelist (at the moment):
* which devices on a specific hypervisor host can be exposed

There is an alias (at the moment):
* groupings of PCI devices

==== The user view of system ====

For GPU passthrough, we need things like:
* user request a "large" GPU
* could be from various vendors or product versions

For network, we need things like:
* user requests a nic for a specific neutron network
* they want to say if it's virtual (the default type) or passthrough (super fast, slow, etc)
* this includes groups by address (virtual function, etc) so it's specific to a particular _group_ of neutron network, each with specific configurations (e.g. VLAN id, a NIC attached to a specific provider network)
* or it involves a NIC that can be programmatically made to attach to a specific neutron network

==== The user view of requesting things ====

For GPU passthrough:
* user requests a flavor extra specs *imply* which possible PCI devices can be connected
* nova boot --image some_image --flavor flavor_that_has_big_GPU_attached some_name

The admin would expose a flavor that gives you, for example, one large GPU and one small GPU:
* nova flavor-key m1.large set "pci_passthrough:alias"=" large_GPU:1,small_GPU:1"
* TODO - this may change in the future

For SRIOV:
* in the most basic case, the user may be given direct access to a network card, just like we do with GPU, but this is less useful than...
* user requests neutron nics, on specific neutron networks, but connected in a specific way (i.e. high speed SRIOV vs virtual)
* note that some of the nics may be virtual, some may be passthrough, and some might be a different type of passthrough
* nova boot --flavor m1.large --image <image_id> --nic net-id=<net-id>,nic-type=<slow | fast | foobar> <vm-name>
* (where slow is a virtual connection, fast is a PCI passthrough, and foobar is some other type of PCI passthrough)
* consider several nics, of different types: nova boot --flavor m1.large --image <image_id> --nic net-id=<net-id-1> --nic net-id=<net-id-2>,nic-type=fast --nic net-id=<net-id-3>,nic-type=faster <vm-name>
* when hot-plugging hot-unplugging, we also need to specify vnic-type in a similar way
* also, this should work nova boot --flavor m1.large --image <image_id> --nic port-id=<port-id>, given
* quantum port-create --fixed-ip subnet_id=<subnet-id>,ip_address=192.168.57.101 <net-id> --nic-type=<slow | fast | foobar>

TODO: need agreement, but one idea for admin...
* pci_alias_1='{"name":"Cisco.VIC", devices:[{"vendor_id":"1137","product_id":"0071", address:"*", "attach-type":"macvtap"}],"nic-type":"fast"}'
* pci_alias_2='{"name":"Fast",devices:[{"vendor_id":"1137","product_id":"0071", address:"*","attach-type":"direct"}, {"vendor_id":"123","product_id":"0081", address:"*","attach-type":"macvtap"}],"nic-type":"faster", }'

==== New Proposal for admin view ====

Whitelist:
* only certain devices exposed to Nova
* just a list of addresses that are allowed (including wildcards)
* by default, nothing is allowed
* this is assumed to be (mostly) static for the lifetime of the machine
* contained in nova.conf

PCI flavors:
* specify groups of PCI devices, to be used in Neutron port descriptions or Server flavor extra specs
* configured using host aggregates API:
** a combination of whitelist, alias and group
** raw device passthrough (grouped by device_id and product_id)
** network device passthrough (grouped by device address also)
** note: there might be several options for each (GPU v3 and GPU v4 in a single flavor)
* only servers in the aggregate will be considered by the scheduler for each PCI flavor
* these are shared across the whole child cell (or if no cells, whole nova deploy)

Scheduler updates:
* on periodic update, report current status of devices
* if any devices are in the whitelist, look up host aggregates to check what device types to report
* report the number of free devices per PCI flavor
* device usage tracked by resource manager as normal, looking at all devices in whitelist

On attach of PCI device:
* scheduler picks host it things has a free device
* check with resource manage in usual way
* assign device to VM
* ignoring migration for now

On attach of VIF device (through boot or otherwise):
* TBD... very sketchy plan...
* ideally neutron port contains associated PCI flavor/alias, or its assumed to be a virtual port
* neutron supplies the usual information, VLAN-id, etc
* neutron and nova negotiate which VIF driver to use, in usual way, given extra info about nic-type from PCI alias settings, etc
* VIF driver given a hypervisor agnostic lib to attach the PCI device, extracted from Nova attach PCI device code
* VIF driver is free to configure the specific PCI device before attaching it using the callback into the Nova driver (or modify Nova code to extend the create API)

=== Agenda on Jan. 8th, 2014 ===

:Let's go over the key concepts and use cases. In the use cases, neutron or neutron plugin specific configurations are not mentioned.

==== Key Concepts ====
'''Note that paragraphs with ''Itablic'' text were added by Iawells as comments.'''
* PCI Groups
:# A PCI group is a collection of PCI devices that share the same functions or belong to the same subsystem in a cloud.

''In fact, two proposals exist for PCI group definition - via API, with the implication that they're stored centrally in the database, and via config, specifically a (match expression -> PCI group name) in the compute node configuration. A competing proposal is PCI aliases, which work on the current assumption that all PCI device data is returned to the database and PCI devices can be selected by doing matching at schedule time and thus a name -> match expression mapping is all that need be saved. Thus the internal question of "should all device information be returned to the controller" drives some of the design options.''

:# it's worth mentioning that using an API to define PCI groups make them owned by the tenant who creates them.

* Pre-defined PCI Groups
:For each PCI device class that openstack supports, a PCI group is defined and associated with the PCI devices belonging to that device class. For example, for the PCI device class ''net'', there is a predefined PCI group named ''net''

* User-defined PCI Groups
:User can define PCI groups using a Nova API.

* PCI Passthrough List (whitelist)
:# Specified on a compute node to define all the PCI passthrough devices and their associated PCI groups that are available on the node.
:# blacklist (exclude list) may be added later if deemed necessary.

* vnic_type:
:# virtio: a virtual port that is attached to a virtual switch
:# direct: SRIOV without macvtap
:# macvtap: SRIOV with macvtap
''This configuration item is not essential to PCI passthrough. It's also a Neutron configuration item.''

* nova boot: new parameters in --nic option
:# vnic-type=“vnic” | “direct” | “macvtap”
:# pci-group=pci-group-name
:# port-profile=port-profile-name ''This property is not related directly to use of PCI passthrough for networks. It is a requirement of 802.1BR-based systems.''

* neutron port-create: new arguments
:# --vnic-type “vnic” | “direct” | “macvtap”
:# --pci-group pci-group-name
:# port-profile port-profile-name

* Nova SRIOV Configuration
:# vnic_type = <vnic-type>: specified in controller node to indicate the default vnic-type that VMs will be booted with. default value is "vnic"
:# sriov_auto_all = <on | off>: specified in compute nodes to indicate that all sriov capable ports are added into the ‘net’ PCI group.
:# sriov_only = <on | off>: specified in compute nodes to indicate that nova can only place VMs with sriov vnics onto these nodes. Default value is on for nodes with SRIOV ports.
:# sriov_pci_group = <pci-group-name>: specified in compute nodes in which all of its SRIOV ports belong to a single pci group.
''The SRIOV cofiguration items are enhancements to the base proposal that make it much easier to configure compute hosts where it is known that all VFs will be available to cloud users.''

==== Use Cases ====
''These use cases do not include non-network passthrough cases.''
* SRIOV based cloud
:# All the compute nodes are identical and all the NICs are SRIOV based
:# All the NICs are connected to the same physical network

:In this cloud, the admin only needs to specify vnic_type=direct on the controller and sriov_auto_all=on on the compute nodes in the nova configuration file. In addition, the new arguments introduced in the nova boot command are not required.

* A cloud with mixed Vnics
:# On compute nodes with sriov ports only, set sriov_auto_all = on
:# On compute nodes without sriov ports, no change is required

:In such a cloud, when booting a VM with sriov vnic, the nova boot command would look like:

nova boot --flavor m1.large --image <image_id>
--nic net-id=<net-id>,vnic-type=direct <vm-name>

:This will require some minimum change in the existing applications.

* A Cloud that requires multiple SRIOV PCI groups
:# create all the pci-groups in the cloud by invoking a Nova API
:# on compute nodes that support a single pci group and in which all of the SRIOV ports belong to this group, set sriov_auto_all=on, sriov_pci_group=<group_name>
:# on compute nodes that support multiple pci groups, define the pci-passthrough-list

:In such a cloud, when booting a VM with sriov macvtap, the nova boot command would look like:
nova boot --flavor m1.large --image <image_id>
--nic net-id=<net-id>,vnic-type=macvtap,pci-group=<group-name> <vm-name>

* Introducing new compute nodes with SRIOV into an existing cloud
:Depending on the cloud and the compute node being introduced:
:# it could be as simple as adding sriov_auto_all=on into the nova config file
:# it could be setting sriov_auto_all=on and pci_group=<group_name>
:# it could be defining the pci-passthrough-list.

* NIC hot plug

=== Evolving Design Doc ===

https://docs.google.com/document/d/1EMwDg9J8zOxzvTnQJ9HwZdiotaVstFWKIuKrPse6JOs/edit?usp=sharing

Ian typed up a complete proposal in two sections in that document, which is pulled out here: https://docs.google.com/document/d/1svN89UXKbFoka0EF6MFUP6OwdNvhY4OkdjEZN-rD-0Q/edit# - this proposal takes the 'PCI groups via compute node config' approach and makes no attempt at proposing APIs.

=== Previous Meetings ===
http://eavesdrop.openstack.org/meetings/pci_passthrough_meeting/2013/pci_passthrough_meeting.2013-12-24-14.02.log.html
http://eavesdrop.openstack.org/meetings/pci_passthrough/

:'''Meeting log on Dec. 17th, 2013'''
[[Meetings/Passthrough/dec-17th-2013.log]]

:'''Meeting log on June. 17th, 2014'''
http://eavesdrop.openstack.org/meetings/passthrough/2014/passthrough.2014-06-17-13.07.log.html

Meetings/Passthrough

2014-06-24T10:39:01Z

Itzikb: /* Agenda on May 26th, 2014 */

Next meeting will be on Wed May 7th at 1300 UTC on #openstack-meeting-alt

=== Agenda on May 26th, 2014 ===
* Nova SR-IOV networking
** [https://review.openstack.org/#/c/86606/ nova spec review]
** patch submission soon
** tempest tests
* Bugs
** [https://review.openstack.org/#/c/82206/ Init PCI manager after compute node initialization]
** [https://review.openstack.org/#/c/81954/ makes sure correct PCI device allocation]
* Other works/features
** Live migration
** Attach SR-IOV port to an existing instance
** possibly changing the names of existing vnic-types
** associate a default vnic-type with a neutron network
** admin controlled knobs
** HA
** others

Irenab: Sorry, I'll miss this one.

=== Agenda on March 25th, 2014 ===
* PCI SR-IOV Networking use case
Refer to https://docs.google.com/document/d/1zgMaXqrCnad01-jQH7Mkmf6amlghw9RMScGLBrKslmw/edit

=== Recap of Discussions ===
==== Current PCI Passthrough ====

* How to Use Wiki: https://wiki.openstack.org/wiki/Pci_passthrough

* PCI whitelist: defines all the PCI passthrough devices that are available on a compute node. It's currently based on <vendor_id> <product_id>
* PCI stats group: defines the keys based on which a PCI device is accounted. It's currently based on the keys: <vendor_id> <product_id> <extra_info or simply PCI address of a physical function>.
* PCI Alias: specifies a list of PCI device requirements. A PCI requirement specifies a dictionary with keys: <alias name> <vendor_id> <product_id> <device_type>. Multiple requirements can be specified per PCI Alias with the same <device_type> (which may not work yet due to bugs). <alias name> is required. Others are optional.
* Nova Server Flavor: PCI requirements can be added into a nova flavor as it's extra-spec in the syntax "pci_passthrough:alias"="<PCI Alias Name>:<count>{,<PCI Alias Name>:<count>}"
* PCI Passthrough filter for nova scheduler: this filter works based on the PCI stats group and PCI aliases referenced in the nova flavor. If mutliple aliases exist in the flavor, all of the them have to be satified. If multiple PCI requirements exist in one PCI alias, only one of the requirements defined in the alias needs to be satisfied. Suppose a PCI requirement is represented as R, and a PCI alias as (R1 OR R2 OR ...):count. Further assuming two PCI aliases, with the first one having two PCI requirements, with the second having one PCI requirement. Logically speaking, in order to choose a host as candidate, it must satisfy
(R11 OR R12):count AND R21:count

Given a PCI requirement, it's used to match against PCI stats groups until one or multiple matches are found to satisfy the count. Note that matching is based on the keys existing in the PCI requirement. Therefore it's possible for a PCI requirement to be matching multiple PCI stats groups.

==== Motivation for enhancement ====

* PCI whitelist only uses <vendor_id> and <product_id>. This doesn't work if a compute node has multiple vNICs from the same vendor that are not used in the same way.
* PCI requirement only uses <vendor_id> and <product_id>. This doesn't work when the vNIC ports are connected to different physical networks . In other words, PCI devices from the same vendor may not be treated equally when it comes to choose vNIC ports for a VM.

==== What we have discussed so far to satisfy the above requirements ====

===== PCI Group =====

* Refer to https://docs.google.com/document/d/1EMwDg9J8zOxzvTnQJ9HwZdiotaVstFWKIuKrPse6JOs/edit#heading=h.30de7p6sgoxp. The doc becomes messy and almost unreadable. But refer to https://wiki.openstack.org/wiki/Meetings/Passthrough#Agenda_on_Jan._8th.2C_2014 for the main ideas behind it.
* PCI Group: a collection of PCI devices that share the same functions or belong to the same subsystem in a cloud
* PCI whitelist: each entry is defined as a tuple of <device-filter, pci-group-name>. The device-filter uses vendor_id, product_id and PCI addresses to specify a collection of PCI devices.
* PCI stats group: each PCI group is a PCI stats group
* PCI Alias: alias is no longer needed
* Nova Server Flavor: Use G to represent a PCI group. An exampel of PCI requirements can be specified as:
(G1 OR G2):count AND G3:count
It defines that the VM needs "count" of PCI devices that are either from G1 or G2, and 'count" of PCI devices from G3.

It supports only one well-defined tag called PCI group, which is considered too limited by some of the folks.
It simplifies the existing implementation without losing any of its capabilities (or use cases the current implementation can support).

===== PCI Flavor =====

* Refer to https://docs.google.com/document/d/1vadqmurlnlvZ5bv3BlUbFeXRS_wh-dsgi5plSjimWjU/edit

* PCI whitelist: it's called PCI information in the above wiki
* PCI stats group: not clearly defined
* PCI flavor Attributes: refer to the wiki
* PCI flavor API: refer to the wiki

This can be considered a generalized version of PCI Group as it supports arbitrary number of arbitrary tags. But system behaviors (PCI stats groups and scheduling, for example) are not clearly specified. Use cases to justify it is yet to be established.

===== Host Aggregate =====

It was brought up during the discussion that the existing host aggregate may be used to support the new requirement. A couple of issues with host aggregate were discussed:
* it doesn't support dynamic host join to a host aggregate
* scheduling with host aggregate is not stats based.

===== Nic Type/Flavor =====
Refer to http://lists.openstack.org/pipermail/openstack-dev/2014-January/023981.html and http://lists.openstack.org/pipermail/openstack-dev/2013-December/022737.html

===== A compromised implementation =====
to speed up the development and facilitate integration testing with neutron.
Refer to https://blueprints.launchpad.net/nova/+spec/pci-passthrough-sriov. Also refer to the submitted patch.

=== BPs & Patches ===
==== Neutron Side ====
* RKukura (binding:vif_details and binding:profile)
** https://blueprints.launchpad.net/neutron/+spec/vif-details
** https://blueprints.launchpad.net/neutron/+spec/ml2-binding-profile
* Sadasu (Cisco MD)
** https://blueprints.launchpad.net/neutron/+spec/ml2-ucs-manager-mechanism-driver
* Irenab (vnic_type and mlnx MD)
** https://blueprints.launchpad.net/neutron/+spec/ml2-sriov-nic-switch
** https://blueprints.launchpad.net/neutron/+spec/mlnx-ml2-support
** https://blueprints.launchpad.net/neutron/+spec/ml2-request-vnic-type
* Baoli (for anything else that's needed for SRIOV in neutron)
** https://blueprints.launchpad.net/neutron/+spec/pci-passthrough-sriov

==== Nova Side ====
* Baoli
** https://blueprints.launchpad.net/nova/+spec/pci-passthrough-sriov
* Yongli
** https://blueprints.launchpad.net/nova/+spec/pci-extra-info
* Yunhong
** https://blueprints.launchpad.net/nova/+spec/pci-extra-info-icehouse

==== Itzik's patch to add physical network in VIF ====
https://review.openstack.org/#/c/59093/

===Agenda on Feb. 13th, 2014===
* Discuss the Neutron SR-IOV's requirements on the nova generic PCI support, and their availability
** in the PCI information definition:
*** the pci device filter expression (I don't see it's given a name in https://wiki.openstack.org/wiki/PCI_passthrough_SRIOV_support_Icehouse) should support (which may be done as an enhancement).
**** specifying a physical function in the form of domain:bus:slot.func
**** specifying a physical function in the form of a ethernet interface name.
*** tagging with an attribute called "net-group" or simply "physical_network". The tag name "phyical_network" now may make more sense based on the consensus reached on Feb. 12th.
** PCI stats based on the tag "physical_network"
** An API to retrieve a tag's value on a per PCI device basis, something like: get_pci_device_tag_value(pci_dev, tag_name)
** An API to create a PCI request for scheduling purpose
** An API to retrieve a PCI device that is linked to the original PCI request.
** The existing pci_manager.get_instance_pci_devs(instance) shouldn't return PCI devices that are allocated as a result of the aforementioned PCI requests.

===Feb. 12th, 2014 Recap===
Consensus reached during this meeting:
* The binding:profile dictionary will now have these keys defined: 'physical_network', 'pci_vendor_info', 'pci_slot'.
** physical_network: its value is the physical network name that has been chosen for the nic (neutron port) to attach to. ML2 MD will use this information:
*** In the non-SR-IOV case with the agent-based MDs, the ML2 plugin's port binding code iterates over the registered MDs to try to bind, calling bind_port() on each. Within the AgentMechanismDriverBase.bind_port() implementation, it iterates over the network segments, calling check_segment_for_agent() on the derived class for each segment. The 1st segment that is tried for which the agent on the node identified by binding:host_id has a mapping for the segment's physical_network is used.
*** in the non-SR_IOV case, need to add a line of code in the existing agent-based MDs check_segment_for_agent() to make sure vnic_type == 'virtio', so it won't bind when SR-IOV is required. Irenab's vnic BP will take care of this.
*** in the SR-IOV case, the physical network names that a neutron port can potentially attach to will be used for scheduling, and the change to the scheduling filter in the case of multiprovidernet extension is TBD.
*** in the SR-IOV case, SR-IOV MDs will need to iterate over the segments looking for the 1st one that has network_type of 'vlan' and that has the physical_network specified in binding:profile:physical_network; the SR-IOV MDs can include the segment's segmentation_id within the binding:vif_details so that VIF driver can put that into the libvirt XML
** pci_vendor_info: its value is a string with the format "vendor_id:product_id". Both vendor_id and product_id corresponds to the PCI device's vendor_id and product_id
** pci_slot: its value is a string with the format "domain:bus:slot.func" that correspond's to the PCI device's slot as named on a linux system.
*The binding:vif_details will have keys depending on the neutron port's vif type
** profileid: this key will be used to support the vif type VIF_TYPE_802_1QBH
** vlan_id: this key will be used to support the vif type VIF_TYPE_HW_VEB [irenab - neutron uses 'segmentation_id'. Let's pass it in the binding:vif_details]
* interface config and resulting interface XML will be generated based on both vnic_type and vif_type. The vnic_type, if not present as a key in the top level port dictionary, defaults to 'virtio'

===Agenda on Jan 28th, 2014===
[[nova-neutron-sriov]]
* Openstack Icehouse schedule: https://wiki.openstack.org/wiki/Icehouse_Release_Schedule
* Work Items for the initial release:
** Nova: generic PCI-passthrough [Nova folks to add details]
*** pci_information support, add tag/extra information to pci devices.
**** pci_information = { { 'device_id': "8086", 'vendor_id': "000[1-2]" }, { 'e.group' :'gpu' } }
*** pci flavor define attr can be used in the pci flavor and how pci stats report it's pool
**** pci_flavor_attrs = ['e.group']
*** pci schduler to support corresponding extra information.
** Nova: SRIOV
*** Dependencies on Nova Generic PCI-Passthrough
**** Support of PCI attribute sriov_group in the PCI passthrough device list
**** Support of PCI stats based on sriov_group
*** overal change breakdown
**** Nova server and scheduler changes:
***** requested network and SRIOV request spec management
**** Nova compute
***** SRIOV request spec--vif association
***** neutronv2 API in nova that supports interaction with neutron. Particularly speaking:
****** to support the enhanced port binding [irenab: vnic_type, pci slot record]
****** to support an enhanced vif dictionary (e.g., vlan id is missing from the dictionary)
***** libvirt driver to support sriov
***** vif driver to support sriov, especially to generate config for sriov device and to generate interface/network xml
***** to support live migration assuming per interface network xml
** Neutron:
**** enhance neutron port-create to support vnic-type and port profile [ irenab: pci slot record]
**** to support the enhanced port binding and as a result API changes in the main plugin
**** to enhance the neutron client in support of the enhanced interaction between nova and neutron [Might not be needed]
**** to enhance various plugins, especially ml2 plugin to support the enhanced port binding (particularly vnic-type)

===Agenda on Jan 27th, 2014===
* Main focus is on SRIOV
* References:
**https://docs.google.com/document/d/1RfxfXBNB0mD_kH9SamwqPI8ZM-jg797ky_Fze7SakRc/edit?pli=1
**https://docs.google.com/document/d/1vadqmurlnlvZ5bv3BlUbFeXRS_wh-dsgi5plSjimWjU/edit?pli=1#
* Some ideas that came out of discussions
** Assuming there is an attribute (based on pci flavor terminology) for sriov, which name is yet to be determined. Call it sriov_group temporarily for reference in below
** Assuming PCI stats will be supported based on this attribute
** Assuming for each physical network that's supported in the cloud, there is a corresponding sriov_group. To make it simple, say if there are two physical networks: physnet1 and physnet2, then there are two sriov_groups named physnet1 and physnet2
** sriov devices in the pci passthrough device list (the pci whitelist) will be tagged by "sriov_group: physnet1" or "sriov_group: physnet2" based on which physical network the device belongs to.
** Initially, we don't have to change nova CLI/API. Instead, the neutron port-create will be enhanced to support the following new arguments:
*** --vnic-type = “vnic” | “direct” | “macvtap”. OPTIONAL with default as "vnic". The actually names of the three possible vnic types for further discussion
*** port-profile=port-profile-name for support of IEEE802.1BR. OPTIONAL.
*** pci-flavor=pci-flavor-name. Support of this argument is debatable. If sriov devices can be used outside of the physical network, or it's desirable to further partition a physical network, it might be necessary to introduce this argument. In that case, pci flavors need to be explicitly created once it's available. This argument may not be needed initially.
** to create a VM with sriov NICs
*** Configuration on the controller node: pci attribute sriov_group
*** configurations on a compute node: physical networks as required by the neutron plugin, pci passthrough device list that are properly tagged with sriov_group.
*** neutron net-create that may be provided with the provider network name, or otherwise uses default physical network name as configured in the neutron plugin
*** neutron port-create --vnic-type direct <net-uuid>
*** nova boot --flavor m1.large --image <image-uuid> --nic port-id=<port-uuid-from-port-create>

* Possible Work Items
** Existing BPs that are relevant based on the above
*** https://blueprints.launchpad.net/neutron/+spec/ml2-request-vnic-type
*** https://blueprints.launchpad.net/neutron/+spec/pci-passthrough-sriov
*** https://blueprints.launchpad.net/nova/+spec/pci-passthrough-sriov
** The above BPs may be redefined or new BPs can be created based on the actual work items.
** Roughly speaking, there are the following work items
*** nova side: a proof of concept patch is captured in here: https://docs.google.com/document/d/196pcKK0iQBJwQfCP0MRaXndjnO-RGfgW1zfeo1YcY4A/edit?pli=1 and https://review.openstack.org/67500
**** Nova server and scheduler changes:
***** although no changes to --nic options for now, the semantics of the port-id parameter is enhanced to support sriov. Basically, if it's a sriov port, request_specs for scheduling should be created and managed.
***** how do we address this requirement that a compute node may support SRIOV port only, a new filter or enhancement to the existing pci passthrough filter? [irenab - May use host aggregate ]
**** Nova compute
***** Nova compute manager to associate a requested network with a request spec, thus corresponds a requested network with a PCI device that is allocated for it
***** neutronv2 API in nova that supports interaction with neutron. Particularly speaking:
****** to support the enhanced port binding
****** to support an enhanced vif dictionary (e.g., vlan id is missing from the dictionary)
***** libvirt driver to support sriov
***** vif driver to support sriov, especially to generate config for sriov device and to generate interface xml
***** to support live migration assuming per interface network xml
*** neutron side:
**** to support the enhanced port binding in the main plugin
**** to enhance the neutron client in support of the enhanced interaction between nova and neutron [irenab - if we use binding:profile, there should be no changes to existing support]
**** to enhance various plugins, especially ml2 plugin to support the enhanced port binding (particularly vnic-type)

===Agenda on Jan 15th, 2014===
* Ian's proposal for flavor and backend tagging: https://docs.google.com/document/d/1vadqmurlnlvZ5bv3BlUbFeXRS_wh-dsgi5plSjimWjU/edit#
* Review this against known use cases
* Document any use cases not in that document
=== Agenda on Jan. 14th 2014===
* PCI group versus PCI flavor: let's sort out what exactly they are, APIs around them, and pros and cons of each.
* Please check [https://wiki.openstack.org/wiki/Meetings/Passthrough#POC_Implementation]
* Division of works

=== POC Implementation ===
See [https://docs.google.com/document/d/196pcKK0iQBJwQfCP0MRaXndjnO-RGfgW1zfeo1YcY4A/edit?usp=sharing POC implementation]

=== Definitions ===

A specific PCI attachment (could by a virtual function) is described by:
* vendor_id
* product_id
* address

There is a whitelist (at the moment):
* which devices on a specific hypervisor host can be exposed

There is an alias (at the moment):
* groupings of PCI devices

==== The user view of system ====

For GPU passthrough, we need things like:
* user request a "large" GPU
* could be from various vendors or product versions

For network, we need things like:
* user requests a nic for a specific neutron network
* they want to say if it's virtual (the default type) or passthrough (super fast, slow, etc)
* this includes groups by address (virtual function, etc) so it's specific to a particular _group_ of neutron network, each with specific configurations (e.g. VLAN id, a NIC attached to a specific provider network)
* or it involves a NIC that can be programmatically made to attach to a specific neutron network

==== The user view of requesting things ====

For GPU passthrough:
* user requests a flavor extra specs *imply* which possible PCI devices can be connected
* nova boot --image some_image --flavor flavor_that_has_big_GPU_attached some_name

The admin would expose a flavor that gives you, for example, one large GPU and one small GPU:
* nova flavor-key m1.large set "pci_passthrough:alias"=" large_GPU:1,small_GPU:1"
* TODO - this may change in the future

For SRIOV:
* in the most basic case, the user may be given direct access to a network card, just like we do with GPU, but this is less useful than...
* user requests neutron nics, on specific neutron networks, but connected in a specific way (i.e. high speed SRIOV vs virtual)
* note that some of the nics may be virtual, some may be passthrough, and some might be a different type of passthrough
* nova boot --flavor m1.large --image <image_id> --nic net-id=<net-id>,nic-type=<slow | fast | foobar> <vm-name>
* (where slow is a virtual connection, fast is a PCI passthrough, and foobar is some other type of PCI passthrough)
* consider several nics, of different types: nova boot --flavor m1.large --image <image_id> --nic net-id=<net-id-1> --nic net-id=<net-id-2>,nic-type=fast --nic net-id=<net-id-3>,nic-type=faster <vm-name>
* when hot-plugging hot-unplugging, we also need to specify vnic-type in a similar way
* also, this should work nova boot --flavor m1.large --image <image_id> --nic port-id=<port-id>, given
* quantum port-create --fixed-ip subnet_id=<subnet-id>,ip_address=192.168.57.101 <net-id> --nic-type=<slow | fast | foobar>

TODO: need agreement, but one idea for admin...
* pci_alias_1='{"name":"Cisco.VIC", devices:[{"vendor_id":"1137","product_id":"0071", address:"*", "attach-type":"macvtap"}],"nic-type":"fast"}'
* pci_alias_2='{"name":"Fast",devices:[{"vendor_id":"1137","product_id":"0071", address:"*","attach-type":"direct"}, {"vendor_id":"123","product_id":"0081", address:"*","attach-type":"macvtap"}],"nic-type":"faster", }'

==== New Proposal for admin view ====

Whitelist:
* only certain devices exposed to Nova
* just a list of addresses that are allowed (including wildcards)
* by default, nothing is allowed
* this is assumed to be (mostly) static for the lifetime of the machine
* contained in nova.conf

PCI flavors:
* specify groups of PCI devices, to be used in Neutron port descriptions or Server flavor extra specs
* configured using host aggregates API:
** a combination of whitelist, alias and group
** raw device passthrough (grouped by device_id and product_id)
** network device passthrough (grouped by device address also)
** note: there might be several options for each (GPU v3 and GPU v4 in a single flavor)
* only servers in the aggregate will be considered by the scheduler for each PCI flavor
* these are shared across the whole child cell (or if no cells, whole nova deploy)

Scheduler updates:
* on periodic update, report current status of devices
* if any devices are in the whitelist, look up host aggregates to check what device types to report
* report the number of free devices per PCI flavor
* device usage tracked by resource manager as normal, looking at all devices in whitelist

On attach of PCI device:
* scheduler picks host it things has a free device
* check with resource manage in usual way
* assign device to VM
* ignoring migration for now

On attach of VIF device (through boot or otherwise):
* TBD... very sketchy plan...
* ideally neutron port contains associated PCI flavor/alias, or its assumed to be a virtual port
* neutron supplies the usual information, VLAN-id, etc
* neutron and nova negotiate which VIF driver to use, in usual way, given extra info about nic-type from PCI alias settings, etc
* VIF driver given a hypervisor agnostic lib to attach the PCI device, extracted from Nova attach PCI device code
* VIF driver is free to configure the specific PCI device before attaching it using the callback into the Nova driver (or modify Nova code to extend the create API)

=== Agenda on Jan. 8th, 2014 ===

:Let's go over the key concepts and use cases. In the use cases, neutron or neutron plugin specific configurations are not mentioned.

==== Key Concepts ====
'''Note that paragraphs with ''Itablic'' text were added by Iawells as comments.'''
* PCI Groups
:# A PCI group is a collection of PCI devices that share the same functions or belong to the same subsystem in a cloud.

''In fact, two proposals exist for PCI group definition - via API, with the implication that they're stored centrally in the database, and via config, specifically a (match expression -> PCI group name) in the compute node configuration. A competing proposal is PCI aliases, which work on the current assumption that all PCI device data is returned to the database and PCI devices can be selected by doing matching at schedule time and thus a name -> match expression mapping is all that need be saved. Thus the internal question of "should all device information be returned to the controller" drives some of the design options.''

:# it's worth mentioning that using an API to define PCI groups make them owned by the tenant who creates them.

* Pre-defined PCI Groups
:For each PCI device class that openstack supports, a PCI group is defined and associated with the PCI devices belonging to that device class. For example, for the PCI device class ''net'', there is a predefined PCI group named ''net''

* User-defined PCI Groups
:User can define PCI groups using a Nova API.

* PCI Passthrough List (whitelist)
:# Specified on a compute node to define all the PCI passthrough devices and their associated PCI groups that are available on the node.
:# blacklist (exclude list) may be added later if deemed necessary.

* vnic_type:
:# virtio: a virtual port that is attached to a virtual switch
:# direct: SRIOV without macvtap
:# macvtap: SRIOV with macvtap
''This configuration item is not essential to PCI passthrough. It's also a Neutron configuration item.''

* nova boot: new parameters in --nic option
:# vnic-type=“vnic” | “direct” | “macvtap”
:# pci-group=pci-group-name
:# port-profile=port-profile-name ''This property is not related directly to use of PCI passthrough for networks. It is a requirement of 802.1BR-based systems.''

* neutron port-create: new arguments
:# --vnic-type “vnic” | “direct” | “macvtap”
:# --pci-group pci-group-name
:# port-profile port-profile-name

* Nova SRIOV Configuration
:# vnic_type = <vnic-type>: specified in controller node to indicate the default vnic-type that VMs will be booted with. default value is "vnic"
:# sriov_auto_all = <on | off>: specified in compute nodes to indicate that all sriov capable ports are added into the ‘net’ PCI group.
:# sriov_only = <on | off>: specified in compute nodes to indicate that nova can only place VMs with sriov vnics onto these nodes. Default value is on for nodes with SRIOV ports.
:# sriov_pci_group = <pci-group-name>: specified in compute nodes in which all of its SRIOV ports belong to a single pci group.
''The SRIOV cofiguration items are enhancements to the base proposal that make it much easier to configure compute hosts where it is known that all VFs will be available to cloud users.''

==== Use Cases ====
''These use cases do not include non-network passthrough cases.''
* SRIOV based cloud
:# All the compute nodes are identical and all the NICs are SRIOV based
:# All the NICs are connected to the same physical network

:In this cloud, the admin only needs to specify vnic_type=direct on the controller and sriov_auto_all=on on the compute nodes in the nova configuration file. In addition, the new arguments introduced in the nova boot command are not required.

* A cloud with mixed Vnics
:# On compute nodes with sriov ports only, set sriov_auto_all = on
:# On compute nodes without sriov ports, no change is required

:In such a cloud, when booting a VM with sriov vnic, the nova boot command would look like:

nova boot --flavor m1.large --image <image_id>
--nic net-id=<net-id>,vnic-type=direct <vm-name>

:This will require some minimum change in the existing applications.

* A Cloud that requires multiple SRIOV PCI groups
:# create all the pci-groups in the cloud by invoking a Nova API
:# on compute nodes that support a single pci group and in which all of the SRIOV ports belong to this group, set sriov_auto_all=on, sriov_pci_group=<group_name>
:# on compute nodes that support multiple pci groups, define the pci-passthrough-list

:In such a cloud, when booting a VM with sriov macvtap, the nova boot command would look like:
nova boot --flavor m1.large --image <image_id>
--nic net-id=<net-id>,vnic-type=macvtap,pci-group=<group-name> <vm-name>

* Introducing new compute nodes with SRIOV into an existing cloud
:Depending on the cloud and the compute node being introduced:
:# it could be as simple as adding sriov_auto_all=on into the nova config file
:# it could be setting sriov_auto_all=on and pci_group=<group_name>
:# it could be defining the pci-passthrough-list.

* NIC hot plug

=== Evolving Design Doc ===

https://docs.google.com/document/d/1EMwDg9J8zOxzvTnQJ9HwZdiotaVstFWKIuKrPse6JOs/edit?usp=sharing

Ian typed up a complete proposal in two sections in that document, which is pulled out here: https://docs.google.com/document/d/1svN89UXKbFoka0EF6MFUP6OwdNvhY4OkdjEZN-rD-0Q/edit# - this proposal takes the 'PCI groups via compute node config' approach and makes no attempt at proposing APIs.

=== Previous Meetings ===
http://eavesdrop.openstack.org/meetings/pci_passthrough_meeting/2013/pci_passthrough_meeting.2013-12-24-14.02.log.html
http://eavesdrop.openstack.org/meetings/pci_passthrough/

:'''Meeting log on Dec. 17th, 2013'''
[[Meetings/Passthrough/dec-17th-2013.log]]

:'''Meeting log on June. 17th, 2014'''
http://eavesdrop.openstack.org/meetings/passthrough/2014/passthrough.2014-06-17-13.07.log.html

Mellanox-Neutron-Icehouse-Redhat

2014-06-12T14:41:45Z

Itzikb: /* Configuration */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

You can do it by using packstack:

1.Create an answer file
#packstack --gen-answer-file=GEN_ANSWER_FILE

2. Change the following in GEN_ANSWER_FILE
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_VLAN_RANGES=default:2:10
CONFIG_NEUTRON_L2_AGENT=openvswitch

3. Run packstack
#packstack --answer-file=GEN_ANSWER_FILE

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = mlnx,openvswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
vif_driver=mlnxvif.vif.MlxEthVIFDriver
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc_support_old_agents''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-openvswitch-agent start
#/etc/init.d/neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#/etc/init.d/neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
#/etc/init.d/eswitchd restart
#/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

In case you use MLNX-OFED older than 2.1, Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
#/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
#/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-06-10T14:07:17Z

Itzikb: /* Prerequisites */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

You can do it by using packstack:

1.Create an answer file
#packstack --gen-answer-file=GEN_ANSWER_FILE

2. Change the following in GEN_ANSWER_FILE
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_VLAN_RANGES=default:2:10
CONFIG_NEUTRON_L2_AGENT=openvswitch

3. Run packstack
#packstack --answer-file=GEN_ANSWER_FILE

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = mlnx,openvswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
vif_driver=mlnxvif.vif.MlxEthVIFDriver
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-openvswitch-agent start
#/etc/init.d/neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#/etc/init.d/neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
#/etc/init.d/eswitchd restart
#/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

In case you use MLNX-OFED older than 2.1, Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
#/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
#/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-06-10T13:58:45Z

Itzikb: /* Prerequisites */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

You can do it by using packstack:

1.Create an answer file
#packstack --gen-answer-file=GEN_ANSWER_FILE

2. Change the following in GEN_ANSWER_FILE
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_VLAN_RANGES=default:2:10
CONFIG_NEUTRON_L2_AGENT=openvswitch

3. Run packstack
#packstack --answer-file==GEN_ANSWER_FILE

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = mlnx,openvswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
vif_driver=mlnxvif.vif.MlxEthVIFDriver
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-openvswitch-agent start
#/etc/init.d/neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#/etc/init.d/neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
#/etc/init.d/eswitchd restart
#/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

In case you use MLNX-OFED older than 2.1, Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
#/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
#/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-06-08T07:07:46Z

Itzikb: /* Installation */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA.
Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron pluign.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types.
For vnic type configuration API details, please refer to configuration reference guide http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html.
Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

* Mellanox Openstack Neutron Agent (L2 Agent) runs on each compute node.
* Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed.Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3.Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = mlnx,openvswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. If you didn't download Mellanox OpenStack repo file download it :
#wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
vif_driver=mlnxvif.vif.MlxEthVIFDriver
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#//etc/init.d/neutron-openvswitch-agent start
#/etc/init.d/neutron-dhcp-agent start

5.Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6.Start the l3 Agent:
#/etc/init.d/neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

/etc/init.d/ufmd restart

High-availability

/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
/etc/init.d/eswitchd restart
/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd
'''Note:''' If you have OFED >= 2.1 the above step is not needed

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-06-08T07:07:02Z

Itzikb: /* Installation */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA.
Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron pluign.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types.
For vnic type configuration API details, please refer to configuration reference guide http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html.
Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

* Mellanox Openstack Neutron Agent (L2 Agent) runs on each compute node.
* Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed.Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at /etc/neutron/plugins/ml2/ml2_conf.ini

3.Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = mlnx,openvswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. If you didn't download Mellanox OpenStack repo file download it :
#wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
vif_driver=mlnxvif.vif.MlxEthVIFDriver
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#//etc/init.d/neutron-openvswitch-agent start
#/etc/init.d/neutron-dhcp-agent start

5.Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6.Start the l3 Agent:
#/etc/init.d/neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

/etc/init.d/ufmd restart

High-availability

/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
/etc/init.d/eswitchd restart
/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd
'''Note:''' If you have OFED >= 2.1 the above step is not needed

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-06-08T07:06:06Z

Itzikb: /* Prerequisites */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA.
Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron pluign.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types.
For vnic type configuration API details, please refer to configuration reference guide http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html.
Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

* Mellanox Openstack Neutron Agent (L2 Agent) runs on each compute node.
* Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed.Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at /etc/neutron/plugins/ml2/ml2_conf.ini

3.Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = mlnx,openvswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. If you didn't download Mellanox OpenStack repo file download it :
#wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
vif_driver=mlnxvif.vif.MlxEthVIFDriver
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#//etc/init.d/neutron-openvswitch-agent start
#/etc/init.d/neutron-dhcp-agent start

5.Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6.Start the l3 Agent:
#/etc/init.d/neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

/etc/init.d/ufmd restart

High-availability

/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
/etc/init.d/eswitchd restart
/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd
'''Note:''' If you have OFED >= 2.1 the above step is not needed

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-06-08T07:04:49Z

Itzikb: /* Prerequisites */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA.
Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron pluign.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types.
For vnic type configuration API details, please refer to configuration reference guide http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html.
Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

* Mellanox Openstack Neutron Agent (L2 Agent) runs on each compute node.
* Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.0.3 or greater is installed.Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at /etc/neutron/plugins/ml2/ml2_conf.ini

3.Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = mlnx,openvswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. If you didn't download Mellanox OpenStack repo file download it :
#wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
vif_driver=mlnxvif.vif.MlxEthVIFDriver
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#//etc/init.d/neutron-openvswitch-agent start
#/etc/init.d/neutron-dhcp-agent start

5.Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6.Start the l3 Agent:
#/etc/init.d/neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

/etc/init.d/ufmd restart

High-availability

/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
/etc/init.d/eswitchd restart
/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd
'''Note:''' If you have OFED >= 2.1 the above step is not needed

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-05-26T07:18:26Z

Itzikb: /* Network Node */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA.
Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron pluign.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types.
For vnic type configuration API details, please refer to configuration reference guide http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html.
Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

* Mellanox Openstack Neutron Agent (L2 Agent) runs on each compute node.
* Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.0.3 or greater is installed.Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.0.3 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at /etc/neutron/plugins/ml2/ml2_conf.ini

3.Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = mlnx,openvswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. If you didn't download Mellanox OpenStack repo file download it :
#wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
vif_driver=mlnxvif.vif.MlxEthVIFDriver
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#//etc/init.d/neutron-openvswitch-agent start
#/etc/init.d/neutron-dhcp-agent start

5.Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6.Start the l3 Agent:
#/etc/init.d/neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

/etc/init.d/ufmd restart

High-availability

/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
/etc/init.d/eswitchd restart
/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd
'''Note:''' If you have OFED >= 2.1 the above step is not needed

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-05-26T07:17:27Z

Itzikb: /* Network Node */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA.
Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron pluign.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types.
For vnic type configuration API details, please refer to configuration reference guide http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html.
Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

* Mellanox Openstack Neutron Agent (L2 Agent) runs on each compute node.
* Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.0.3 or greater is installed.Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.0.3 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at /etc/neutron/plugins/ml2/ml2_conf.ini

3.Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = mlnx,openvswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. If you didn't download Mellanox OpenStack repo file download it :
#wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
vif_driver=mlnxvif.vif.MlxEthVIFDriver
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#//etc/init.d/neutron-openvswitch-agent start
#/etc/init.d/neutron-dhcp-agent start

5.Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

/etc/init.d/ufmd restart

High-availability

/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
/etc/init.d/eswitchd restart
/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd
'''Note:''' If you have OFED >= 2.1 the above step is not needed

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-05-26T07:11:06Z

Itzikb: /* Network Node */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA.
Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron pluign.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types.
For vnic type configuration API details, please refer to configuration reference guide http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html.
Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

* Mellanox Openstack Neutron Agent (L2 Agent) runs on each compute node.
* Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.0.3 or greater is installed.Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.0.3 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at /etc/neutron/plugins/ml2/ml2_conf.ini

3.Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = mlnx,openvswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. If you didn't download Mellanox OpenStack repo file download it :
#wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
vif_driver=mlnxvif.vif.MlxEthVIFDriver
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#//etc/init.d/neutron-openvswitch-agent start
#/etc/init.d/neutron-dhcp-agent start

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

/etc/init.d/ufmd restart

High-availability

/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
/etc/init.d/eswitchd restart
/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd
'''Note:''' If you have OFED >= 2.1 the above step is not needed

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-05-26T06:52:52Z

Itzikb: /* Configuration */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA.
Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron pluign.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types.
For vnic type configuration API details, please refer to configuration reference guide http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html.
Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

* Mellanox Openstack Neutron Agent (L2 Agent) runs on each compute node.
* Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.0.3 or greater is installed.Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.0.3 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at /etc/neutron/plugins/ml2/ml2_conf.ini

3.Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = mlnx,openvswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. If you didn't download Mellanox OpenStack repo file download it :
#wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
vif_driver=mlnxvif.vif.MlxEthVIFDriver
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron linux bridge plugin and Neutron dhcp l3-agent:

#yum install openstack-neutron-linuxbridge

2. Change the following configuration of the ini file (/etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .

physical_interface_mappings = default:eth2

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-linuxbridge-agent start
#/etc/init.d/neutron-dhcp-agent start

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

/etc/init.d/ufmd restart

High-availability

/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
/etc/init.d/eswitchd restart
/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd
'''Note:''' If you have OFED >= 2.1 the above step is not needed

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-05-26T06:47:34Z

Itzikb: /* Prerequisites */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA.
Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron pluign.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types.
For vnic type configuration API details, please refer to configuration reference guide http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html.
Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

* Mellanox Openstack Neutron Agent (L2 Agent) runs on each compute node.
* Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.0.3 or greater is installed.Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.0.3 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at /etc/neutron/plugins/ml2/ml2_conf.ini

3.Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = mlnx,openvswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. If you didn't download Mellanox OpenStack repo file download it :
#wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
vif_driver=mlnxvif.vif.MlxEthVIFDriver
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config *ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron linux bridge plugin and Neutron dhcp l3-agent:

#yum install openstack-neutron-linuxbridge

2. Change the following configuration of the ini file (/etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .

physical_interface_mappings = default:eth2

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-linuxbridge-agent start
#/etc/init.d/neutron-dhcp-agent start

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

/etc/init.d/ufmd restart

High-availability

/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
/etc/init.d/eswitchd restart
/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd
'''Note:''' If you have OFED >= 2.1 the above step is not needed

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-05-25T14:26:40Z

Itzikb: /* Configuration */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA.
Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron pluign.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types.
For vnic type configuration API details, please refer to configuration reference guide http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html.
Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

* Mellanox Openstack Neutron Agent (L2 Agent) runs on each compute node.
* Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.0.3 or greater is installed.Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. The software package ethtool ([http://www.kernel.org/pub/software/network/ethtool/ Code]) must be installed (version 3.8 or higher). Required only to be installed on compute nodes.

7. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.0.3 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at /etc/neutron/plugins/ml2/ml2_conf.ini

3.Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = mlnx,openvswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. If you didn't download Mellanox OpenStack repo file download it :
#wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
vif_driver=mlnxvif.vif.MlxEthVIFDriver
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config *ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron linux bridge plugin and Neutron dhcp l3-agent:

#yum install openstack-neutron-linuxbridge

2. Change the following configuration of the ini file (/etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .

physical_interface_mappings = default:eth2

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-linuxbridge-agent start
#/etc/init.d/neutron-dhcp-agent start

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

/etc/init.d/ufmd restart

High-availability

/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
/etc/init.d/eswitchd restart
/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd
'''Note:''' If you have OFED >= 2.1 the above step is not needed

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-05-25T14:24:24Z

Itzikb: /* Configuration */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA.
Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron pluign.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types.
For vnic type configuration API details, please refer to configuration reference guide http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html.
Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

* Mellanox Openstack Neutron Agent (L2 Agent) runs on each compute node.
* Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.0.3 or greater is installed.Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. The software package ethtool ([http://www.kernel.org/pub/software/network/ethtool/ Code]) must be installed (version 3.8 or higher). Required only to be installed on compute nodes.

7. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.0.3 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at /etc/neutron/plugins/ml2/ml2_conf.ini

3.Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = mlnx,openvswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. If you didn't download Mellanox OpenStack repo file download it :
#wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
vif_driver=mlnxvif.vif.MlxEthVIFDriver
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Havana), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config *ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron linux bridge plugin and Neutron dhcp l3-agent:

#yum install openstack-neutron-linuxbridge

2. Change the following configuration of the ini file (/etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .

physical_interface_mappings = default:eth2

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-linuxbridge-agent start
#/etc/init.d/neutron-dhcp-agent start

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

/etc/init.d/ufmd restart

High-availability

/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
/etc/init.d/eswitchd restart
/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd
'''Note:''' If you have OFED >= 2.1 the above step is not needed

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-05-25T14:00:45Z

Itzikb: /* Installation */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA.
Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron pluign.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types.
For vnic type configuration API details, please refer to configuration reference guide http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html.
Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

* Mellanox Openstack Neutron Agent (L2 Agent) runs on each compute node.
* Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.0.3 or greater is installed.Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. The software package ethtool ([http://www.kernel.org/pub/software/network/ethtool/ Code]) must be installed (version 3.8 or higher). Required only to be installed on compute nodes.

7. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.0.3 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at /etc/neutron/plugins/ml2/ml2_conf.ini

3.Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = mlnx,openvswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. If you didn't download Mellanox OpenStack repo file download it :
#wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver
security_group_api=nova
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only releavant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''vnic_type''' - type of VM network interface: 'mlnx_direct' or 'hostdev' according to libvirt terminology.
hostdev: this is the traditional method of assigning any generic PCI device to a guest (SR-IOV).
mlnx_direct: this is a method to provide macvtap device on top of the PCI device (default).
bridge: - When using Linux Bridge Plugin on top of eIPoIB device
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:5001')
daemon_endpoint should be changed to tcp://127.0.0.1:60001
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

'''Note''': daemon_endpoint should be changed to tcp://127.0.0.1:60001

For a plugin configuration file example (Havana), please refer to [https://github.com/openstack/neutron/blob/stable/havana/etc/neutron/plugins/mlnx/mlnx_conf.inii Mellanox config *ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron linux bridge plugin and Neutron dhcp l3-agent:

#yum install openstack-neutron-linuxbridge

2. Change the following configuration of the ini file (/etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .

physical_interface_mappings = default:eth2

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-linuxbridge-agent start
#/etc/init.d/neutron-dhcp-agent start

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

/etc/init.d/ufmd restart

High-availability

/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
/etc/init.d/eswitchd restart
/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd
'''Note:''' If you have OFED >= 2.1 the above step is not needed

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-05-25T14:00:27Z

Itzikb: /* Installation */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA.
Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron pluign.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types.
For vnic type configuration API details, please refer to configuration reference guide http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html.
Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

* Mellanox Openstack Neutron Agent (L2 Agent) runs on each compute node.
* Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.0.3 or greater is installed.Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. The software package ethtool ([http://www.kernel.org/pub/software/network/ethtool/ Code]) must be installed (version 3.8 or higher). Required only to be installed on compute nodes.

7. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.0.3 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at /etc/neutron/plugins/ml2/ml2_conf.ini

3.Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = mlnx
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. If you didn't download Mellanox OpenStack repo file download it :
#wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver
security_group_api=nova
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only releavant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''vnic_type''' - type of VM network interface: 'mlnx_direct' or 'hostdev' according to libvirt terminology.
hostdev: this is the traditional method of assigning any generic PCI device to a guest (SR-IOV).
mlnx_direct: this is a method to provide macvtap device on top of the PCI device (default).
bridge: - When using Linux Bridge Plugin on top of eIPoIB device
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:5001')
daemon_endpoint should be changed to tcp://127.0.0.1:60001
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

'''Note''': daemon_endpoint should be changed to tcp://127.0.0.1:60001

For a plugin configuration file example (Havana), please refer to [https://github.com/openstack/neutron/blob/stable/havana/etc/neutron/plugins/mlnx/mlnx_conf.inii Mellanox config *ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron linux bridge plugin and Neutron dhcp l3-agent:

#yum install openstack-neutron-linuxbridge

2. Change the following configuration of the ini file (/etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .

physical_interface_mappings = default:eth2

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-linuxbridge-agent start
#/etc/init.d/neutron-dhcp-agent start

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

/etc/init.d/ufmd restart

High-availability

/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
/etc/init.d/eswitchd restart
/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd
'''Note:''' If you have OFED >= 2.1 the above step is not needed

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-05-25T14:00:11Z

Itzikb: /* Installation */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA.
Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron pluign.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types.
For vnic type configuration API details, please refer to configuration reference guide http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html.
Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

* Mellanox Openstack Neutron Agent (L2 Agent) runs on each compute node.
* Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.0.3 or greater is installed.Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. The software package ethtool ([http://www.kernel.org/pub/software/network/ethtool/ Code]) must be installed (version 3.8 or higher). Required only to be installed on compute nodes.

7. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.0.3 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at /etc/neutron/plugins/ml2/ml2_conf.ini

3.Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = mlnx
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. If you didn't download Mellanox OpenStack repo file download it :
#http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver
security_group_api=nova
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only releavant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''vnic_type''' - type of VM network interface: 'mlnx_direct' or 'hostdev' according to libvirt terminology.
hostdev: this is the traditional method of assigning any generic PCI device to a guest (SR-IOV).
mlnx_direct: this is a method to provide macvtap device on top of the PCI device (default).
bridge: - When using Linux Bridge Plugin on top of eIPoIB device
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:5001')
daemon_endpoint should be changed to tcp://127.0.0.1:60001
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

'''Note''': daemon_endpoint should be changed to tcp://127.0.0.1:60001

For a plugin configuration file example (Havana), please refer to [https://github.com/openstack/neutron/blob/stable/havana/etc/neutron/plugins/mlnx/mlnx_conf.inii Mellanox config *ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron linux bridge plugin and Neutron dhcp l3-agent:

#yum install openstack-neutron-linuxbridge

2. Change the following configuration of the ini file (/etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .

physical_interface_mappings = default:eth2

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-linuxbridge-agent start
#/etc/init.d/neutron-dhcp-agent start

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

/etc/init.d/ufmd restart

High-availability

/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
/etc/init.d/eswitchd restart
/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd
'''Note:''' If you have OFED >= 2.1 the above step is not needed

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-05-25T13:59:03Z

Itzikb: /* Start Services */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA.
Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron pluign.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types.
For vnic type configuration API details, please refer to configuration reference guide http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html.
Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

* Mellanox Openstack Neutron Agent (L2 Agent) runs on each compute node.
* Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.0.3 or greater is installed.Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. The software package ethtool ([http://www.kernel.org/pub/software/network/ethtool/ Code]) must be installed (version 3.8 or higher). Required only to be installed on compute nodes.

7. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.0.3 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at /etc/neutron/plugins/ml2/ml2_conf.ini

3.Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = mlnx
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. If you didn't download Mellanox OpenStack repo file download it :
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/repo/mlnx-havana/mlnx-havana.repo -O /etc/yum.repos.d/mlnx-havana.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver
security_group_api=nova
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only releavant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''vnic_type''' - type of VM network interface: 'mlnx_direct' or 'hostdev' according to libvirt terminology.
hostdev: this is the traditional method of assigning any generic PCI device to a guest (SR-IOV).
mlnx_direct: this is a method to provide macvtap device on top of the PCI device (default).
bridge: - When using Linux Bridge Plugin on top of eIPoIB device
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:5001')
daemon_endpoint should be changed to tcp://127.0.0.1:60001
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

'''Note''': daemon_endpoint should be changed to tcp://127.0.0.1:60001

For a plugin configuration file example (Havana), please refer to [https://github.com/openstack/neutron/blob/stable/havana/etc/neutron/plugins/mlnx/mlnx_conf.inii Mellanox config *ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron linux bridge plugin and Neutron dhcp l3-agent:

#yum install openstack-neutron-linuxbridge

2. Change the following configuration of the ini file (/etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .

physical_interface_mappings = default:eth2

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-linuxbridge-agent start
#/etc/init.d/neutron-dhcp-agent start

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

/etc/init.d/ufmd restart

High-availability

/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
/etc/init.d/eswitchd restart
/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd
'''Note:''' If you have OFED >= 2.1 the above step is not needed

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-05-25T13:58:47Z

Itzikb: /* Installation */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA.
Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron pluign.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types.
For vnic type configuration API details, please refer to configuration reference guide http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html.
Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

* Mellanox Openstack Neutron Agent (L2 Agent) runs on each compute node.
* Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.0.3 or greater is installed.Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. The software package ethtool ([http://www.kernel.org/pub/software/network/ethtool/ Code]) must be installed (version 3.8 or higher). Required only to be installed on compute nodes.

7. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.0.3 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at /etc/neutron/plugins/ml2/ml2_conf.ini

3.Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = mlnx
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#/etc/init.d/neutron-server start

== Compute Nodes ==

=== Installation ===
1. If you didn't download Mellanox OpenStack repo file download it :
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/repo/mlnx-havana/mlnx-havana.repo -O /etc/yum.repos.d/mlnx-havana.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver
security_group_api=nova
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only releavant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''vnic_type''' - type of VM network interface: 'mlnx_direct' or 'hostdev' according to libvirt terminology.
hostdev: this is the traditional method of assigning any generic PCI device to a guest (SR-IOV).
mlnx_direct: this is a method to provide macvtap device on top of the PCI device (default).
bridge: - When using Linux Bridge Plugin on top of eIPoIB device
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:5001')
daemon_endpoint should be changed to tcp://127.0.0.1:60001
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

'''Note''': daemon_endpoint should be changed to tcp://127.0.0.1:60001

For a plugin configuration file example (Havana), please refer to [https://github.com/openstack/neutron/blob/stable/havana/etc/neutron/plugins/mlnx/mlnx_conf.inii Mellanox config *ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron linux bridge plugin and Neutron dhcp l3-agent:

#yum install openstack-neutron-linuxbridge

2. Change the following configuration of the ini file (/etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .

physical_interface_mappings = default:eth2

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-linuxbridge-agent start
#/etc/init.d/neutron-dhcp-agent start

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

/etc/init.d/ufmd restart

High-availability

/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
/etc/init.d/eswitchd restart
/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd
'''Note:''' If you have OFED >= 2.1 the above step is not needed

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-05-25T13:58:18Z

Itzikb: /* Installation */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA.
Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron pluign.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types.
For vnic type configuration API details, please refer to configuration reference guide http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html.
Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

* Mellanox Openstack Neutron Agent (L2 Agent) runs on each compute node.
* Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.0.3 or greater is installed.Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. The software package ethtool ([http://www.kernel.org/pub/software/network/ethtool/ Code]) must be installed (version 3.8 or higher). Required only to be installed on compute nodes.

7. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.0.3 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at /etc/neutron/plugins/ml2/ml2_conf.ini

3.Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = mlnx
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

4.Restart Neutron server
service neutron-server restart

=== Start Services ===
Start (or restart) the Neutron server
#/etc/init.d/neutron-server start

== Compute Nodes ==

=== Installation ===
1. If you didn't download Mellanox OpenStack repo file download it :
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/repo/mlnx-havana/mlnx-havana.repo -O /etc/yum.repos.d/mlnx-havana.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver
security_group_api=nova
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only releavant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''vnic_type''' - type of VM network interface: 'mlnx_direct' or 'hostdev' according to libvirt terminology.
hostdev: this is the traditional method of assigning any generic PCI device to a guest (SR-IOV).
mlnx_direct: this is a method to provide macvtap device on top of the PCI device (default).
bridge: - When using Linux Bridge Plugin on top of eIPoIB device
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:5001')
daemon_endpoint should be changed to tcp://127.0.0.1:60001
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

'''Note''': daemon_endpoint should be changed to tcp://127.0.0.1:60001

For a plugin configuration file example (Havana), please refer to [https://github.com/openstack/neutron/blob/stable/havana/etc/neutron/plugins/mlnx/mlnx_conf.inii Mellanox config *ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron linux bridge plugin and Neutron dhcp l3-agent:

#yum install openstack-neutron-linuxbridge

2. Change the following configuration of the ini file (/etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .

physical_interface_mappings = default:eth2

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-linuxbridge-agent start
#/etc/init.d/neutron-dhcp-agent start

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

/etc/init.d/ufmd restart

High-availability

/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
/etc/init.d/eswitchd restart
/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd
'''Note:''' If you have OFED >= 2.1 the above step is not needed

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.