OpenStack - User contributions [en]

Mellanox-Neutron-ML2-Kilo

2015-07-22T22:13:30Z

Ophir Maor:

Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) vnic type. For vnic type configuration API details, please refer to configuration reference guide (click here). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.
* Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.
* Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

For '''Kilo''' release Information refer to the relevant OS as follows:

Infiniband
* [[Mellanox-Neutron-Kilo-Redhat-InfiniBand | Redhat7]]
* [[Mellanox-Neutron-Kilo-Ubuntu-InfiniBand | Ubuntu]]

Ethernet(SR-IOV)
* [[Mellanox-Neutron-Kilo-Redhat-Ethernet |Redhat7/Ubuntu ]]

Mellanox-Neutron-Icehouse-Ubuntu

2014-11-30T07:20:39Z

Ophir Maor: Replaced content with "This page was redirected to [https://wiki.openstack.org/wiki/Mellanox-Neutron-ML2 here]"

This page was redirected to [https://wiki.openstack.org/wiki/Mellanox-Neutron-ML2 here]

Mellanox-Neutron-Icehouse-Ubuntu

2014-11-30T07:19:06Z

Ophir Maor:

This page was redirected to [https://wiki.openstack.org/wiki/Mellanox-Neutron-ML2 here]

= Overview =

This wiki page describes how to tune Mellanox OpenStack on Ubuntu 12.04 servers

= Prerequisites =

You need at least 3 nodes:
* Controller node running the neutron-server service.
* Compute node running the eswitchd, nova-compute & neutron-plugin-mlnx-agent services.
* Network node running the neutron-plugin-linuxbridge-agent, neutron-dhcp-agent and neutron-l3-agent services.

= Ethernet Network =

== Controller Node ==

In /etc/nova/nova.conf ensure you have the following line:
security_group_api=nova

If you change, restart the nova services
# for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done

== Neutron Server Node ==
=== Installation ===

1. Neutron server uses MySQL database. Make sure you have running MySQL database for neutron.

If Neutron server is already running, you should stop it.

#/etc/init.d/neutron-server stop

If you want to use a database that was created you have to drop it and create another one. For example:

mysql -u root -ppassword <<EOF
drop database if exists neutron;
create database neutron;
EOF

If you want to create a new one:

mysql -u root -ppassword <<EOF
drop database if exists neutron;
create database neutron;
GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON neutron.* TO 'neutron' IDENTIFIED BY 'password';
FLUSH PRIVILEGES;
EOF

2. Install the required Deb for the Mellanox Neutron plugin:

apt-get install -y neutron-plugin-mlnx-agent

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

4. Change /etc/default/neutron-server :

NEUTRON_PLUGIN_CONFIG="/etc/neutron/plugins/mlnx/mlnx_conf.ini"

=== Configuration ===

1. Make the Mellanox plugin the current Neutron plugin by editing neutron.conf and changing core_plugin.

core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Plugin configuration: Edit the configuration file: /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
tenant_network_type - must be set on of supported tenant network types.
Possible values: 'vlan' for Ethernet or 'ib' for Infiniband.'vlan' is the default.
network_vlan_ranges - must be configured to specify the names of the physical

networks managed by the mellanox plugin,
along with the ranges of VLAN IDs available on each physical network for allocation to virtual networks.
Possible range is vlan range is 1-4093.
The default is "default:1:100"
<fabric name >:<vlan range start>:<vlan end range>

[agent]
rpc_support_old_agents = True
Note: rpc_support_old_agents should be set to 'True' (non default).

=== Start Services ===

Start (or restart) the Neutron server
#/etc/init.d/neutron-server start

== Compute Nodes ==

=== Installation ===

1. If you didn't download Mellanox OpenStack debian files download it :

http://www.mellanox.com/downloads/solutions/openstack/

2. Install the eswitchd debian files:
dpkg -i eswitchd_0.8-1_amd64.deb
dpkg -i python-mlnxvif_0.6-1_all.deb
apt-get install -f # fix dependencies
apt-get install -y neutron-plugin-mlnx-agent

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver
security_group_api=nova
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:

libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.
[AGENT]
rpc_support_old_agents = True
[ESWITCH]
physical_interface_mapping - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only releavant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
vnic_type - type of VM network interface: 'mlnx_direct' or 'hostdev' according to libvirt terminology.
hostdev: this is the traditional method of assigning any generic PCI device to a guest (SR-IOV).
mlnx_direct: this is a method to provide macvtap device on top of the PCI device (default).
bridge: - When using Linux Bridge Plugin on top of eIPoIB device
daemon_endpoint - 'tcp://127.0.0.1:60001'

For a plugin configuration file example (Havana), please refer to Mellanox config *ini file (click [https://github.com/openstack/neutron/blob/stable/havana/etc/neutron/plugins/mlnx/mlnx_conf.ini here])

4. (Optional) Modify /etc/init/neutron-plugin-mlnx-agent.conf as described in LP bug #[https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1257732 1257732]

5. Check that ebrctl is listed somewhere in /etc/nova/rootwrap.d/*, otherwise add this to /etc/nova/rootwrap.d/compute.filters file in the [Filters] section.
ebrctl: CommandFilter, ebrctl, root

=== Start Services ===

1. Restart Nova.

#/etc/init.d/nova-compute restart

2. Start eswitch Daemon

#/etc/init.d/eswitchd start

3. Start the Neutron agent

#/etc/init.d/neutron-plugin-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install the Neutron Linux bridge plugin, the Neutron DHCP agent and the Neutron L3 agent:

# apt-get install neutron-plugin-linuxbridge-agent neutron-dhcp-agent neutron-l3-agent

2. Copy /etc/neutron/neutron.conf from the Compute node.

3. Change the following configuration of the ini file (/etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .
[linux_bridge]
physical_interface_mappings = default:eth2

4. Configure the DHCP agent according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
[DEFAULT]
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_dhcp_agent OpenStack documentation]

5. Configure the L3 agent according to the following guidelines

Update the following file: /etc/neutron/l3_agent.ini with:
[DEFAULT]
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_l3_agent OpenStack documentation]

6. Configure the Metadata agent according to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#nova_config_metadata OpenStack documentation]

7. Start the Neutron Linux Bridge, DHCP and L3 agents
# restart neutron-plugin-linuxbridge-agent
# restart neutron-dhcp-agent
# restart neutron-l3-agent
# restart neutron-metadata-agent

== Troubleshooting ==

* If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.

Mellanox-Neutron-ML2-Icehouse1

2014-11-17T11:57:05Z

Ophir Maor:

Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

* Mellanox Openstack Neutron Agent (L2 Agent) runs on each compute node.
* Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

For '''Icehouse''' release Information refer to the relevant OS as follows:

* [[Mellanox-Neutron-Icehouse-Ubuntu|Ubuntu]]
* [[Mellanox-Neutron-Icehouse-Redhat|RedHat]]

* [[Mellanox-Neutron-Icehouse-Redhat-Ethernet|Redhat Ethernet]]
* [[Mellanox-Neutron-Icehouse-Ubuntu-Ethernet|Ubuntu Ethernet]]
* [[Mellanox-Neutron-Icehouse-Redhat-InfiniBand|Redhat InfiniBand]]
* [[Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand|Ubuntu InfiniBand]]

Mellanox-Neutron-ML2-Icehouse1

2014-11-17T11:56:51Z

Ophir Maor:

Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

* Mellanox Openstack Neutron Agent (L2 Agent) runs on each compute node.
* Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

For '''Icehouse''' release Information refer to the relevant OS as follows:

* [[Mellanox-Neutron-Icehouse-Ubuntu|Ubuntu]]
* [[Mellanox-Neutron-Icehouse-Redhat|RedHat]]

* [[Mellanox-Neutron-Icehouse-Redhat-Ethernet|Redhat Ethernet]]
* [[Mellanox-Neutron-Icehouse-Ubuntu-Ethernet|Ubuntu Ethernet]]
* [[Mellanox-Neutron-Icehouse-Redhat-InfiniBand|Redhat InfiniBand]]
* [[Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand|Ubuntu InfiniBand]]

Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand

2014-11-17T11:56:35Z

Ophir Maor: Created page with "test"

test

Mellanox-Neutron-Icehouse-Redhat-InfiniBand

2014-11-17T11:55:53Z

Ophir Maor: Created page with "test"

test

Mellanox-Neutron-ML2-Icehouse1

2014-11-17T11:55:39Z

Ophir Maor:

Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

* Mellanox Openstack Neutron Agent (L2 Agent) runs on each compute node.
* Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

For '''Icehouse''' release Information refer to the relevant OS as follows:

* [[Mellanox-Neutron-Icehouse-Ubuntu|Ubuntu]]
* [[Mellanox-Neutron-Icehouse-Redhat|RedHat]]

* [[Mellanox-Neutron-Icehouse-Redhat-Ethernet|Redhat Ethernet]]
* [[Mellanox-Neutron-Icehouse-Ubuntu-Ethernet|Ubuntu Ethernet]]

* [[Mellanox-Neutron-Icehouse-Redhat-InfiniBand|Redhat InfiniBand]]
* [[Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand|Ubuntu InfiniBand]]

Mellanox-Neutron-Icehouse-Ubuntu-Ethernet

2014-11-17T11:54:00Z

Ophir Maor: Created page with "test"

test

Talk:Mellanox

2014-11-17T11:52:32Z

Ophir Maor: Created page with "test"

test

Mellanox-OpenStack

2014-11-17T11:45:45Z

Ophir Maor: /* References */

= Mellanox Neutron Plugin =
Mellanox supports the OpenStack Neutron releases with open source networking components through Neutron. It delivers higher compute and storage performance and additional functionality, such as NIC based switching to provide better security and isolation for virtual cloud environments.

For additional information about Mellanox NeutronPlugin refer to [https://wiki.openstack.org/wiki/Mellanox-Neutron Mellanox Neutron] wiki page.

= Mellanox Cinder Plugin =
Mellanox provides an OpenStack Storage plugin that enables running storage volume traffic over native RDMA transport, providing 5X faster throughput over standard TCP/IP, while significantly increasing infrastructure efficiency by using less CPU resources.

For additional information about Mellanox Cinder refer to [https://wiki.openstack.org/wiki/Mellanox-Cinder Mellanox Cinder] wiki page.

= References =
1. [http://www.mellanox.com/openstack/ OpenStack solution page at Mellanox site]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED web page]

4. [http://community.mellanox.com/community/solutions Mellanox Community]

5. [https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking SR-IOV Passthrough for Networking]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Mellanox

2014-11-17T11:45:00Z

Ophir Maor:

This page has been redirected to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack]

Mellanox

2014-11-17T11:44:43Z

Ophir Maor:

This page has been redirected to [https://wiki.openstack.org/wiki/Mellanox-OpenStack | Mellanox-OpenStack]

Mellanox

2014-11-17T11:44:15Z

Ophir Maor:

This page has been redirected to [https://wiki.openstack.org/wiki/Mellanox-OpenStack]

Mellanox-Neutron-Icehouse-Redhat-Ethernet

2014-11-17T11:41:53Z

Ophir Maor:

test

# command

Mellanox-Neutron-Icehouse-Redhat-Ethernet

2014-11-17T11:41:24Z

Ophir Maor:

test

ddd

Mellanox-Neutron-Icehouse-Redhat-Ethernet

2014-11-17T11:40:59Z

Ophir Maor:

test

ddd

Mellanox-Neutron-ML2-Icehouse1

2014-11-17T11:39:59Z

Ophir Maor:

Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

* Mellanox Openstack Neutron Agent (L2 Agent) runs on each compute node.
* Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

For '''Icehouse''' release Information refer to the relevant OS as follows:

* [[Mellanox-Neutron-Icehouse-Ubuntu|Ubuntu]]
* [[Mellanox-Neutron-Icehouse-Redhat|RedHat]]

* [[Mellanox-Neutron-Icehouse-Redhat-Ethernet|Redhat Ethernet]]

Mellanox-Neutron-Icehouse-Redhat-Ethernet

2014-11-17T11:38:25Z

Ophir Maor: Created page with "test"

test

Mellanox-Cinder

2014-11-17T09:28:10Z

Ophir Maor: /* Grizzly Release */

= Overview =
iSER (iSCSI over RDMA) Mellanox OpenStack support to Cinder. 
This can allow 5x faster bandwidth compared to using iSCSI TCP. 
For example, over RAM device LUN I got ~1.3GBps Vs. ~5.5GBps (TCP Vs. iSER), and much lower CPU overhead.

= Havana Release =

== Installation ==
The Mellanox Cinder pacakge is embeded within the OpenStack Cinder, no patches or plugins are needed.
Just follow the formal OpenStack Cinder installation

== Configuration ==
In order to enable iSER, need to adjust these values at '''/etc/cinder/cinder.conf''':

iser_ip_address = <ipoib/roce_address>
volume_driver = cinder.volume.drivers.lvm.LVMISERDriver

'''iser_ip_address''' is required to do a "discovery" over the IB/RoCE interface from the initiator side.

'''volume_driver''' points cinder to use the ISERDriver, instead the LVMISCSIDriver.

On nova-compute side change the following in '''/etc/nova/nova.conf''':

libvirt_volume_drivers = iser=nova.virt.libvirt.volume.LibvirtISERVolumeDriver

Restart Cinder and Nova compute services

#/etc/init.d/openstack-cinder-volume restart
#/etc/init.d/openstack-nova-compute restart

For troubleshooting issues refer to section [https://wiki.openstack.org/wiki/Mellanox-Cinder#Known_Issues Known Issues]

= Known Issues =

(1) “scsi-target-utils” package

Note: for RH6.4 or below: “scsi-target-utils” package needs to be 1.0.38 and up, if you use RDO installation you should check the version.
The RPM can be found in http://www.mellanox.com/downloads/solutions/rpms/

#wget http://www.mellanox.com/downloads/solutions/rpms/scsi-target-utils-1.0.39-v1.0.39.c1135a.x86_64.rpm
#rpm -Uvh scsi-target-utils-1.0.39-v1.0.39.c1135a.x86_64.rpm

In addition, RH6.5 has “scsi-target-utils” package with version 1.0.24-10, which is good enough, no further action is needed for RHEL6.5

(2) Boot Instance from volume fails when iSER is enabled

Steps to reproduce the bug:

- Create new volume with image source.

- Launch a new instance boot from the above volume using iSER transport.

The bug is defined here:
[https://bugs.launchpad.net/nova/+bug/1253578]

A patch can be found in here: [http://www.mellanox.com/downloads/solutions/openstack/havana/patches/volume.patch]

To apply the patch run the following commands on '''all nova-compute nodes''':
# wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/volume.patch
# patch -p1 < volume.patch
# service nova-compute restart

(3) Flow control

In case the network is Ethernet, Make sure that flow control is enabled on the switches on the relevant ports across the network.

= References =
1. [http://www.mellanox.com/openstack/ OpenStack solution page at Mellanox site]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED web page]

4. [http://docs.openstack.org/havana/install-guide/install/yum/content/cinder-controller.html Cinder-controller]

5. [http://docs.openstack.org/havana/install-guide/install/yum/content/cinder-node.html Cinder-node]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

For more details, please refer any inquiries to [mailto:openstack@mellanox.com openstack@mellanox.com].

Mellanox-Neutron-Icehouse-Ubuntu

2014-10-20T06:55:32Z

Ophir Maor: /* Page is under construction */

= Overview =

This wiki page describes how to tune Mellanox OpenStack on Ubuntu 12.04 servers

= Prerequisites =

You need at least 3 nodes:
* Controller node running the neutron-server service.
* Compute node running the eswitchd, nova-compute & neutron-plugin-mlnx-agent services.
* Network node running the neutron-plugin-linuxbridge-agent, neutron-dhcp-agent and neutron-l3-agent services.

= Ethernet Network =

== Controller Node ==

In /etc/nova/nova.conf ensure you have the following line:
security_group_api=nova

If you change, restart the nova services
# for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done

== Neutron Server Node ==
=== Installation ===

1. Neutron server uses MySQL database. Make sure you have running MySQL database for neutron.

If Neutron server is already running, you should stop it.

#/etc/init.d/neutron-server stop

If you want to use a database that was created you have to drop it and create another one. For example:

mysql -u root -ppassword <<EOF
drop database if exists neutron;
create database neutron;
EOF

If you want to create a new one:

mysql -u root -ppassword <<EOF
drop database if exists neutron;
create database neutron;
GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON neutron.* TO 'neutron' IDENTIFIED BY 'password';
FLUSH PRIVILEGES;
EOF

2. Install the required Deb for the Mellanox Neutron plugin:

apt-get install -y neutron-plugin-mlnx-agent

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

4. Change /etc/default/neutron-server :

NEUTRON_PLUGIN_CONFIG="/etc/neutron/plugins/mlnx/mlnx_conf.ini"

=== Configuration ===

1. Make the Mellanox plugin the current Neutron plugin by editing neutron.conf and changing core_plugin.

core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Plugin configuration: Edit the configuration file: /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
tenant_network_type - must be set on of supported tenant network types.
Possible values: 'vlan' for Ethernet or 'ib' for Infiniband.'vlan' is the default.
network_vlan_ranges - must be configured to specify the names of the physical

networks managed by the mellanox plugin,
along with the ranges of VLAN IDs available on each physical network for allocation to virtual networks.
Possible range is vlan range is 1-4093.
The default is "default:1:100"
<fabric name >:<vlan range start>:<vlan end range>

[agent]
rpc_support_old_agents = True
Note: rpc_support_old_agents should be set to 'True' (non default).

=== Start Services ===

Start (or restart) the Neutron server
#/etc/init.d/neutron-server start

== Compute Nodes ==

=== Installation ===

1. If you didn't download Mellanox OpenStack debian files download it :

http://www.mellanox.com/downloads/solutions/openstack/

2. Install the eswitchd debian files:
dpkg -i eswitchd_0.8-1_amd64.deb
dpkg -i python-mlnxvif_0.6-1_all.deb
apt-get install -f # fix dependencies
apt-get install -y neutron-plugin-mlnx-agent

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver
security_group_api=nova
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:

libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.
[AGENT]
rpc_support_old_agents = True
[ESWITCH]
physical_interface_mapping - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only releavant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
vnic_type - type of VM network interface: 'mlnx_direct' or 'hostdev' according to libvirt terminology.
hostdev: this is the traditional method of assigning any generic PCI device to a guest (SR-IOV).
mlnx_direct: this is a method to provide macvtap device on top of the PCI device (default).
bridge: - When using Linux Bridge Plugin on top of eIPoIB device
daemon_endpoint - 'tcp://127.0.0.1:60001'

For a plugin configuration file example (Havana), please refer to Mellanox config *ini file (click [https://github.com/openstack/neutron/blob/stable/havana/etc/neutron/plugins/mlnx/mlnx_conf.ini here])

4. (Optional) Modify /etc/init/neutron-plugin-mlnx-agent.conf as described in LP bug #[https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1257732 1257732]

5. Check that ebrctl is listed somewhere in /etc/nova/rootwrap.d/*, otherwise add this to /etc/nova/rootwrap.d/compute.filters file in the [Filters] section.
ebrctl: CommandFilter, ebrctl, root

=== Start Services ===

1. Restart Nova.

#/etc/init.d/nova-compute restart

2. Start eswitch Daemon

#/etc/init.d/eswitchd start

3. Start the Neutron agent

#/etc/init.d/neutron-plugin-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install the Neutron Linux bridge plugin, the Neutron DHCP agent and the Neutron L3 agent:

# apt-get install neutron-plugin-linuxbridge-agent neutron-dhcp-agent neutron-l3-agent

2. Copy /etc/neutron/neutron.conf from the Compute node.

3. Change the following configuration of the ini file (/etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .
[linux_bridge]
physical_interface_mappings = default:eth2

4. Configure the DHCP agent according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
[DEFAULT]
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_dhcp_agent OpenStack documentation]

5. Configure the L3 agent according to the following guidelines

Update the following file: /etc/neutron/l3_agent.ini with:
[DEFAULT]
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_l3_agent OpenStack documentation]

6. Configure the Metadata agent according to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#nova_config_metadata OpenStack documentation]

7. Start the Neutron Linux Bridge, DHCP and L3 agents
# restart neutron-plugin-linuxbridge-agent
# restart neutron-dhcp-agent
# restart neutron-l3-agent
# restart neutron-metadata-agent

== Troubleshooting ==

* If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.

Mellanox-Neutron-Icehouse-Redhat

2014-10-20T06:54:51Z

Ophir Maor: /* Page is under construction */

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

You can do it by using packstack:

1.Create an answer file
#packstack --gen-answer-file=GEN_ANSWER_FILE

2. Change the following in GEN_ANSWER_FILE
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_VLAN_RANGES=default:2:10
CONFIG_NEUTRON_L2_AGENT=openvswitch

3. Run packstack
#packstack --answer-file=GEN_ANSWER_FILE

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = openvswitch,mlnx
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. In /etc/nova/nova.conf

Check that the compute driver is libvirt:
[DEFAULT]
compute_driver=libvirt.LibvirtDriver

Change the VIF driver:
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized mode make sure vif driver is as follows:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc_support_old_agents''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#service openstack-nova-compute restart

2. Start eswitch Daemon
#service eswitchd start

3. Start the Neutron agent
#service neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/ml2/ml2_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#service neutron-openvswitch-agent start
#service neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#service neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#service opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==
We are using the linuxbridge mechanism driver so we can use the DHCP Server with the Linux Bridge interface driver.

Edit /etc/neutron/plugins/ml2/ml2_conf.ini as follows (The VLAN range is an example)

[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = linuxbridge,mlnx
[ml2_type_vlan]
network_vlan_ranges=default:2:10
[securitygroup]
enable_security_group = True
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

===Configuration===

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#service openibd restart
eswitchd should be started and then Neutron agent should be started
#service eswitchd restart
#service neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#service openstack-nova-compute restart

=== Start Services ===

1. Restart Nova.
#service openstack-nova-compute restart

2. Start eswitch Daemon
#service eswitchd start

3. Start the Neutron agent
#service neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:

E_IPOIB_LOAD=yes
And restart openibd:

#service openibd restart
Please refer to eIPoIB configuration in Mellanox OFED User Manual
Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example: Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")

#ethtool -i <interface>
driver: eth_ipoib
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Restart neutron-linuxbridge-agent and neutron-dhcp-agent

#service neutron-linuxbridge-agent restart

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-dnsmasq-2014.1.1-1.noarch.rpm
# yum localinstall mlnx-dnsmasq-2014.1.1-1.noarch.rpm

In addition, dnsmasq must be upgraded to version 2.66 or higher.
#wget ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/kalyaka/CentOS_CentOS-6/x86_64/dnsmasq-2.66-3.1.x86_64.rpm
#yum localinstall dnsmasq-2.66-3.1.x86_64.rpm

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Start dhcp server
# service neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-10-06T07:59:14Z

Ophir Maor: /* OpenSM configuration - Without UFM */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

You can do it by using packstack:

1.Create an answer file
#packstack --gen-answer-file=GEN_ANSWER_FILE

2. Change the following in GEN_ANSWER_FILE
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_VLAN_RANGES=default:2:10
CONFIG_NEUTRON_L2_AGENT=openvswitch

3. Run packstack
#packstack --answer-file=GEN_ANSWER_FILE

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = openvswitch,mlnx
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. In /etc/nova/nova.conf

Check that the compute driver is libvirt:
[DEFAULT]
compute_driver=libvirt.LibvirtDriver

Change the VIF driver:
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized mode make sure vif driver is as follows:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc_support_old_agents''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#service openstack-nova-compute restart

2. Start eswitch Daemon
#service eswitchd start

3. Start the Neutron agent
#service neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/ml2/ml2_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#service neutron-openvswitch-agent start
#service neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#service neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#service opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==
We are using the linuxbridge mechanism driver so we can use the DHCP Server with the Linux Bridge interface driver.

Edit /etc/neutron/plugins/ml2/ml2_conf.ini as follows (The VLAN range is an example)

[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = linuxbridge,mlnx
[ml2_type_vlan]
network_vlan_ranges=default:2:10
[securitygroup]
enable_security_group = True
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

===Configuration===

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#service openibd restart
eswitchd should be started and then Neutron agent should be started
#service eswitchd restart
#service neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#service openstack-nova-compute restart

=== Start Services ===

1. Restart Nova.
#service openstack-nova-compute restart

2. Start eswitch Daemon
#service eswitchd start

3. Start the Neutron agent
#service neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:

E_IPOIB_LOAD=yes
And restart openibd:

#service openibd restart
Please refer to eIPoIB configuration in Mellanox OFED User Manual
Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example: Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")

#ethtool -i <interface>
driver: eth_ipoib
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Restart neutron-linuxbridge-agent and neutron-dhcp-agent

#service neutron-linuxbridge-agent restart

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-dnsmasq-2014.1.1-1.noarch.rpm
# yum localinstall mlnx-dnsmasq-2014.1.1-1.noarch.rpm

In addition, dnsmasq must be upgraded to version 2.66 or higher.
#wget ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/kalyaka/CentOS_CentOS-6/x86_64/dnsmasq-2.66-3.1.x86_64.rpm
#yum localinstall dnsmasq-2.66-3.1.x86_64.rpm

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Start dhcp server
# service neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Obsolete:Mellanox-CloudX

2014-09-17T18:25:04Z

Ophir Maor: Ophir Maor moved page Mellanox-CloudX to Obsolete:Mellanox-CloudX

Mellanox-CloudX

2014-09-17T18:25:04Z

Ophir Maor: Ophir Maor moved page Mellanox-CloudX to Obsolete:Mellanox-CloudX

#REDIRECT [[Obsolete:Mellanox-CloudX]]

Obsolete:Mellanox-OpenStack-CloudX

2014-09-17T18:24:28Z

Ophir Maor: Ophir Maor moved page Mellanox-OpenStack-CloudX to Obsolete:Mellanox-OpenStack-CloudX

Mellanox-OpenStack-CloudX

2014-09-17T18:24:28Z

Ophir Maor: Ophir Maor moved page Mellanox-OpenStack-CloudX to Obsolete:Mellanox-OpenStack-CloudX

#REDIRECT [[Obsolete:Mellanox-OpenStack-CloudX]]

Obsolete:Mellanox-OpenStack-CloudX

2014-09-17T18:23:54Z

Ophir Maor: /* Overview */

Obsolete:Mellanox-CloudX

2014-09-17T18:23:09Z

Ophir Maor: Blanked the page

Mellanox-Neutron-Icehouse-Redhat

2014-09-16T08:35:27Z

Ophir Maor: /* Network Node */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

You can do it by using packstack:

1.Create an answer file
#packstack --gen-answer-file=GEN_ANSWER_FILE

2. Change the following in GEN_ANSWER_FILE
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_VLAN_RANGES=default:2:10
CONFIG_NEUTRON_L2_AGENT=openvswitch

3. Run packstack
#packstack --answer-file=GEN_ANSWER_FILE

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = openvswitch,mlnx
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. In /etc/nova/nova.conf

Check that the compute driver is libvirt:
[DEFAULT]
compute_driver=libvirt.LibvirtDriver

Change the VIF driver:
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized mode make sure vif driver is as follows:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc_support_old_agents''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#service openstack-nova-compute restart

2. Start eswitch Daemon
#service eswitchd start

3. Start the Neutron agent
#service neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/ml2/ml2_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#service neutron-openvswitch-agent start
#service neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#service neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#service opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==
We are using the linuxbridge mechanism driver so we can use the DHCP Server with the Linux Bridge interface driver.

Edit /etc/neutron/plugins/ml2/ml2_conf.ini as follows (The VLAN range is an example)

[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = linuxbridge,mlnx
[ml2_type_vlan]
network_vlan_ranges=default:2:10
[securitygroup]
enable_security_group = True
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

===Configuration===

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#service openibd restart
eswitchd should be started and then Neutron agent should be started
#service eswitchd restart
#service neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#service openstack-nova-compute restart

=== Start Services ===

1. Restart Nova.
#service openstack-nova-compute restart

2. Start eswitch Daemon
#service eswitchd start

3. Start the Neutron agent
#service neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:

E_IPOIB_LOAD=yes
And restart openibd:

#service openibd restart
Please refer to eIPoIB configuration in Mellanox OFED User Manual
Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example: Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")

#ethtool -i <interface>
driver: eth_ipoib
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Restart neutron-linuxbridge-agent and neutron-dhcp-agent

#service neutron-linuxbridge-agent restart

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-dnsmasq-2014.1.1-1.noarch.rpm
# yum localinstall mlnx-dnsmasq-2014.1.1-1.noarch.rpm

In addition, dnsmasq must be upgraded to version 2.66 or higher.
#wget ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/kalyaka/CentOS_CentOS-6/x86_64/dnsmasq-2.66-3.1.x86_64.rpm
#yum localinstall dnsmasq-2.66-3.1.x86_64.rpm

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Start dhcp server
# service neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-09-16T08:34:34Z

Ophir Maor: /* Compute Nodes */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

You can do it by using packstack:

1.Create an answer file
#packstack --gen-answer-file=GEN_ANSWER_FILE

2. Change the following in GEN_ANSWER_FILE
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_VLAN_RANGES=default:2:10
CONFIG_NEUTRON_L2_AGENT=openvswitch

3. Run packstack
#packstack --answer-file=GEN_ANSWER_FILE

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = openvswitch,mlnx
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. In /etc/nova/nova.conf

Check that the compute driver is libvirt:
[DEFAULT]
compute_driver=libvirt.LibvirtDriver

Change the VIF driver:
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized mode make sure vif driver is as follows:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc_support_old_agents''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#service openstack-nova-compute restart

2. Start eswitch Daemon
#service eswitchd start

3. Start the Neutron agent
#service neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/ml2/ml2_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#service neutron-openvswitch-agent start
#service neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#service neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#service opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==
We are using the linuxbridge mechanism driver so we can use the DHCP Server with the Linux Bridge interface driver.

Edit /etc/neutron/plugins/ml2/ml2_conf.ini as follows (The VLAN range is an example)

[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = linuxbridge,mlnx
[ml2_type_vlan]
network_vlan_ranges=default:2:10
[securitygroup]
enable_security_group = True
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

===Configuration===

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#service openibd restart
eswitchd should be started and then Neutron agent should be started
#service eswitchd restart
#service neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#service openstack-nova-compute restart

== Network Node ==

Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:

E_IPOIB_LOAD=yes
And restart openibd:

#service openibd restart
Please refer to eIPoIB configuration in Mellanox OFED User Manual
Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example: Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")

#ethtool -i <interface>
driver: eth_ipoib
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Restart neutron-linuxbridge-agent and neutron-dhcp-agent

#service neutron-linuxbridge-agent restart

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-dnsmasq-2014.1.1-1.noarch.rpm
# yum localinstall mlnx-dnsmasq-2014.1.1-1.noarch.rpm

In addition, dnsmasq must be upgraded to version 2.66 or higher.
#wget ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/kalyaka/CentOS_CentOS-6/x86_64/dnsmasq-2.66-3.1.x86_64.rpm
#yum localinstall dnsmasq-2.66-3.1.x86_64.rpm

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Start dhcp server
# service neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-09-16T08:29:53Z

Ophir Maor: /* DHCP Server */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

You can do it by using packstack:

1.Create an answer file
#packstack --gen-answer-file=GEN_ANSWER_FILE

2. Change the following in GEN_ANSWER_FILE
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_VLAN_RANGES=default:2:10
CONFIG_NEUTRON_L2_AGENT=openvswitch

3. Run packstack
#packstack --answer-file=GEN_ANSWER_FILE

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = openvswitch,mlnx
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. In /etc/nova/nova.conf

Check that the compute driver is libvirt:
[DEFAULT]
compute_driver=libvirt.LibvirtDriver

Change the VIF driver:
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized mode make sure vif driver is as follows:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc_support_old_agents''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#service openstack-nova-compute restart

2. Start eswitch Daemon
#service eswitchd start

3. Start the Neutron agent
#service neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/ml2/ml2_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#service neutron-openvswitch-agent start
#service neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#service neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#service opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==
We are using the linuxbridge mechanism driver so we can use the DHCP Server with the Linux Bridge interface driver.

Edit /etc/neutron/plugins/ml2/ml2_conf.ini as follows (The VLAN range is an example)

[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = linuxbridge,mlnx
[ml2_type_vlan]
network_vlan_ranges=default:2:10
[securitygroup]
enable_security_group = True
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

== Compute Nodes ==

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#service openibd restart
eswitchd should be started and then Neutron agent should be started
#service eswitchd restart
#service neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#service openstack-nova-compute restart

== Network Node ==

Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:

E_IPOIB_LOAD=yes
And restart openibd:

#service openibd restart
Please refer to eIPoIB configuration in Mellanox OFED User Manual
Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example: Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")

#ethtool -i <interface>
driver: eth_ipoib
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Restart neutron-linuxbridge-agent and neutron-dhcp-agent

#service neutron-linuxbridge-agent restart

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-dnsmasq-2014.1.1-1.noarch.rpm
# yum localinstall mlnx-dnsmasq-2014.1.1-1.noarch.rpm

In addition, dnsmasq must be upgraded to version 2.66 or higher.
#wget ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/kalyaka/CentOS_CentOS-6/x86_64/dnsmasq-2.66-3.1.x86_64.rpm
#yum localinstall dnsmasq-2.66-3.1.x86_64.rpm

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Start dhcp server
# service neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-09-16T08:27:48Z

Ophir Maor: /* DHCP Server */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

You can do it by using packstack:

1.Create an answer file
#packstack --gen-answer-file=GEN_ANSWER_FILE

2. Change the following in GEN_ANSWER_FILE
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_VLAN_RANGES=default:2:10
CONFIG_NEUTRON_L2_AGENT=openvswitch

3. Run packstack
#packstack --answer-file=GEN_ANSWER_FILE

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = openvswitch,mlnx
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. In /etc/nova/nova.conf

Check that the compute driver is libvirt:
[DEFAULT]
compute_driver=libvirt.LibvirtDriver

Change the VIF driver:
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized mode make sure vif driver is as follows:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc_support_old_agents''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#service openstack-nova-compute restart

2. Start eswitch Daemon
#service eswitchd start

3. Start the Neutron agent
#service neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/ml2/ml2_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#service neutron-openvswitch-agent start
#service neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#service neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#service opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==
We are using the linuxbridge mechanism driver so we can use the DHCP Server with the Linux Bridge interface driver.

Edit /etc/neutron/plugins/ml2/ml2_conf.ini as follows (The VLAN range is an example)

[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = linuxbridge,mlnx
[ml2_type_vlan]
network_vlan_ranges=default:2:10
[securitygroup]
enable_security_group = True
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

== Compute Nodes ==

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#service openibd restart
eswitchd should be started and then Neutron agent should be started
#service eswitchd restart
#service neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#service openstack-nova-compute restart

== Network Node ==

Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:

E_IPOIB_LOAD=yes
And restart openibd:

#service openibd restart
Please refer to eIPoIB configuration in Mellanox OFED User Manual
Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example: Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")

#ethtool -i <interface>
driver: eth_ipoib
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Restart neutron-linuxbridge-agent and neutron-dhcp-agent

#service neutron-linuxbridge-agent restart

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-dnsmasq-2014.1.1-1.noarch.rpm
# yum localinstall mlnx-dnsmasq

In addition, dnsmasq must be upgraded to version 2.66 or higher.
ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/kalyaka/CentOS_CentOS-6/x86_64/dnsmasq-2.66-3.1.x86_64.rpm

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Start dhcp server
# service neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-09-16T08:26:41Z

Ophir Maor: /* DHCP Server */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

You can do it by using packstack:

1.Create an answer file
#packstack --gen-answer-file=GEN_ANSWER_FILE

2. Change the following in GEN_ANSWER_FILE
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_VLAN_RANGES=default:2:10
CONFIG_NEUTRON_L2_AGENT=openvswitch

3. Run packstack
#packstack --answer-file=GEN_ANSWER_FILE

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = openvswitch,mlnx
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. In /etc/nova/nova.conf

Check that the compute driver is libvirt:
[DEFAULT]
compute_driver=libvirt.LibvirtDriver

Change the VIF driver:
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized mode make sure vif driver is as follows:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc_support_old_agents''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#service openstack-nova-compute restart

2. Start eswitch Daemon
#service eswitchd start

3. Start the Neutron agent
#service neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/ml2/ml2_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#service neutron-openvswitch-agent start
#service neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#service neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#service opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==
We are using the linuxbridge mechanism driver so we can use the DHCP Server with the Linux Bridge interface driver.

Edit /etc/neutron/plugins/ml2/ml2_conf.ini as follows (The VLAN range is an example)

[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = linuxbridge,mlnx
[ml2_type_vlan]
network_vlan_ranges=default:2:10
[securitygroup]
enable_security_group = True
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

== Compute Nodes ==

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#service openibd restart
eswitchd should be started and then Neutron agent should be started
#service eswitchd restart
#service neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#service openstack-nova-compute restart

== Network Node ==

Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:

E_IPOIB_LOAD=yes
And restart openibd:

#service openibd restart
Please refer to eIPoIB configuration in Mellanox OFED User Manual
Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example: Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")

#ethtool -i <interface>
driver: eth_ipoib
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Restart neutron-linuxbridge-agent and neutron-dhcp-agent

#service neutron-linuxbridge-agent restart

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-dnsmasq-2014.1.1-1.noarch.rpm
# yum install mlnx-dnsmasq

In addition, dnsmasq must be upgraded to version 2.66 or higher.
ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/kalyaka/CentOS_CentOS-6/x86_64/dnsmasq-2.66-3.1.x86_64.rpm

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Start dhcp server
#service neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-09-16T08:26:30Z

Ophir Maor: /* DHCP Server */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

You can do it by using packstack:

1.Create an answer file
#packstack --gen-answer-file=GEN_ANSWER_FILE

2. Change the following in GEN_ANSWER_FILE
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_VLAN_RANGES=default:2:10
CONFIG_NEUTRON_L2_AGENT=openvswitch

3. Run packstack
#packstack --answer-file=GEN_ANSWER_FILE

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = openvswitch,mlnx
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. In /etc/nova/nova.conf

Check that the compute driver is libvirt:
[DEFAULT]
compute_driver=libvirt.LibvirtDriver

Change the VIF driver:
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized mode make sure vif driver is as follows:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc_support_old_agents''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#service openstack-nova-compute restart

2. Start eswitch Daemon
#service eswitchd start

3. Start the Neutron agent
#service neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/ml2/ml2_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#service neutron-openvswitch-agent start
#service neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#service neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#service opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==
We are using the linuxbridge mechanism driver so we can use the DHCP Server with the Linux Bridge interface driver.

Edit /etc/neutron/plugins/ml2/ml2_conf.ini as follows (The VLAN range is an example)

[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = linuxbridge,mlnx
[ml2_type_vlan]
network_vlan_ranges=default:2:10
[securitygroup]
enable_security_group = True
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

== Compute Nodes ==

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#service openibd restart
eswitchd should be started and then Neutron agent should be started
#service eswitchd restart
#service neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#service openstack-nova-compute restart

== Network Node ==

Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:

E_IPOIB_LOAD=yes
And restart openibd:

#service openibd restart
Please refer to eIPoIB configuration in Mellanox OFED User Manual
Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example: Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")

#ethtool -i <interface>
driver: eth_ipoib
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Restart neutron-linuxbridge-agent and neutron-dhcp-agent

#service neutron-linuxbridge-agent restart

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
#wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-dnsmasq-2014.1.1-1.noarch.rpm
# yum install mlnx-dnsmasq

In addition, dnsmasq must be upgraded to version 2.66 or higher.
ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/kalyaka/CentOS_CentOS-6/x86_64/dnsmasq-2.66-3.1.x86_64.rpm

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Start dhcp server
#service neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-09-16T08:24:38Z

Ophir Maor: /* DHCP Server */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

You can do it by using packstack:

1.Create an answer file
#packstack --gen-answer-file=GEN_ANSWER_FILE

2. Change the following in GEN_ANSWER_FILE
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_VLAN_RANGES=default:2:10
CONFIG_NEUTRON_L2_AGENT=openvswitch

3. Run packstack
#packstack --answer-file=GEN_ANSWER_FILE

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = openvswitch,mlnx
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. In /etc/nova/nova.conf

Check that the compute driver is libvirt:
[DEFAULT]
compute_driver=libvirt.LibvirtDriver

Change the VIF driver:
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized mode make sure vif driver is as follows:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc_support_old_agents''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#service openstack-nova-compute restart

2. Start eswitch Daemon
#service eswitchd start

3. Start the Neutron agent
#service neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/ml2/ml2_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#service neutron-openvswitch-agent start
#service neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#service neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#service opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==
We are using the linuxbridge mechanism driver so we can use the DHCP Server with the Linux Bridge interface driver.

Edit /etc/neutron/plugins/ml2/ml2_conf.ini as follows (The VLAN range is an example)

[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = linuxbridge,mlnx
[ml2_type_vlan]
network_vlan_ranges=default:2:10
[securitygroup]
enable_security_group = True
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

== Compute Nodes ==

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#service openibd restart
eswitchd should be started and then Neutron agent should be started
#service eswitchd restart
#service neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#service openstack-nova-compute restart

== Network Node ==

Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:

E_IPOIB_LOAD=yes
And restart openibd:

#service openibd restart
Please refer to eIPoIB configuration in Mellanox OFED User Manual
Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example: Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")

#ethtool -i <interface>
driver: eth_ipoib
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Restart neutron-linuxbridge-agent and neutron-dhcp-agent

#service neutron-linuxbridge-agent restart

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

In addition, dnsmasq must be upgraded to version 2.66 or higher.
ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/kalyaka/CentOS_CentOS-6/x86_64/dnsmasq-2.66-3.1.x86_64.rpm

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Start dhcp server
#service neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-09-16T08:22:26Z

Ophir Maor: /* Network Node */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

You can do it by using packstack:

1.Create an answer file
#packstack --gen-answer-file=GEN_ANSWER_FILE

2. Change the following in GEN_ANSWER_FILE
CONFIG_NEUTRON_L2_PLUGIN=ml2
CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vlan
CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan
CONFIG_NEUTRON_ML2_MECHANISM_DRIVERS=openvswitch
CONFIG_NEUTRON_ML2_VLAN_RANGES=default:2:10
CONFIG_NEUTRON_L2_AGENT=openvswitch

3. Run packstack
#packstack --answer-file=GEN_ANSWER_FILE

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = openvswitch,mlnx
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. In /etc/nova/nova.conf

Check that the compute driver is libvirt:
[DEFAULT]
compute_driver=libvirt.LibvirtDriver

Change the VIF driver:
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized mode make sure vif driver is as follows:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc_support_old_agents''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#service openstack-nova-compute restart

2. Start eswitch Daemon
#service eswitchd start

3. Start the Neutron agent
#service neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/ml2/ml2_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#service neutron-openvswitch-agent start
#service neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#service neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#service opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==
We are using the linuxbridge mechanism driver so we can use the DHCP Server with the Linux Bridge interface driver.

Edit /etc/neutron/plugins/ml2/ml2_conf.ini as follows (The VLAN range is an example)

[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = linuxbridge,mlnx
[ml2_type_vlan]
network_vlan_ranges=default:2:10
[securitygroup]
enable_security_group = True
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

== Compute Nodes ==

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#service openibd restart
eswitchd should be started and then Neutron agent should be started
#service eswitchd restart
#service neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#service openstack-nova-compute restart

== Network Node ==

Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:

E_IPOIB_LOAD=yes
And restart openibd:

#service openibd restart
Please refer to eIPoIB configuration in Mellanox OFED User Manual
Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example: Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")

#ethtool -i <interface>
driver: eth_ipoib
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Restart neutron-linuxbridge-agent and neutron-dhcp-agent

#service neutron-linuxbridge-agent restart

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

Start dhcp server
#service neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Ubuntu

2014-09-16T08:15:54Z

Ophir Maor: /* Installation */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =

This wiki page describes how to tune Mellanox OpenStack on Ubuntu 12.04 servers

= Prerequisites =

You need at least 3 nodes:
* Controller node running the neutron-server service.
* Compute node running the eswitchd, nova-compute & neutron-plugin-mlnx-agent services.
* Network node running the neutron-plugin-linuxbridge-agent, neutron-dhcp-agent and neutron-l3-agent services.

= Ethernet Network =

== Controller Node ==

In /etc/nova/nova.conf ensure you have the following line:
security_group_api=nova

If you change, restart the nova services
# for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done

== Neutron Server Node ==
=== Installation ===

1. Neutron server uses MySQL database. Make sure you have running MySQL database for neutron.

If Neutron server is already running, you should stop it.

#/etc/init.d/neutron-server stop

If you want to use a database that was created you have to drop it and create another one. For example:

mysql -u root -ppassword <<EOF
drop database if exists neutron;
create database neutron;
EOF

If you want to create a new one:

mysql -u root -ppassword <<EOF
drop database if exists neutron;
create database neutron;
GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON neutron.* TO 'neutron' IDENTIFIED BY 'password';
FLUSH PRIVILEGES;
EOF

2. Install the required Deb for the Mellanox Neutron plugin:

apt-get install -y neutron-plugin-mlnx-agent

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

4. Change /etc/default/neutron-server :

NEUTRON_PLUGIN_CONFIG="/etc/neutron/plugins/mlnx/mlnx_conf.ini"

=== Configuration ===

1. Make the Mellanox plugin the current Neutron plugin by editing neutron.conf and changing core_plugin.

core_plugin = neutron.plugins.mlnx.mlnx_plugin.MellanoxEswitchPlugin

2. Plugin configuration: Edit the configuration file: /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
tenant_network_type - must be set on of supported tenant network types.
Possible values: 'vlan' for Ethernet or 'ib' for Infiniband.'vlan' is the default.
network_vlan_ranges - must be configured to specify the names of the physical

networks managed by the mellanox plugin,
along with the ranges of VLAN IDs available on each physical network for allocation to virtual networks.
Possible range is vlan range is 1-4093.
The default is "default:1:100"
<fabric name >:<vlan range start>:<vlan end range>

[agent]
rpc_support_old_agents = True
Note: rpc_support_old_agents should be set to 'True' (non default).

=== Start Services ===

Start (or restart) the Neutron server
#/etc/init.d/neutron-server start

== Compute Nodes ==

=== Installation ===

1. If you didn't download Mellanox OpenStack debian files download it :

http://www.mellanox.com/downloads/solutions/openstack/

2. Install the eswitchd debian files:
dpkg -i eswitchd_0.8-1_amd64.deb
dpkg -i python-mlnxvif_0.6-1_all.deb
apt-get install -f # fix dependencies
apt-get install -y neutron-plugin-mlnx-agent

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver
security_group_api=nova
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:

libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.
[AGENT]
rpc_support_old_agents = True
[ESWITCH]
physical_interface_mapping - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only releavant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
vnic_type - type of VM network interface: 'mlnx_direct' or 'hostdev' according to libvirt terminology.
hostdev: this is the traditional method of assigning any generic PCI device to a guest (SR-IOV).
mlnx_direct: this is a method to provide macvtap device on top of the PCI device (default).
bridge: - When using Linux Bridge Plugin on top of eIPoIB device
daemon_endpoint - 'tcp://127.0.0.1:60001'

For a plugin configuration file example (Havana), please refer to Mellanox config *ini file (click [https://github.com/openstack/neutron/blob/stable/havana/etc/neutron/plugins/mlnx/mlnx_conf.ini here])

4. (Optional) Modify /etc/init/neutron-plugin-mlnx-agent.conf as described in LP bug #[https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1257732 1257732]

5. (Optional) Check that ebrctl is listed somewhere in /etc/nova/rootwrap.d/*, otherwise add this to /etc/nova/rootwrap.d/compute.filters file in the [Filters] section.
ebrctl: CommandFilter, ebrctl, root

=== Start Services ===

1. Restart Nova.

#/etc/init.d/nova-compute restart

2. Start eswitch Daemon

#/etc/init.d/eswitchd start

3. Start the Neutron agent

#/etc/init.d/neutron-plugin-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install the Neutron Linux bridge plugin, the Neutron DHCP agent and the Neutron L3 agent:

# apt-get install neutron-plugin-linuxbridge-agent neutron-dhcp-agent neutron-l3-agent

2. Copy /etc/neutron/neutron.conf from the Compute node.

3. Change the following configuration of the ini file (/etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .
[linux_bridge]
physical_interface_mappings = default:eth2

4. Configure the DHCP agent according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
[DEFAULT]
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_dhcp_agent OpenStack documentation]

5. Configure the L3 agent according to the following guidelines

Update the following file: /etc/neutron/l3_agent.ini with:
[DEFAULT]
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_l3_agent OpenStack documentation]

6. Configure the Metadata agent according to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#nova_config_metadata OpenStack documentation]

7. Start the Neutron Linux Bridge, DHCP and L3 agents
# restart neutron-plugin-linuxbridge-agent
# restart neutron-dhcp-agent
# restart neutron-l3-agent
# restart neutron-metadata-agent

== Troubleshooting ==

* If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.

Mellanox-OpenStack

2014-09-15T13:14:20Z

Ophir Maor: /* References */

= Mellanox Neutron Plugin =
Mellanox supports the OpenStack Neutron releases with open source networking components through Neutron. It delivers higher compute and storage performance and additional functionality, such as NIC based switching to provide better security and isolation for virtual cloud environments.

For additional information about Mellanox NeutronPlugin refer to [https://wiki.openstack.org/wiki/Mellanox-Neutron Mellanox Neutron] wiki page.

= Mellanox Cinder Plugin =
Mellanox provides an OpenStack Storage plugin that enables running storage volume traffic over native RDMA transport, providing 5X faster throughput over standard TCP/IP, while significantly increasing infrastructure efficiency by using less CPU resources.

For additional information about Mellanox Cinder refer to [https://wiki.openstack.org/wiki/Mellanox-Cinder Mellanox Cinder] wiki page.

= References =
1. [http://www.mellanox.com/openstack/ OpenStack solution page at Mellanox site]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED web page]

4. [http://community.mellanox.com/community/support/solutions Mellanox Community]

5. [https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking SR-IOV Passthrough for Networking]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Juno-SR-IOV

2014-09-15T13:14:01Z

Ophir Maor: Replaced content with "Refer to this page: https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking"

Refer to this page: https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking

SR-IOV-Passthrough-For-Networking

2014-09-15T13:12:32Z

Ophir Maor: Created page with "=SR-IOV Networking in OpenStack Juno= OpenStack Juno adds inbox support to request VM access to virtual network via SR-IOV NIC. With the introduction of SR-IOV based NIC, the..."

=SR-IOV Networking in OpenStack Juno=
OpenStack Juno adds inbox support to request VM access to virtual network via SR-IOV NIC. With the introduction of SR-IOV based NIC, the traditional virtual bridge is no longer required. Each SR-IOV port is associated with a virtual function (VF). SR-IOV ports may be provided by Hardware-based Virtual Eithernet Bridging (HW VEB); or they may be extended to an upstream physical switch (IEEE 802.1br).
There are two ways that SR-IOV port may be connected:
* directly connected to its VF
* connected with a macvtap device that resides on the host, which is then connected to the corresponding VF.

==Nova==
Nova support for SR-IOV enables to schedule an instance with SR-IOV ports based on their network connectivity. The neutron ports' associated physical networks have to be used in making the scheduling decision.
PCI Whitelist was enchanced to allow tags to be associated with PCI devices. PCI devices available for SR-IOV networking should be tagged with physical_network label.

For SR-IOV networking, a pre-defined tag "physical_network" is used to define the physical network that the devices are attached to. A whitelist entry is defined as:
["device_id": "<id>",] ["product_id": "<id>",]
["address": "[[[[<domain>]:]<bus>]:][<slot>][.[<function>]]" |
"devname": "Ethernet Interface Name",]
"physical_network":"name string of the physical network"

<id> can be a '*' or a valid device/product id as displayed by the linux utility lspci. The address uses the same syntax as it's in lspci. The devname can be a valid PCI device name. The only device names that are supportedare those that are displayed by the linux utility ifconfig -a and correspond to either a PF or a VF on a vNIC.

If the device defined by the address or devname corresponds to a SR-IOV PF, all the VFs under the PF will match the entry.

Multiple whitelist entries per host are supported.

==Neutron==
Neutron support for SR-IOV requires ML2 Plugin with SR-IOV supporting Mechanism Driver.
Currently there is ML2 Mechanism Driver for SR-IOV capable NIC based switching (HW VEB).
There are Network Adapters from different vendors that may differ by supporting various functionality.
If VF link state update is supported by vendor Network Adapter, the SR-IOV NIC L2 Agent should be deployed to leverage this functionality .

==VM creation flow with SR-IOV vNIC==
* Create one or more neutron ports
neutron port-create <net-id> --binding:vnic-type direct

* Boot VM with one or more neutron ports
nova boot --flavor m1.large --image <image>
--nic port-id=<port1> --nic port-id=<port2>

Note that in the nova boot API, users can specify either a port-id or a net-id. If it's the latter case, it's assumed that the user is requesting a normal virtual port (which is not a SR-IOV port).

=SR-IOV Configuration=

===Neutron Server===
Using ML2 Neutron plugin modify /etc/neutron/plugins/ml2/ml2_conf.ini:

[ml2]
tenant_network_types = vlan
type_drivers = vlan
mechanism_drivers = openvswitch,sriovnicswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100

Make sure /etc/neutron/plugins/ml2/ml2_conf_sriov.ini has the following section:

[ml2_sriov]
agent_required = True

Neutron server should be run with the two configuration files /etc/neutron/plugins/ml2/ml2_conf.in and /etc/neutron/plugins/ml2/ml2_conf_sriov.ini
neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini

==Compute==
===Nova===
On each compute you have to associate the Virtual Functions available to each Physical Network.
You do it by configuring pci_passthrough_whitelist in /etc/nova/noca.conf

For example:
pci_passthrough_whitelist = {"address":"*:0a:00.*","physical_network":"physnet1"}
This will associate any VF with address that includes ':0a:00.' in it's address to the Physical network physnet1

After configuring the white list you have to restart nova-compute service.

===Neutron===
If the hardware supports and and you want to enable changing the port admin_state you have to run the Neutron SR-IOV agent. 

'''Note:'''If you configured agent_required=True on the Neutron server you must run the Agent on each compute node.

In /etc/neutron/plugins/ml2/ml2_conf.ini make sure you have the following:
[securitygroup]
firewall_driver = neutron.agent.firewall.NoopFirewallDriver

Modify /etc/neutron/plugins/ml2/ml2_conf_sriov.ini as follows

[sriov_nic]
physical_device_mappings = physnet1:eth1
exclude_devices =

physnet1 is the physical network
eth1 if the Physical Function (PF)
exclude_devices is empty so All the VFs associated with eth1 are allowed to be configured by the agent.

After modifying the configuration file start the Neutron SR-IOV agent:
neutron-sriov-nic-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini

====Exclude VFs====
If you want to exclude some of the VFs so the agent won't configure them you need to list them in the sriov_nic section: 

'''Example:''' exclude_devices = eth1:0000:07:00.2; 0000:07:00.3, eth2:0000:05:00.1; 0000:05:00.2

=References=

[http://community.mellanox.com/docs/DOC-1484 Openstack ML2 SR-IOV driver support]

Juno-SR-IOV

2014-09-12T02:32:23Z

Ophir Maor: /* References */

=This Page is under construction=
Once it will be ready, the content might move to another location.
=SR-IOV in OpenStack Juno=
OpenStack Juno adds inbox support for SR-IOV ....
==Nova==
Nova support for SR-IOV is ....

==Neutron==
Neutron support for SR-IOV is ...
embedded NIC is ...

=SR-IOV Configuration=

===Neutron Server===
Using ML2 Neutron plugin modify /etc/neutron/plugins/ml2/ml2_conf.ini:

[ml2]
tenant_network_types = vlan
type_drivers = vlan
mechanism_drivers = openvswitch,sriovnicswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100

Make sure /etc/neutron/plugins/ml2/ml2_conf_sriov.ini has the following section:

[ml2_sriov]
agent_required = True

Neutron server should be run with the two configuration files /etc/neutron/plugins/ml2/ml2_conf.in and /etc/neutron/plugins/ml2/ml2_conf_sriov.ini
neutron-server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini

==Compute==
===Nova===
On each compute you have to associate the Virtual Functions available to each Physical Network.
You do it by configuring pci_passthrough_whitelist in /etc/nova/noca.conf

For example:
pci_passthrough_whitelist = {"address":"*:0a:00.*","physical_network":"physnet1"}
This will associate any VF with address that includes ':0a:00.' in it's address to the Physical network physnet1

After configuring the white list you have to restart nova-compute service.

===Neutron===
If the hardware supports and and you want to enable changing the port admin_state you have to run the Neutron SR-IOV agent. 

'''Note:'''If you configured agent_required=True on the Neutron server you must run the Agent on each compute node.

In /etc/neutron/plugins/ml2/ml2_conf.ini make sure you have the following:
[securitygroup]
firewall_driver = neutron.agent.firewall.NoopFirewallDriver

Modify /etc/neutron/plugins/ml2/ml2_conf_sriov.ini as follows

[sriov_nic]
physical_device_mappings = physnet1:eth1
exclude_devices =

physnet1 is the physical network
eth1 if the Physical Function (PF)
exclude_devices is empty so All the VFs associated with eth1 are allowed to be configured by the agent.

After modifying the configuration file start the Neutron SR-IOV agent:
neutron-sriov-nic-agent --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini

====Exclude VFs====
If you want to exclude some of the VFs so the agent won't configure them you need to list them in the sriov_nic section: 

'''Example:''' exclude_devices = eth1:0000:07:00.2; 0000:07:00.3, eth2:0000:05:00.1; 0000:05:00.2

=References=

[http://community.mellanox.com/docs/DOC-1484 Openstack ML2 SR-IOV driver support]

Juno-SR-IOV

2014-09-12T02:31:51Z

Ophir Maor: /* Exclude VFs */

Mellanox-OpenStack

2014-09-03T07:41:50Z

Ophir Maor:

= Mellanox Neutron Plugin =
Mellanox supports the OpenStack Neutron releases with open source networking components through Neutron. It delivers higher compute and storage performance and additional functionality, such as NIC based switching to provide better security and isolation for virtual cloud environments.

For additional information about Mellanox NeutronPlugin refer to [https://wiki.openstack.org/wiki/Mellanox-Neutron Mellanox Neutron] wiki page.

= Mellanox Cinder Plugin =
Mellanox provides an OpenStack Storage plugin that enables running storage volume traffic over native RDMA transport, providing 5X faster throughput over standard TCP/IP, while significantly increasing infrastructure efficiency by using less CPU resources.

For additional information about Mellanox Cinder refer to [https://wiki.openstack.org/wiki/Mellanox-Cinder Mellanox Cinder] wiki page.

= References =
1. [http://www.mellanox.com/openstack/ OpenStack solution page at Mellanox site]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED web page]

4. [http://community.mellanox.com/community/support/solutions Mellanox Community]

5. [https://wiki.openstack.org/wiki/Juno-SR-IOV SR-IOV for Juno]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Juno-SR-IOV

2014-09-03T07:41:07Z

Ophir Maor:

Juno-SR-IOV

2014-09-03T07:40:05Z

Ophir Maor: Created page with "=SR-IOV in OpenStack Juno= OpenStack Juno adds inbox support for SR-IOV .... ==Nova== Nova support for SR-IOV is .... ==Neutron== Neutron support for SR-IOV is ... embedded..."

=SR-IOV in OpenStack Juno=
OpenStack Juno adds inbox support for SR-IOV ....
==Nova==
Nova support for SR-IOV is ....

==Neutron==
Neutron support for SR-IOV is ...
embedded NIC is ...

=SR-IOV Configuration=
General configuration

Mellanox-Neutron-Icehouse-Redhat

2014-06-08T11:23:34Z

Ophir Maor: /* Usage Examples */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = mlnx,openvswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
vif_driver=mlnxvif.vif.MlxEthVIFDriver
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-openvswitch-agent start
#/etc/init.d/neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#/etc/init.d/neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
#/etc/init.d/eswitchd restart
#/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

In case you use MLNX-OFED older than 2.1, Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
#/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
#/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-06-08T11:23:10Z

Ophir Maor: /* References */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = mlnx,openvswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
vif_driver=mlnxvif.vif.MlxEthVIFDriver
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-openvswitch-agent start
#/etc/init.d/neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#/etc/init.d/neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
#/etc/init.d/eswitchd restart
#/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

In case you use MLNX-OFED older than 2.1, Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
#/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
#/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-06-08T11:20:41Z

Ophir Maor: /* Network Node */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = mlnx,openvswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
vif_driver=mlnxvif.vif.MlxEthVIFDriver
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-openvswitch-agent start
#/etc/init.d/neutron-dhcp-agent start

5. Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6. Start the l3 Agent:
#/etc/init.d/neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
#/etc/init.d/eswitchd restart
#/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

In case you use MLNX-OFED older than 2.1, Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
#/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
#/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat

2014-06-08T11:15:10Z

Ophir Maor: /* Installation */

= Page is under construction =

Note: The page is not ready yet.

Once it will be ready, the note will be removed.

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==

'''Installation with Red Hat Enterprise Linux OpenStack Platform'''

Make sure you follow Red Hat prerequisites as required by Red Hat Enterprise Linux OpenStack Platform. Please refer to https://access.redhat.com/products/Cloud/OpenStack/ for additional information regarding this product.

It is assumed that OpenStack is installed with ML2 plugin.

Other Red Hat references:

1. [http://www.redhat.com/resourcelibrary/reference-architectures/deploying-and-using-red-hat-openstack-rhos-2-dot-1 RHOS reference document]

2. [http://openstack.redhat.com/Quickstart RDO QuickStart]

'''Neutron server'''

No special prerequisites needed.

'''Compute Nodes'''

1. python-pip (use "yum install python-pip")

2. Compute nodes should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

3. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/docs/DOC-1317 Mellanox Community]

5. The software package iproute2 - ([https://www.kernel.org/pub/linux/utils/net/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Required only to be installed on compute nodes

6. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.

'''Network Node'''

1. Network node should be equipped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])

2. Mellanox OFED 2.2 or greater is installed. Refer to Mellanox website for the latest OFED ([http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers])

= Ethernet Network =
== Neutron Server Node ==

=== Installation ===
1. Make sure ML2 plugin is the current Neutron plugin by checking core_plugin option in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Make sure /etc/neutron/plugin.ini is pointing at (symbolic link) /etc/neutron/plugins/ml2/ml2_conf.ini

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini and include the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = mlnx,openvswitch
[ml2_type_vlan]
network_vlan_ranges = default:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True

Click [http://docs.openstack.org/trunk/config-reference/content/networking-options-plugins-ml2.html here] for ML2 configuration options.

=== Start Services ===
Start (or restart) the Neutron server
#service neutron-server restart

== Compute Nodes ==

=== Installation ===
1. Download Mellanox OpenStack repo file download it :
#wget -O /etc/yum.repos.d/mlnx-icehouse.repo http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo

2. Install the eswitchd RPM:
#yum install eswitchd

3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
#yum install mlnxvif

4. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox

=== Configuration ===

1. Configure /etc/eswitchd/eswitchd.conf if needed

Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])

2. Modify /etc/nova/nova.conf

compute_driver=nova.virt.libvirt.driver.LibvirtDriver
vif_driver=mlnxvif.vif.MlxEthVIFDriver
connection_type=libvirt

In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver

3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.

[AGENT]
'''polling_interval''' - Polling interval (in seconds) for existing vNICs. The default is 2 seconds.
'''rpc''' - must be set to 'True'

[ESWITCH]
'''physical_interface_mapping''' - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only relevant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth".
'''daemon_endpoint''' - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:60001')
'''request_timeout''' - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)

For a plugin configuration file example (Icehouse), please refer to [https://github.com/openstack/neutron/blob/stable/icehouse/etc/neutron/plugins/mlnx/mlnx_conf.ini Mellanox config ini file].

=== Start Services ===

1. Restart Nova.
#/etc/init.d/openstack-nova-compute restart

2. Start eswitch Daemon
#/etc/init.d/eswitchd start

3. Start the Neutron agent
#/etc/init.d/neutron-mlnx-agent start

Note: eswitch Daemon should be running before the Neutron Agent is started.

== Network Node ==

1. Install Neutron Open vSwitch Agent, Neutron dhcp Agent and l3 Agent:

#yum install openstack-neutron-openvswitch

2. Change the following configuration of the ini file (/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .

bridge_mappings = default:br-eth3,public:br-ex
(Here public is the physical external network)

3. Configure the DHCP server according to the following guidelines

Update the following file: /etc/neutron/dhcp_agent.ini with:
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver

For additional information refer to:
http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_dhcp_agent.html

4. Start the DHCP server
#/etc/init.d/neutron-openvswitch-agent start
#/etc/init.d/neutron-dhcp-agent start

5.Configure the l3 agent configuration file /etc/neutron/l3_agent.ini

gateway_external_network_id = d4fdfebb-e027-4acd-bed4-1d96e896f336
router_id = 41bf1aa0-3daf-4f51-9d23-0a4b15020c36
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
external_network_bridge = br-ex

The above is an example for configuring one router for tenants. Your values for gateway_external_network_id,router_id and external_network_bridge may differ.

6.Start the l3 Agent:
#/etc/init.d/neutron-l3-agent restart

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Configuration ==

=== OpenSM configuration - Without UFM===

All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)
(Automatic cofiguration is planned in future phase)

Add/Change the following in the partitions.conf file
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

For example:
If we have 10 VLANs defined in configuration in /etc/neutron/plugins/mlnx/mlnx_conf.ini

[MLNX]
network_vlan_ranges = default:1:10

We'll have the following configuration of the the partitions.conf file:

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

Change the following in /etc/opensm/opensm.conf:
allow_both_pkeys TRUE

Restart the openSM
#/etc/init.d/opensmd restart

=== OpenSM configuration - With UFM===

1. Make sure UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

#Allow both full and limited membership on the same partition
allow_both_pkeys TRUE

b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

#SM assigned Alias GUIDs algorithm
sm_assign_guid_func uniq_count

3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

b) add the following line to file which enable both full and limited management pkey:

management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

4. Restart UFM

Stand-alone

#/etc/init.d/ufmd restart

High-availability

#/etc/init.d/ufmha restart

== Neutron Server Node==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Change the following in the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = hostdev
network_vlan_ranges = default:1:10
(sql_connection and reconnect_interval can be configured as described above)

The mapping between VLAN and PKEY is as follows: VLAN X = PKEY 0x8000 + X. For example: vlan 2 is pkey 0x8002

=== Para-Virtualized ===

On the controller, configure the file /etc/neutron/plugins/mlnx/mlnx_conf.ini
tenant_network_type = ib
vnic_type = bridge

== Compute Nodes ==

=== SR-IOV ===
SR-IOV is a passthrough mode.
Perform the following changes:

Create the file /etc/modprobe.d/mlx4_ib.conf and put:
options mlx4_ib sm_guid_assign=0

In The file /etc/neutron/plugins/mlnx/mlnx_conf.ini
physical_interface_mapping = default:autoib

Tenant_network_type , vnic_type and network_vlan_ranges parameters should be configured as the controller.

autoib can be replaced by the name of the PF.

Change the file /etc/eswitchd/eswitchd.conf
fabrics = default:autoib (or default:ib0)

The driver should be restarted
#/etc/init.d/openibd restart
eswitchd should be started and then Neutron agent should be started
#/etc/init.d/eswitchd restart
#/etc/init.d/neutron-mlnx-agent restart

Verify Mellanox VIF driver is configured in /etc/nova/nova.conf
libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

=== Para-Virtualized ===
Here we use the Linux Bridge plugin.

eIPoIB module should be up and configured.In /etc/infiniband/openib.conf:
E_IPOIB_LOAD=yes

And restart openibd:
#/etc/init.d/openibd restart

Please refer to eIPoIB configuration in Mellanox OFED User Manual 

Once we have the eIPoIB interface, we use it in the linux bridge agent configuration:

For Example:
Assuming eth1 is the eIPoIB interface:

To check that the interface type is eIPoIB run the command (verify that the driver is "eth_ipoib")
#ethtool -i <interface>
'''driver: eth_ipoib'''
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

Change the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini

[linux_bridge]
physical_interface_mappings = default:eth1

Configure the Linux Bridge VIF Driver in /etc/nova/nova.conf
#libvirt_vif_driver=nova.virt.libvirt.vif.NeutronLinuxBridgeVIFDriver

Restart nova
#/etc/init.d/openstack-nova-compute restart

After confiuration Linux Bridge agent should be restarted.
#/etc/init.d/neutron-linuxbridge-agent restart

== Network Node ==

Configure the Linux bridge plugin as described in InfiniBand Para-Virtualized section (above).

=== DHCP Server ===

For DHCP support – The Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
# yum install mlnx-dnsmasq

Change the following in /etc/neutron/dhcp_agent.ini
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

In case you use MLNX-OFED older than 2.1, Copy ipoibd and override /sbin/ipoibd
#wget http://www.mellanox.com/downloads/solutions/openstack/havana/ipoibd
#cp ipoibd /sbin/ipoibd

Download and apply a patch for Linux Bridge Agent:

#wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/linuxbridge_neutron_agent.py.patch
#cd /usr/lib/python2.6/site-packages/neutron/plugins/linuxbridge/agent
#patch -p1 < linux_bridge_agent.py.patch

Restart openibd
#/etc/init.d/openibd restart

Start linux bridge agent
#/etc/init.d/neutron-linuxbridge-agent restart

Start dhcp server
#/etc/init.d/neutron-dhcp-agent restart

= Usage Examples =
* In order to create SR-IOV interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
* In order to create Para-Virtualized interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.