OpenStack - User contributions [en]

Mellanox-OpenStack

2015-01-12T08:43:02Z

Yaell:

= Mellanox Neutron Plugin =
Mellanox supports the OpenStack Neutron releases with open source networking components through Neutron. It delivers higher compute and storage performance and additional functionality, such as NIC based switching to provide better security and isolation for virtual cloud environments.

For additional information about Mellanox NeutronPlugin refer to [https://wiki.openstack.org/wiki/Mellanox-Neutron Mellanox Neutron] wiki page.

= Mellanox Cinder Plugin =
Mellanox provides an OpenStack Storage plugin that enables running storage volume traffic over native RDMA transport, providing 5X faster throughput over standard TCP/IP, while significantly increasing infrastructure efficiency by using less CPU resources.

For additional information about Mellanox Cinder refer to [https://wiki.openstack.org/wiki/Mellanox-Cinder Mellanox Cinder] wiki page.

= References =
1. [http://www.mellanox.com/openstack/ OpenStack solution page at Mellanox site]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED web page]

4. [http://community.mellanox.com/community/solutions Mellanox Community]

5. [https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking SR-IOV Passthrough for Networking]

For more details, please refer your question to [mailto:support@mellanox.com support@mellanox.com]

Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand

2015-01-01T09:52:31Z

Yaell:

= Overview =

This wiki page describes how to tune Mellanox OpenStack on Ubuntu 12.04 servers

= Prerequisites =

* A running OpenStack environment installed with the ML2 plugin on top of Linux Bridge.
* All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119)
* Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers
* SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community [http://community.mellanox.com/docs/DOC-1317 click here].
* The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes
* VLANs configured on the ports in the switch.

= InfiniBand Network =

== OpenSM Node ==

To configure OpenSM node:

1. Edit the file /etc/opensm/partitions.conf by adding the following line:
management=0xffff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
Create the file if it does not exist.

For each VLAN value in the VLAN range used for VLAN network segmentation, add the following line in the file /etc/opensm/partitions.conf (for vlan-pkey association):
vlan<decimal-value>=0x<hexadecimal-value>, ipoib, sl=0, defmember=full: ALL;
In this manual, the VLAN range is 2-10 (mlnx_agent.conf /ml2_conf.ini). Therefore, add the following lines:
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;
2. Modify the following line in the file /etc/opensm/opensm.conf from FALSE to TRUE:
allow_both_pkeys TRUE
3. Restart OpenSM:
service opensmd restart
4. Verify that all ports on all nodes have their ib interface (i.e. ib0) active, run ibdev2netdev on all hosts, and make sure that the port is up:
mlx4_0 port 1 ==> ib0 (Up)
If the port is down, try bringing it up by running “ifconfig ib0 up”, and make sure your switch ports are of type “InfiniBand”, and that all nodes can reach the OpenSM node by running “sminfo”.

== Controller Node ==

To configure the Controller node:

1. Ensure that the file /etc/nova/nova.conf includes the following line:
security_group_api=nova

2. If the security group value is changed, restart the Nova services:
# for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done

== Neutron Server Node ==

NOTE: The Neutron Server can sometimes be part of the Controller node.

To configure the Neutron Server node:

1. Edit /etc/neutron/neutron.conf and make sure Neutron is using ML2 plugin:
[DEFAULT]
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin
2. Edit the ML2 configuration file /etc/neutron/plugins/ml2/ml2_conf.ini as follows:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = linuxbridge,mlnx

[ml2_type_vlan]
network_vlan_ranges=<physical_network>:<min_vlan>:<max_vlan> //(i.e. default:2:10)

[securitygroup]
enable_security_group = True

[eswitch]
vnic_type = hostdev
apply_profile_patch = True
3. Install the required Deb for the Mellanox Neutron plugin (from Ubuntu’s repository):
apt-get install neutron-plugin-mlnx-agent
4. Edit /etc/neutron/plugins/mlnx/mlnx_conf.ini to match your environment:
[mlnx]
network_vlan_ranges = <physical_network>:<min_vlan:<max_vlan> //(i.e. default:2:10)
tenant_network_type = vlan

[eswitch]
physical_interface_mappings = <physical_network>:<ib interface> //(i.e. default:ib0)
vnic_type = hostdev

[agent]
support_old_agents = True

5. Restart Neutron Server:
/etc/init.d/neutron-server start

== Compute Nodes ==

To configure the Compute node:

1. Create the file /etc/modprobe.d/mlx4_ib.conf with this line:
options mlx4_ib sm_guid_assign=0
2. Download mlnxvif, eSwitch Daemon (eSwitchd), and mlnx-dnsmasq from this link:
http://www.mellanox.com/downloads/solutions/openstack/icehouse/debian/
3. Install the Mellanox plugin services mlnxvif and eSwitchd:
dpkg –i python-mlnxvif_2014.1.1-1_all.deb
dpkg –i python-eswitchd_0.10-1_all.deb
4. Edit /etc/eswitchd/eswitchd.conf as follows:
[DAEMON]
fabrics=<physical_network>:<ib-interface> //(i.e. default:ib0)
5. Edit /etc/nova/nova.conf as follows:
[DEFAULT]
compute_driver = libvirt.LibvirtDriver
libvirt_vif_driver = mlnxvif.vif.MlxEthVIFDriver
security_group_api = nova
6. Check if “ebrctl” appears in one of Nova’s rootwrap files:
grep 'ebrctl' /etc/nova/rootwrap.d/*
If “ebrctl” does not appear in one of Nova’s rootwrap files, edit the file /etc/nova/rootwrap.d/compute.filter as follows:
[Filters]
ebrctl: CommandFilter, ebrctl, root
7. Install the Neutron Mellanox agent (from Ubuntu repository):
apt-get install neutron-plugin-mlnx-agent
8. Copy /etc/neutron/plugins/mlnx/mlnx_conf.ini from the Neutron Server node to the same path on each Compute node.

9. Restart the following services:
/etc/init.d/nova-compute restart
/etc/init.d/eswitchd start
/etc/init.d/neutron-plugin-mlnx-agent start

== Network Node ==

To configure the Network node:

1. Configure eIPoIB (for Linux bridge plugin) by editing the file /etc/infiniband/openib.conf as follows:
E_IPOIB_LOAD=yes
2. Restart OFED:
service openibd restart
3. Verify that the new Ethernet interfaces now appear with ibdev2netdev:
# ibdev2netdev
mlx4_0 port 1 ==> ib0 (Up)
mlx4_0 port 2 ==> ib1 (Down)
mlx4_0 port 1 ==> eth2 (Up)
mlx4_0 port 2 ==> eth1 (Down)
In eIPoIB, every ib interface has a matching eth interface. The ib interface used here is ib0, and the matching one is eth2 (same card and port names: mlx4_0 port 1). By using Ethtool, you can verify that ib0 is working on ib_ipoib driver, and eth2 is working on eth_ipoib driver, as follows:
# ethtool -i ib0
driver: ib_ipoib
version: 2.3-1.5.4_Sep 30 2014
firmware-version: 2.32.5100
bus-info: 0000:07:00.0# ethtool -i eth2
driver: eth_ipoib
version: 2.3-1.5.4 (Sep 30 2014)
firmware-version: 1
bus-info: ib0
4. Edit /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows:
[linux_bridge]
physical_interface_mappings = <physical_network>:<eth_ipoib-interface>
//(i.e. default:eth2)
5. Restart the Linux bridge agent:
service neutron-linuxbridge-agent restart
6. To allow DHCP support, download mlnx-dnsmadq from this link: http://www.mellanox.com/downloads/solutions/openstack/icehouse/debian/

7. Install the package:
dpkg –i python-mlnx-dnsmasq_2014.1.1-1_all.deb
8. Configure the file /etc/neutron/dhcp_agent.ini:
[DEFAULT]
dhcp_driver = mlnx_dhcp.MlnxDnsmasq
9. Make sure that the Dnsmasq which is installed as a part of the OpenStack (not mlnx-dnsmasq) environment is of version 2.65.

10. Copy /etc/neutron/neutron.conf from the Compute node to the same path on the Network node.

11. Edit /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows:
[linux_bridge]
physical_interface_mappings = <physical_network>:<eth_ipoib interface>
(i.e. default:eth2)
12. Edit /etc/neutron/dhcp_agent.ini and /etc/neutron/l3_agent.ini as follows:
[DEFAULT]
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver
13. Restart the following services:
service neutron-plugin-linuxbridge-agent restart
service neutron-dhcp-agent restart
service neutron-l3-agent restart

== Troubleshooting ==

* If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.

Mellanox-Neutron-Icehouse-Redhat-InfiniBand

2015-01-01T09:51:16Z

Yaell:

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==
* A running OpenStack environment installed with the ML2 plugin on top of Linux Bridge.
* All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119)
* Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers
* SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community [http://community.mellanox.com/docs/DOC-1317 click here].
* The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes
* VLANs configured on the ports in the switch.

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Node ==

=== OpenSM Configuration – Without UFM ===

To configure OpenSM:

1. Make sure that all the PKeys are predefined in the partitions.conf file (/etc/opensm/partitions.conf).

2. Add/Change the following in the partitions.conf file:
management=0xffff,ipoib, sl=0, defmember=full: ALL, ALL_SWITCHES=full,SELF=full;
3. For every network you want to configure in Neutron, you have to configure the PKey associated with the VLAN of this network (defined in Neutron):
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
Below is an example of the configuration of the partitions.conf file in case you have 10 VLANs defined in the configuration of the file: /etc/neutron/plugins/mlnx/mlnx_conf.ini. ( network_vlan_ranges = default:1:10)

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full: ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full: ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full: ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full: ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full: ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full: ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full: ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full: ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full: ALL_CAS;

4. Modify the following line in the file /etc/opensm/opensm.conf from FALSE to TRUE:
allow_both_pkeys TRUE
5. Restart the OpenSM:
#service opensmd restart

=== OpenSM configuration - With UFM===

To configure OpenSM:

1. Make sure that UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf by changing the following values:

a. Set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):
Allow both full and limited membership on the same partition
allow_both_pkeys TRUE
b. Set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port):
SM assigned Alias GUIDs algorithm sm_assign_guid_func uniq_count
3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a. Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty right after UFM installation)
Add the following line to the file to enable both full and limited management PKeys:
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;
b. Add the additional PKey definitions relevant to the specific setup.
For Example:
vlan1=0x1, ipoib, sl=0, defmember=full: ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full: ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full: ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full: ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full: ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full: ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full: ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full: ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full: ALL_CAS;
4. Restart UFM

• Stand-alone
#/etc/init.d/ufmd restart
• High-availability
#/etc/init.d/ufmha restart

== Neutron Server Node==

To configure the Neutron Server node:

1. Make sure ML2 is the current Neutron plugin by checking the core_plugin parameter in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin
2. Make sure /etc/neutron/plugin.ini is pointing to /etc/neutron/plugins/ml2/ml2_conf.ini (symbolic link)

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini by adding the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = linuxbridge,mlnx
[ml2_type_vlan]
network_vlan_ranges = <network_name>:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True
4. Start (or restart) the Neutron server:
#service neutron-server restart

== Compute Nodes ==

To configure the Compute Node:

1. Download Mellanox OpenStack repo file:
#wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo -O /etc/yum.repos.d/mlnx-icehouse.repo
2. Install the eSwitch Daemon (eSwitchd) RPM:
#yum install eswitchd
3. Install Mellanox VIF driver:
#yum install mlnxvif
3. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox
4. Create the file /etc/modprobe.d/mlx4_ib.conf and add the following:
options mlx4_ib sm_guid_assign=0
5. Restart the driver:
#service openibd restart
6.In the file /etc/neutron/plugins/mlnx/mlnx_conf.ini, the parameters tenant_network_type , vnic_type and network_vlan_ranges should be configured as the controllers:
physical_interface_mapping = default:ib<your_interface>
7. Modify the file /etc/eswitchd/eswitchd.conf as follows:
fabrics = default:ib<your_interface> (for example default:ib0)
8. Verify Mellanox VIF driver is configured in /etc/nova/nova.conf:
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver
9. Restart Nova:
#service openstack-nova-compute restart
10. Start eSwitchd:
#service eswitchd start
11. Start the Neutron agent:
#service neutron-mlnx-agent start

== Network Node ==

To configure the Network node:

1. Make sure that eIPoIB module is up and configured in /etc/infiniband/openib.conf:
For more information, please refer to eIPoIB configuration in Mellanox OFED User Manual.
E_IPOIB_LOAD=yes
2. Restart openibd:
#service openibd restart
3. Modify the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows:
[linux_bridge]
physical_interface_mappings = default:<eIPoIB interface>

NOTE: In order to obtain the eIPoIB interface name, run the ethtool tool command (see below) and check the driver name:

#ethtool -i <eIPoIB_interface>
driver: eth_ipoib
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

4. Restart neutron-linuxbridge-agent and neutron-dhcp-agent:
#service neutron-linuxbridge-agent restart

NOTE: For DHCP support, the Network node should use the Mellanox Dnsmasq driver as the DHCP driver.

=== DHCP Server (Usally part of the Network node) ===

1. Install the Mellanox Dnsmasq:
#wget
http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-dnsmasq-2014.1.1-1.noarch.rpm
#yum localinstall mlnx-dnsmasq-2014.1.1-1.noarch.rpm

2. In addition, Dnsmasq must be upgraded to version 2.66 or higher:
#wget
ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/kalyaka/CentOS_CentOS-6/x86_64/dnsmasq-2.66-3.1.x86_64.rpm

#yum localinstall dnsmasq-2.66-3.1.x86_64.rpm
3. Modify /etc/neutron/dhcp_agent.ini as follows:
dhcp_driver = mlnx_dhcp.MlnxDnsmasq
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver
4. Start DHCP server:
#service neutron-dhcp-agent restart

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat-InfiniBand

2014-12-08T15:47:15Z

Yaell:

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==
* A running OpenStack environment installed with the ML2 plugin on top of Linux Bridge.
* All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119)
* Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers
* SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community [http://community.mellanox.com/docs/DOC-1317 click here].
* The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes
* VLANs configured on the ports in the switch.

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Node ==

=== OpenSM Configuration – Without UFM ===

To configure OpenSM:

1. Make sure that all the PKeys are predefined in the partitions.conf file (/etc/opensm/partitions.conf).

2. Add/Change the following in the partitions.conf file:
management=0xffff,ipoib, sl=0, defmember=both: ALL, ALL_SWITCHES=full,SELF=full;
3. For every network you want to configure in Neutron, you have to configure the PKey associated with the VLAN of this network (defined in Neutron):
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
Below is an example of the configuration of the partitions.conf file in case you have 10 VLANs defined in the configuration of the file: /etc/neutron/plugins/mlnx/mlnx_conf.ini. ( network_vlan_ranges = default:1:10)

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full: ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full: ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full: ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full: ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full: ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full: ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full: ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full: ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full: ALL_CAS;

4. Modify the following line in the file /etc/opensm/opensm.conf from FALSE to TRUE:
allow_both_pkeys TRUE
5. Restart the OpenSM:
#service opensmd restart

=== OpenSM configuration - With UFM===

To configure OpenSM:

1. Make sure that UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf by changing the following values:

a. Set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):
Allow both full and limited membership on the same partition
allow_both_pkeys TRUE
b. Set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port):
SM assigned Alias GUIDs algorithm sm_assign_guid_func uniq_count
3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a. Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty right after UFM installation)
Add the following line to the file to enable both full and limited management PKeys:
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;
b. Add the additional PKey definitions relevant to the specific setup.
For Example:
vlan1=0x1, ipoib, sl=0, defmember=full: ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full: ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full: ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full: ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full: ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full: ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full: ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full: ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full: ALL_CAS;
4. Restart UFM

• Stand-alone
#/etc/init.d/ufmd restart
• High-availability
#/etc/init.d/ufmha restart

== Neutron Server Node==

To configure the Neutron Server node:

1. Make sure ML2 is the current Neutron plugin by checking the core_plugin parameter in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin
2. Make sure /etc/neutron/plugin.ini is pointing to /etc/neutron/plugins/ml2/ml2_conf.ini (symbolic link)

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini by adding the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = linuxbridge,mlnx
[ml2_type_vlan]
network_vlan_ranges = <network_name>:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True
4. Start (or restart) the Neutron server:
#service neutron-server restart

== Compute Nodes ==

To configure the Compute Node:

1. Download Mellanox OpenStack repo file:
#wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo -O /etc/yum.repos.d/mlnx-icehouse.repo
2. Install the eSwitch Daemon (eSwitchd) RPM:
#yum install eswitchd
3. Install Mellanox VIF driver:
#yum install mlnxvif
3. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox
4. Create the file /etc/modprobe.d/mlx4_ib.conf and add the following:
options mlx4_ib sm_guid_assign=0
5. Restart the driver:
#service openibd restart
6.In the file /etc/neutron/plugins/mlnx/mlnx_conf.ini, the parameters tenant_network_type , vnic_type and network_vlan_ranges should be configured as the controllers:
physical_interface_mapping = default:ib<your_interface>
7. Modify the file /etc/eswitchd/eswitchd.conf as follows:
fabrics = default:ib<your_interface> (for example default:ib0)
8. Verify Mellanox VIF driver is configured in /etc/nova/nova.conf:
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver
9. Restart Nova:
#service openstack-nova-compute restart
10. Start eSwitchd:
#service eswitchd start
11. Start the Neutron agent:
#service neutron-mlnx-agent start

== Network Node ==

To configure the Network node:

1. Make sure that eIPoIB module is up and configured in /etc/infiniband/openib.conf:
For more information, please refer to eIPoIB configuration in Mellanox OFED User Manual.
E_IPOIB_LOAD=yes
2. Restart openibd:
#service openibd restart
3. Modify the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows:
[linux_bridge]
physical_interface_mappings = default:<eIPoIB interface>

NOTE: In order to obtain the eIPoIB interface name, run the ethtool tool command (see below) and check the driver name:

#ethtool -i <eIPoIB_interface>
driver: eth_ipoib
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

4. Restart neutron-linuxbridge-agent and neutron-dhcp-agent:
#service neutron-linuxbridge-agent restart

NOTE: For DHCP support, the Network node should use the Mellanox Dnsmasq driver as the DHCP driver.

=== DHCP Server (Usally part of the Network node) ===

1. Install the Mellanox Dnsmasq:
#wget
http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-dnsmasq-2014.1.1-1.noarch.rpm
#yum localinstall mlnx-dnsmasq-2014.1.1-1.noarch.rpm

2. In addition, Dnsmasq must be upgraded to version 2.66 or higher:
#wget
ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/kalyaka/CentOS_CentOS-6/x86_64/dnsmasq-2.66-3.1.x86_64.rpm

#yum localinstall dnsmasq-2.66-3.1.x86_64.rpm
3. Modify /etc/neutron/dhcp_agent.ini as follows:
dhcp_driver = mlnx_dhcp.MlnxDnsmasq
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver
4. Start DHCP server:
#service neutron-dhcp-agent restart

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand

2014-11-30T08:25:14Z

Yaell:

= Overview =

This wiki page describes how to tune Mellanox OpenStack on Ubuntu 12.04 servers

= Prerequisites =

* A running OpenStack environment installed with the ML2 plugin on top of Linux Bridge.
* All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119)
* Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers
* SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community [http://community.mellanox.com/docs/DOC-1317 click here].
* The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes
* VLANs configured on the ports in the switch.

= InfiniBand Network =

== OpenSM Node ==

To configure OpenSM node:

1. Edit the file /etc/opensm/partitions.conf by adding the following line:
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;
Create the file if it does not exist.

For each VLAN value in the VLAN range used for VLAN network segmentation, add the following line in the file /etc/opensm/partitions.conf (for vlan-pkey association):
vlan<decimal-value>=0x<hexadecimal-value>, ipoib, sl=0, defmember=full: ALL;
In this manual, the VLAN range is 2-10 (mlnx_agent.conf /ml2_conf.ini). Therefore, add the following lines:
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;
2. Modify the following line in the file /etc/opensm/opensm.conf from FALSE to TRUE:
allow_both_pkeys TRUE
3. Restart OpenSM:
service opensmd restart
4. Verify that all ports on all nodes have their ib interface (i.e. ib0) active, run ibdev2netdev on all hosts, and make sure that the port is up:
mlx4_0 port 1 ==> ib0 (Up)
If the port is down, try bringing it up by running “ifconfig ib0 up”, and make sure your switch ports are of type “InfiniBand”, and that all nodes can reach the OpenSM node by running “sminfo”.

== Controller Node ==

To configure the Controller node:

1. Ensure that the file /etc/nova/nova.conf includes the following line:
security_group_api=nova

2. If the security group value is changed, restart the Nova services:
# for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done

=== Neutron Server Node ===

NOTE: The Neutron Server can sometimes be part of the Controller node.

To configure the Neutron Server node:

1. Edit /etc/neutron/neutron.conf and make sure Neutron is using ML2 plugin:
[DEFAULT]
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin
2. Edit the ML2 configuration file /etc/neutron/plugins/ml2/ml2_conf.ini as follows:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = linuxbridge,mlnx

[ml2_type_vlan]
network_vlan_ranges=<physical_network>:<min_vlan>:<max_vlan> //(i.e. default:2:10)

[securitygroup]
enable_security_group = True

[eswitch]
vnic_type = hostdev
apply_profile_patch = True
3. Install the required Deb for the Mellanox Neutron plugin (from Ubuntu’s repository):
apt-get install neutron-plugin-mlnx-agent
4. Edit /etc/neutron/plugins/mlnx/mlnx_conf.ini to match your environment:
[mlnx]
network_vlan_ranges = <physical_network>:<min_vlan:<max_vlan> //(i.e. default:2:10)
tenant_network_type = vlan

[eswitch]
physical_interface_mappings = <physical_network>:<ib interface> //(i.e. default:ib0)
vnic_type = hostdev

[agent]
support_old_agents = True

5. Restart Neutron Server:
/etc/init.d/neutron-server start

== Compute Nodes ==

NOTE: The Neutron Server can sometimes be part of the Controller node.

To configure the Neutron Server node:

1. Edit /etc/neutron/neutron.conf and make sure Neutron is using ML2 plugin:
[DEFAULT]
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin
2. Edit the ML2 configuration file /etc/neutron/plugins/ml2/ml2_conf.ini as follows:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = linuxbridge,mlnx

[ml2_type_vlan]
network_vlan_ranges=<physical_network>:<min_vlan>:<max_vlan> //(i.e. default:2:10)

[securitygroup]
enable_security_group = True

[eswitch]
vnic_type = hostdev
apply_profile_patch = True
3. Install the required Deb for the Mellanox Neutron plugin (from Ubuntu’s repository):
apt-get install neutron-plugin-mlnx-agent
4. Edit /etc/neutron/plugins/mlnx/mlnx_conf.ini to match your environment:
[mlnx]
network_vlan_ranges = <physical_network>:<min_vlan:<max_vlan> //(i.e. default:2:10)
tenant_network_type = vlan

[eswitch]
physical_interface_mappings = <physical_network>:<ib interface> //(i.e. default:ib0)
vnic_type = hostdev

[agent]
support_old_agents = True

5. Restart Neutron Server:
/etc/init.d/neutron-server start

== Network Node ==

To configure the Network node:

1. Configure eIPoIB (for Linux bridge plugin) by editing the file /etc/infiniband/openib.conf as follows:
E_IPOIB_LOAD=yes
2. Restart OFED:
service openibd restart
3. Verify that the new Ethernet interfaces now appear with ibdev2netdev:
# ibdev2netdev
mlx4_0 port 1 ==> ib0 (Up)
mlx4_0 port 2 ==> ib1 (Down)
mlx4_0 port 1 ==> eth2 (Up)
mlx4_0 port 2 ==> eth1 (Down)
In eIPoIB, every ib interface has a matching eth interface. The ib interface used here is ib0, and the matching one is eth2 (same card and port names: mlx4_0 port 1). By using Ethtool, you can verify that ib0 is working on ib_ipoib driver, and eth2 is working on eth_ipoib driver, as follows:
# ethtool -i ib0
driver: ib_ipoib
version: 2.3-1.5.4_Sep 30 2014
firmware-version: 2.32.5100
bus-info: 0000:07:00.0# ethtool -i eth2
driver: eth_ipoib
version: 2.3-1.5.4 (Sep 30 2014)
firmware-version: 1
bus-info: ib0
4. Edit /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows:
[linux_bridge]
physical_interface_mappings = <physical_network>:<eth_ipoib-interface>
//(i.e. default:eth2)
5. Restart the Linux bridge agent:
service neutron-linuxbridge-agent restart
6. To allow DHCP support, download mlnx-dnsmadq from this link: http://www.mellanox.com/downloads/solutions/openstack/icehouse/debian/

7. Install the package:
dpkg –i python-mlnx-dnsmasq_2014.1.1-1_all.deb
8. Configure the file /etc/neutron/dhcp_agent.ini:
[DEFAULT]
dhcp_driver = mlnx_dhcp.MlnxDnsmasq
9. Make sure that the Dnsmasq which is installed as a part of the OpenStack (not mlnx-dnsmasq) environment is of version 2.65.

10. Copy /etc/neutron/neutron.conf from the Compute node to the same path on the Network node.

11. Edit /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows:
[linux_bridge]
physical_interface_mappings = <physical_network>:<eth_ipoib interface>
(i.e. default:eth2)
12. Edit /etc/neutron/dhcp_agent.ini and /etc/neutron/l3_agent.ini as follows:
[DEFAULT]
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver
13. Restart the following services:
service neutron-plugin-linuxbridge-agent restart
service neutron-dhcp-agent restart
service neutron-l3-agent restart

== Troubleshooting ==

* If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.

Mellanox-Neutron-Icehouse-Redhat

2014-11-30T08:22:37Z

Yaell: Replaced content with "This page was redirected to [https://wiki.openstack.org/wiki/Mellanox-Neutron-ML2 here]"

This page was redirected to [https://wiki.openstack.org/wiki/Mellanox-Neutron-ML2 here]

Mellanox-Neutron-ML2-Icehouse1

2014-11-27T15:48:23Z

Yaell:

Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

* Mellanox Openstack Neutron Agent (L2 Agent) runs on each compute node.
* Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

For '''Icehouse''' release Information refer to the relevant OS as follows:

* [[Mellanox-Neutron-Icehouse-Redhat-Ethernet|Redhat Ethernet]]
* [[Mellanox-Neutron-Icehouse-Ubuntu-Ethernet|Ubuntu Ethernet]]
* [[Mellanox-Neutron-Icehouse-Redhat-InfiniBand|Redhat InfiniBand]]
* [[Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand|Ubuntu InfiniBand]]

Mellanox-Neutron-Icehouse-Redhat-InfiniBand

2014-11-27T15:32:06Z

Yaell:

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==
* A running OpenStack environment installed with the ML2 plugin on top of Linux Bridge.
* All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119)
* Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers
* SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community [http://community.mellanox.com/docs/DOC-1317 click here].
* The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes
* VLANs configured on the ports in the switch.

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Node ==

=== OpenSM Configuration – Without UFM ===

To configure OpenSM:

1. Make sure that all the PKeys are predefined in the partitions.conf file (/etc/opensm/partitions.conf).

2. Add/Change the following in the partitions.conf file:
management=0xffff,ipoib, sl=0, defmember=both: ALL, ALL_SWITCHES=full,SELF=full;
3. For every network you want to configure in Neutron, you have to configure the PKey associated with the VLAN of this network (defined in Neutron):
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
Below is an example of the configuration of the partitions.conf file in case you have 10 VLANs defined in the configuration of the file: /etc/neutron/plugins/mlnx/mlnx_conf.ini. ( network_vlan_ranges = default:1:10)

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full: ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full: ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full: ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full: ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full: ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full: ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full: ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full: ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full: ALL_CAS;

4. Modify the following line in the file /etc/opensm/opensm.conf from FALSE to TRUE:
allow_both_pkeys TRUE
5. Restart the OpenSM:
#service opensmd restart

=== OpenSM configuration - With UFM===

To configure OpenSM:

1. Make sure that UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf by changing the following values:

a. Set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):
Allow both full and limited membership on the same partition
allow_both_pkeys TRUE
b. Set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port):
SM assigned Alias GUIDs algorithm sm_assign_guid_func uniq_count
3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a. Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty right after UFM installation)
Add the following line to the file to enable both full and limited management PKeys:
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;
b. Add the additional PKey definitions relevant to the specific setup.
For Example:
vlan1=0x1, ipoib, sl=0, defmember=full: ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full: ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full: ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full: ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full: ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full: ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full: ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full: ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full: ALL_CAS;
4. Restart UFM

• Stand-alone
#/etc/init.d/ufmd restart
• High-availability
#/etc/init.d/ufmha restart

== Neutron Server Node==

To configure the Neutron Server node:

1. Make sure ML2 is the current Neutron plugin by checking the core_plugin parameter in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin
2. Make sure /etc/neutron/plugin.ini is pointing to /etc/neutron/plugins/ml2/ml2_conf.ini (symbolic link)

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini by adding the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = linuxbridge,mlnx
[ml2_type_vlan]
network_vlan_ranges = <network_name>:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True
4. Start (or restart) the Neutron server:
#service neutron-server restart

== Compute Nodes ==

To configure the Compute Node:

1. Download Mellanox OpenStack repo file:
#wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo -O /etc/yum.repos.d/mlnx-icehouse.repo
2. Install the eSwitch Daemon (eSwitchd) RPM:
#yum install eswitchd
3. Install Mellanox VIF driver:
#yum install mlnxvif
3. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox
4. Create the file /etc/modprobe.d/mlx4_ib.conf and add the following:
options mlx4_ib sm_guid_assign=0
5. Restart the driver:
#service openibd restart
6.In the file /etc/neutron/plugins/mlnx/mlnx_conf.ini, the parameters tenant_network_type , vnic_type and network_vlan_ranges should be configured as the controllers:
physical_interface_mapping = default:ib<your_interface>
7. Modify the file /etc/eswitchd/eswitchd.conf as follows:
fabrics = default:ib<your_interface> (for example default:ib0)
8. Verify Mellanox VIF driver is configured in /etc/nova/nova.conf:
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver
9. Restart Nova:
#service openstack-nova-compute restart
10. Start eSwitchd:
#service eswitchd start
11. Start the Neutron agent:
#service neutron-mlnx-agent start

== Network Node ==

To configure the Network node:

1. Make sure that eIPoIB module is up and configured in /etc/infiniband/openib.conf:
For more information, please refer to eIPoIB configuration in Mellanox OFED User Manual.
E_IPOIB_LOAD=yes
2. Restart openibd:
#service openibd restart
3. Modify the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows:
[linux_bridge]
physical_interface_mappings = default:<eIPoIB interface>

NOTE: In order to obtain the eIPoIB interface name, run the ethtool tool command (see below) and check the driver name:

#ethtool -i <InfiniBand_interface>
driver: eth_ipoib
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

4. Restart neutron-linuxbridge-agent and neutron-dhcp-agent:
#service neutron-linuxbridge-agent restart

NOTE: For DHCP support, the Network node should use the Mellanox Dnsmasq driver as the DHCP driver.

=== DHCP Server (Usally part of the Network node) ===

1. Install the Mellanox Dnsmasq:
#wget
http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-dnsmasq-2014.1.1-1.noarch.rpm
#yum localinstall mlnx-dnsmasq-2014.1.1-1.noarch.rpm

2. In addition, Dnsmasq must be upgraded to version 2.66 or higher:
#wget
ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/kalyaka/CentOS_CentOS-6/x86_64/dnsmasq-2.66-3.1.x86_64.rpm

#yum localinstall dnsmasq-2.66-3.1.x86_64.rpm
3. Modify /etc/neutron/dhcp_agent.ini as follows:
dhcp_driver = mlnx_dhcp.MlnxDnsmasq
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver
4. Start DHCP server:
#service neutron-dhcp-agent restart

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat-InfiniBand

2014-11-27T15:30:24Z

Yaell:

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==
* A running OpenStack environment installed with the ML2 plugin on top of Linux Bridge.
* All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119)
* Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers
* SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community [http://community.mellanox.com/docs/DOC-1317 click here].
* The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes
* VLANs configured on the ports in the switch.

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Node ==

=== OpenSM Configuration – Without UFM ===

To configure OpenSM:

1. Make sure that all the PKeys are predefined in the partitions.conf file (/etc/opensm/partitions.conf).

2. Add/Change the following in the partitions.conf file:
management=0xffff,ipoib, sl=0, defmember=both: ALL, ALL_SWITCHES=full,SELF=full;
3. For every network you want to configure in Neutron, you have to configure the PKey associated with the VLAN of this network (defined in Neutron):
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
Below is an example of the configuration of the partitions.conf file in case you have 10 VLANs defined in the configuration of the file: /etc/neutron/plugins/mlnx/mlnx_conf.ini. ( network_vlan_ranges = default:1:10)

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full: ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full: ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full: ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full: ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full: ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full: ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full: ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full: ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full: ALL_CAS;

4. Modify the following line in the file /etc/opensm/opensm.conf from FALSE to TRUE:
allow_both_pkeys TRUE
5. Restart the OpenSM:
#service opensmd restart

=== OpenSM configuration - With UFM===

To configure OpenSM:

1. Make sure that UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf by changing the following values:

a. Set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):
Allow both full and limited membership on the same partition
allow_both_pkeys TRUE
b. Set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port):
SM assigned Alias GUIDs algorithm sm_assign_guid_func uniq_count
3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a. Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty right after UFM installation)
Add the following line to the file to enable both full and limited management PKeys:
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;
b. Add the additional PKey definitions relevant to the specific setup.
For Example:
vlan1=0x1, ipoib, sl=0, defmember=full: ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full: ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full: ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full: ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full: ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full: ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full: ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full: ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full: ALL_CAS;
4. Restart UFM

• Stand-alone
#/etc/init.d/ufmd restart
• High-availability
#/etc/init.d/ufmha restart

== Neutron Server Node==

To configure the Neutron Server node:

1. Make sure ML2 is the current Neutron plugin by checking the core_plugin parameter in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin
2. Make sure /etc/neutron/plugin.ini is pointing to /etc/neutron/plugins/ml2/ml2_conf.ini (symbolic link)

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini by adding the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = linuxbridge,mlnx
[ml2_type_vlan]
network_vlan_ranges = <network_name>:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True
4. Start (or restart) the Neutron server:
#service neutron-server restart

== Compute Nodes ==

To configure the Compute Node:

1. Download Mellanox OpenStack repo file:
#wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo -O /etc/yum.repos.d/mlnx-icehouse.repo
2. Install the eSwitch Daemon (eSwitchd) RPM:
#yum install eswitchd
3. Install Mellanox VIF drive
#yum install mlnxvif
3. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox
4. Create the file /etc/modprobe.d/mlx4_ib.conf and add the following:
options mlx4_ib sm_guid_assign=0
5. Restart the driver:
#service openibd restart
6.In the file /etc/neutron/plugins/mlnx/mlnx_conf.ini, the parameters tenant_network_type , vnic_type and network_vlan_ranges should be configured as the controllers:
physical_interface_mapping = default:ib<your_interface>
7. Modify the file /etc/eswitchd/eswitchd.conf as follows:
fabrics = default:ib<your_interface> (for example default:ib0)
8. Verify Mellanox VIF driver is configured in /etc/nova/nova.conf:
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver
9. Restart Nova:
#service openstack-nova-compute restart
10. Start eSwitchd:
#service eswitchd start
11. Start the Neutron agent:
#service neutron-mlnx-agent start

== Network Node ==

To configure the Network node:

1. Make sure that eIPoIB module is up and configured in /etc/infiniband/openib.conf:
For more information, please refer to eIPoIB configuration in Mellanox OFED User Manual.
E_IPOIB_LOAD=yes
2. Restart openibd:
#service openibd restart
3. Modify the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows:
[linux_bridge]
physical_interface_mappings = default:<eIPoIB interface>

NOTE: In order to obtain the eIPoIB interface name, run the ethtool tool command (see below) and check the driver name:

#ethtool -i <InfiniBand_interface>
driver: eth_ipoib
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

4. Restart neutron-linuxbridge-agent and neutron-dhcp-agent:
#service neutron-linuxbridge-agent restart

NOTE: For DHCP support, the Network node should use the Mellanox Dnsmasq driver as the DHCP driver.

=== DHCP Server (Usally part of the Network node) ===

1. Install the Mellanox Dnsmasq:
#wget
http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-dnsmasq-2014.1.1-1.noarch.rpm
#yum localinstall mlnx-dnsmasq-2014.1.1-1.noarch.rpm

2. In addition, Dnsmasq must be upgraded to version 2.66 or higher:
#wget
ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/kalyaka/CentOS_CentOS-6/x86_64/dnsmasq-2.66-3.1.x86_64.rpm

#yum localinstall dnsmasq-2.66-3.1.x86_64.rpm
3. Modify /etc/neutron/dhcp_agent.ini as follows:
dhcp_driver = mlnx_dhcp.MlnxDnsmasq
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver
4. Start DHCP server:
#service neutron-dhcp-agent restart

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat-InfiniBand

2014-11-27T15:29:33Z

Yaell:

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==
* A running OpenStack environment installed with the ML2 plugin on top of Linux Bridge.
* All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119)
* Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers
* SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community [http://community.mellanox.com/docs/DOC-1317 click here].
* The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes
* VLANs configured on the ports in the switch.

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Node ==

=== OpenSM Configuration – Without UFM ===

To configure OpenSM:

1. Make sure that all the PKeys are predefined in the partitions.conf file (/etc/opensm/partitions.conf).

2. Add/Change the following in the partitions.conf file:
management=0xffff,ipoib, sl=0, defmember=both: ALL, ALL_SWITCHES=full,SELF=full;
3. For every network you want to configure in Neutron, you have to configure the PKey associated with the VLAN of this network (defined in Neutron):
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
Below is an example of the configuration of the partitions.conf file in case you have 10 VLANs defined in the configuration of the file: /etc/neutron/plugins/mlnx/mlnx_conf.ini. ( network_vlan_ranges = default:1:10)

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full: ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full: ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full: ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full: ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full: ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full: ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full: ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full: ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full: ALL_CAS;

4. Modify the following line in the file /etc/opensm/opensm.conf from FALSE to TRUE:
allow_both_pkeys TRUE
5. Restart the OpenSM:
#service opensmd restart

=== OpenSM configuration - With UFM===

To configure OpenSM:

1. Make sure that UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf by changing the following values:

a. Set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):
Allow both full and limited membership on the same partition
allow_both_pkeys TRUE
b. Set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port):
SM assigned Alias GUIDs algorithm sm_assign_guid_func uniq_count
3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a. Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty right after UFM installation)
Add the following line to the file to enable both full and limited management PKeys:
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;
b. Add the additional PKey definitions relevant to the specific setup.
For Example:
vlan1=0x1, ipoib, sl=0, defmember=full: ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full: ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full: ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full: ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full: ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full: ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full: ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full: ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full: ALL_CAS;
4. Restart UFM

• Stand-alone
#/etc/init.d/ufmd restart
• High-availability
#/etc/init.d/ufmha restart

== Neutron Server Node==

To configure the Neutron Server node:

1. Make sure ML2 is the current Neutron plugin by checking the core_plugin parameter in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin
2. Make sure /etc/neutron/plugin.ini is pointing to /etc/neutron/plugins/ml2/ml2_conf.ini (symbolic link)

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini by adding the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = linuxbridge,mlnx
[ml2_type_vlan]
network_vlan_ranges = <network_name>:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True
4. Start (or restart) the Neutron server:
#service neutron-server restart

== Compute Nodes ==

To configure the Compute Node:

1. Download Mellanox OpenStack repo file:
#wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo -O /etc/yum.repos.d/mlnx-icehouse.repo
2. Install the eSwitch Daemon (eSwitchd) RPM:
#yum install eswitchd
3. Install Mellanox VIF drive
#yum install mlnxvif
3. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox
4. Create the file /etc/modprobe.d/mlx4_ib.conf and add the following:
options mlx4_ib sm_guid_assign=0
5. Restart the driver:
#service openibd restart
6.In the file /etc/neutron/plugins/mlnx/mlnx_conf.ini, the parameters tenant_network_type , vnic_type and network_vlan_ranges should be configured as the controllers:
physical_interface_mapping = default:ib<your_interface>
7. Modify the file /etc/eswitchd/eswitchd.conf as follows:
fabrics = default:ib<your_interface> (for example default:ib0)
8. Verify Mellanox VIF driver is configured in /etc/nova/nova.conf:
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver
9. Restart Nova:
#service openstack-nova-compute restart
10. Start eSwitchd:
#service eswitchd start
11. Start the Neutron agent:
#service neutron-mlnx-agent start

== Network Node ==

To configure the Network node:

1. Make sure that eIPoIB module is up and configured in /etc/infiniband/openib.conf:
For more information, please refer to eIPoIB configuration in Mellanox OFED User Manual.
E_IPOIB_LOAD=yes
2. Restart openibd:
#service openibd restart
3. Modify the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows:
[linux_bridge]
physical_interface_mappings = default:<eIPoIB interface>

NOTE: In order to obtain the eIPoIB interface name, run the ethtool tool command (see below) and check the driver name:

#ethtool -i <InfiniBand_interface>
driver: eth_ipoib
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

4. Restart neutron-linuxbridge-agent and neutron-dhcp-agent:
#service neutron-linuxbridge-agent restart

NOTE: For DHCP support, the Network node should use the Mellanox Dnsmasq driver as the DHCP driver.

=== DHCP Server (Usally part of the Network node) ===

1. Install the Mellanox Dnsmasq:
#wget
http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-dnsmasq-2014.1.1-1.noarch.rpm
#yum localinstall mlnx-dnsmasq-2014.1.1-1.noarch.rpm

2. In addition, Dnsmasq must be upgraded to version 2.66 or higher:
#wget
ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/kalyaka/CentOS_CentOS-6/x86_64/dnsmasq-2.66-3.1.x86_64.rpm

#yum localinstall dnsmasq-2.66-3.1.x86_64.rpm
3. Modify /etc/neutron/dhcp_agent.ini as follows:
dhcp_driver = mlnx_dhcp.MlnxDnsmasq
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver
4. Start DHCP server:
#service neutron-dhcp-agent restart

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.

Mellanox-Neutron-Icehouse-Redhat-InfiniBand

2014-11-27T15:21:35Z

Yaell:

= Overview =
== Mellanox Neutron ML2 Driver ==
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.

This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.

Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) and MACVTAP (virtual interface with a tap-like software interface) vnic types. For vnic type configuration API details, please refer to configuration reference guide (click [http://docs.openstack.org/api/openstack-network/2.0/content/binding_ext_ports.html here]). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).

The driver supports VLAN network type to facilitate virtual networks either on Ethernet or InfiniBand fabrics.

• Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node.

• Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.

== Mellanox Neutron Plugin ==
Please note,
Mellanox Plug-in is deprecated in the Icehouse release and won't be supported in the Juno release.
The features in the plug-in are now part of the ML2 plug-in in the form of Mellanox mechanism driver.

For details regarding Mellanox Neutron plugin, please refer to https://wiki.openstack.org/wiki/Mellanox-Neutron-Havana-Redhat.

== Mellanox Nova VIF Driver ==
The Mellanox Nova VIF driver should be used when running Mellanox Mechnism Driver. The VIF driver supports the VIF plugin by binding vNIC of type DIRECT to the embedded switch port.
VIF Driver for MACVTAP type is included in Nova libvirt generic vif driver. For SR-IOV pass-through (vnic type DIRECT) one needs to use VIF driver from Mellanox git repository or RPM.

== Prerequisites ==
* A running OpenStack environment installed with the ML2 plugin on top of Linux Bridge.
* All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119)
* Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers
* SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community [http://community.mellanox.com/docs/DOC-1317 click here].
* The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes
* VLANs configured on the ports in the switch.

= InfiniBand Network =
The Mellanox Neutron Plugin use InfiniBand Partitions (PKeys) to separate Networks.

== SM Node ==

=== OpenSM Configuration – Without UFM ===

To configure OpenSM:

1. Make sure that all the PKeys are predefined in the partitions.conf file (/etc/opensm/partitions.conf).

2. Add/Change the following in the partitions.conf file:
management=0xffff,ipoib, sl=0, defmember=both: ALL, ALL_SWITCHES=full,SELF=full;
3. For every network you want to configure in Neutron, you have to configure the PKey associated with the VLAN of this network (defined in Neutron):
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
Below is an example of the configuration of the partitions.conf file in case you have 10 VLANs defined in the configuration of the file: /etc/neutron/plugins/mlnx/mlnx_conf.ini. ( network_vlan_ranges = default:1:10)

management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full: ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full: ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full: ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full: ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full: ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full: ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full: ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full: ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full: ALL_CAS;

4. Modify the following line in the file /etc/opensm/opensm.conf from FALSE to TRUE:
allow_both_pkeys TRUE
5. Restart the OpenSM:
#service opensmd restart

=== OpenSM configuration - With UFM===

To configure OpenSM:

1. Make sure that UFM is installed and connected to your fabric.

2. Edit /opt/ufm/conf/opensm/opensm.conf by changing the following values:

a. Set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):
Allow both full and limited membership on the same partition
allow_both_pkeys TRUE
b. Set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port):
SM assigned Alias GUIDs algorithm sm_assign_guid_func uniq_count
3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

a. Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty right after UFM installation)
Add the following line to the file to enable both full and limited management PKeys:
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;
b. Add the additional PKey definitions relevant to the specific setup.
For Example:
vlan1=0x1, ipoib, sl=0, defmember=full: ALL_CAS;
vlan2=0x2, ipoib, sl=0, defmember=full: ALL_CAS;
vlan3=0x3, ipoib, sl=0, defmember=full: ALL_CAS;
vlan4=0x4, ipoib, sl=0, defmember=full: ALL_CAS;
vlan5=0x5, ipoib, sl=0, defmember=full: ALL_CAS;
vlan6=0x6, ipoib, sl=0, defmember=full: ALL_CAS;
vlan7=0x7, ipoib, sl=0, defmember=full: ALL_CAS;
vlan8=0x8, ipoib, sl=0, defmember=full: ALL_CAS;
vlan9=0x9, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xa, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xb, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xc, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xd, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xe, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0xf, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0x10, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0x11, ipoib, sl=0, defmember=full: ALL_CAS;
vlan10=0x12, ipoib, sl=0, defmember=full: ALL_CAS;
4. Restart UFM

• Stand-alone
#/etc/init.d/ufmd restart
• High-availability
#/etc/init.d/ufmha restart

== Neutron Server Node==

To configure the Neutron Server node:

1. Make sure ML2 is the current Neutron plugin by checking the core_plugin parameter in /etc/neutron/neutron.conf:
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin
2. Make sure /etc/neutron/plugin.ini is pointing to /etc/neutron/plugins/ml2/ml2_conf.ini (symbolic link)

3. Modify /etc/neutron/plugins/ml2/ml2_conf.ini by adding the following:
[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan
mechanism_drivers = openvswitch,mlnx
[ml2_type_vlan]
network_vlan_ranges = <network_name>:2:100
[eswitch]
vnic_type = hostdev
apply_profile_patch = True
4. Start (or restart) the Neutron server:
#service neutron-server restart

== Compute Nodes ==

To configure the Compute Node:

1. Download Mellanox OpenStack repo file:
#wget http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-icehouse.repo -O /etc/yum.repos.d/mlnx-icehouse.repo
2. Install the eSwitch Daemon (eSwitchd) RPM:
#yum install eswitchd
3. Install the required RPM for the Neutron agent:
#yum install openstack-neutron-mellanox
4. Create the file /etc/modprobe.d/mlx4_ib.conf and add the following:
options mlx4_ib sm_guid_assign=0
In the file /etc/neutron/plugins/mlnx/mlnx_conf.ini, the parameters tenant_network_type , vnic_type and network_vlan_ranges should be configured as the controllers:
physical_interface_mapping = default:ib<your_interface>
5. Modify the file /etc/eswitchd/eswitchd.conf as follows:
fabrics = default:ib<your_interface> (for example default:ib0)
6. Restart the driver:
#service openibd restart
7. Restart eSwitchd and the Neutron agent:
#service eswitchd restart
#service neutron-mlnx-agent restart
8. Verify Mellanox VIF driver is configured in /etc/nova/nova.conf:
[libvirt]
vif_driver=mlnxvif.vif.MlxEthVIFDriver
9. Restart Nova:
#service openstack-nova-compute restart
10. Start eSwitchd:
#service eswitchd start
11. Start the Neutron agent:
#service neutron-mlnx-agent start

== Network Node ==

To configure the Network node:

1. Make sure that eIPoIB module is up and configured in /etc/infiniband/openib.conf:
For more information, please refer to eIPoIB configuration in Mellanox OFED User Manual.
E_IPOIB_LOAD=yes
2. Restart openibd:
#service openibd restart
3. Modify the file /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows:
[linux_bridge]
physical_interface_mappings = default:<eIPoIB interface>

NOTE: In order to obtain the eIPoIB interface name, run the ethtool tool command (see below) and check the driver name:

#ethtool -i <InfiniBand_interface>
driver: eth_ipoib
version: 1.0.0
firmware-version: 1
bus-info: ib0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

4. Restart neutron-linuxbridge-agent and neutron-dhcp-agent:
#service neutron-linuxbridge-agent restart

NOTE: For DHCP support, the Network node should use the Mellanox Dnsmasq driver as the DHCP driver.

=== DHCP Server (Usally part of the Network node) ===

1. Install the Mellanox Dnsmasq:
#wget
http://www.mellanox.com/downloads/solutions/openstack/icehouse/repo/mlnx-icehouse/mlnx-dnsmasq-2014.1.1-1.noarch.rpm
#yum localinstall mlnx-dnsmasq-2014.1.1-1.noarch.rpm

2. In addition, Dnsmasq must be upgraded to version 2.66 or higher:
#wget
ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/home:/kalyaka/CentOS_CentOS-6/x86_64/dnsmasq-2.66-3.1.x86_64.rpm

#yum localinstall dnsmasq-2.66-3.1.x86_64.rpm
3. Modify /etc/neutron/dhcp_agent.ini as follows:
dhcp_driver = mlnx_dhcp.MlnxDnsmasq
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver
4. Start DHCP server:
#service neutron-dhcp-agent restart

= Known issues and Troubleshooting =

For known issues and troubleshooting options refer to [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting].

= References =
1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]

2. [https://github.com/mellanox-openstack Source repository]

3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]

4. [http://www.mellanox.com/openstack/pdf/mellanox-openstack-solution.pdf Mellanox OpenStack Solution Reference Architecture]

5. [http://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]

For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]

Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.