Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand
Contents
Overview
This wiki page describes how to tune Mellanox OpenStack on Ubuntu 12.04 servers
Prerequisites
- A running OpenStack environment installed with the ML2 plugin on top of Linux Bridge.
- All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119)
- Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers
- SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community click here.
- The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes
- VLANs configured on the ports in the switch.
InfiniBand Network
OpenSM Node
To configure OpenSM node:
1. Edit the file /etc/opensm/partitions.conf by adding the following line:
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;
Create the file if it does not exist.
For each VLAN value in the VLAN range used for VLAN network segmentation, add the following line in the file /etc/opensm/partitions.conf (for vlan-pkey association):
vlan<decimal-value>=0x<hexadecimal-value>, ipoib, sl=0, defmember=full: ALL;
In this manual, the VLAN range is 2-10 (mlnx_agent.conf /ml2_conf.ini). Therefore, add the following lines:
vlan1=0x1, ipoib, sl=0, defmember=full : ALL; vlan2=0x2, ipoib, sl=0, defmember=full : ALL; vlan3=0x3, ipoib, sl=0, defmember=full : ALL; vlan4=0x4, ipoib, sl=0, defmember=full : ALL; vlan5=0x5, ipoib, sl=0, defmember=full : ALL; vlan6=0x6, ipoib, sl=0, defmember=full : ALL; vlan7=0x7, ipoib, sl=0, defmember=full : ALL; vlan8=0x8, ipoib, sl=0, defmember=full : ALL; vlan9=0x9, ipoib, sl=0, defmember=full : ALL; vlan10=0xa, ipoib, sl=0, defmember=full : ALL;
2. Modify the following line in the file /etc/opensm/opensm.conf from FALSE to TRUE:
allow_both_pkeys TRUE
3. Restart OpenSM:
service opensmd restart
4. Verify that all ports on all nodes have their ib interface (i.e. ib0) active, run ibdev2netdev on all hosts, and make sure that the port is up:
mlx4_0 port 1 ==> ib0 (Up)
If the port is down, try bringing it up by running “ifconfig ib0 up”, and make sure your switch ports are of type “InfiniBand”, and that all nodes can reach the OpenSM node by running “sminfo”.
Controller Node
To configure the Controller node:
1. Ensure that the file /etc/nova/nova.conf includes the following line:
security_group_api=nova
2. If the security group value is changed, restart the Nova services:
# for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done
Neutron Server Node
NOTE: The Neutron Server can sometimes be part of the Controller node.
To configure the Neutron Server node:
1. Edit /etc/neutron/neutron.conf and make sure Neutron is using ML2 plugin:
[DEFAULT] core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin
2. Edit the ML2 configuration file /etc/neutron/plugins/ml2/ml2_conf.ini as follows:
[ml2] type_drivers = vlan,flat tenant_network_types = vlan mechanism_drivers = linuxbridge,mlnx [ml2_type_vlan] network_vlan_ranges=<physical_network>:<min_vlan>:<max_vlan> //(i.e. default:2:10) [securitygroup] enable_security_group = True [eswitch] vnic_type = hostdev apply_profile_patch = True
3. Install the required Deb for the Mellanox Neutron plugin (from Ubuntu’s repository):
apt-get install neutron-plugin-mlnx-agent
4. Edit /etc/neutron/plugins/mlnx/mlnx_conf.ini to match your environment:
[mlnx] network_vlan_ranges = <physical_network>:<min_vlan:<max_vlan> //(i.e. default:2:10) tenant_network_type = vlan [eswitch] physical_interface_mappings = <physical_network>:<ib interface> //(i.e. default:ib0) vnic_type = hostdev [agent] support_old_agents = True
5. Restart Neutron Server:
/etc/init.d/neutron-server start
Compute Nodes
To configure the Compute node: 1. Create the file /etc/modprobe.d/mlx4_ib.conf with this line:
options mlx4_ib sm_guid_assign=0
2. Download mlnxvif, eSwitch Daemon (eSwitchd), and mlnx-dnsmasq from this link:
http://www.mellanox.com/downloads/solutions/openstack/icehouse/debian/
3. Install the Mellanox plugin services mlnxvif and eSwitchd:
dpkg –i python-mlnxvif_2014.1.1-1_all.deb dpkg –i python-eswitchd_0.10-1_all.deb
4. Edit /etc/eswitchd/eswitchd.conf as follows:
[DAEMON] fabrics=<physical_network>:<ib-interface> //(i.e. default:ib0)
5. Edit /etc/nova/nova.conf as follows:
[DEFAULT] compute_driver = libvirt.LibvirtDriver libvirt_vif_driver = mlnxvif.vif.MlxEthVIFDriver security_group_api = nova
6. Check if “ebrctl” appears in one of Nova’s rootwrap files:
grep 'ebrctl' /etc/nova/rootwrap.d/* If “ebrctl” does not appear in one of Nova’s rootwrap files, edit the file /etc/nova/rootwrap.d/compute.filter as follows: [Filters] ebrctl: CommandFilter, ebrctl, root
7. Install the Neutron Mellanox agent (from Ubuntu repository):
apt-get install neutron-plugin-mlnx-agent
8. Copy /etc/neutron/plugins/mlnx/mlnx_conf.ini from the Neutron Server node to the same path on each Compute node. 9. Restart the following services:
/etc/init.d/nova-compute restart /etc/init.d/eswitchd start /etc/init.d/neutron-plugin-mlnx-agent start
Network Node
To configure the Network node:
1. Configure eIPoIB (for Linux bridge plugin) by editing the file /etc/infiniband/openib.conf as follows:
E_IPOIB_LOAD=yes
2. Restart OFED:
service openibd restart
3. Verify that the new Ethernet interfaces now appear with ibdev2netdev:
# ibdev2netdev mlx4_0 port 1 ==> ib0 (Up) mlx4_0 port 2 ==> ib1 (Down) mlx4_0 port 1 ==> eth2 (Up) mlx4_0 port 2 ==> eth1 (Down)
In eIPoIB, every ib interface has a matching eth interface. The ib interface used here is ib0, and the matching one is eth2 (same card and port names: mlx4_0 port 1). By using Ethtool, you can verify that ib0 is working on ib_ipoib driver, and eth2 is working on eth_ipoib driver, as follows:
# ethtool -i ib0 driver: ib_ipoib version: 2.3-1.5.4_Sep 30 2014 firmware-version: 2.32.5100 bus-info: 0000:07:00.0# ethtool -i eth2 driver: eth_ipoib version: 2.3-1.5.4 (Sep 30 2014) firmware-version: 1 bus-info: ib0
4. Edit /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows:
[linux_bridge] physical_interface_mappings = <physical_network>:<eth_ipoib-interface> //(i.e. default:eth2)
5. Restart the Linux bridge agent:
service neutron-linuxbridge-agent restart
6. To allow DHCP support, download mlnx-dnsmadq from this link: http://www.mellanox.com/downloads/solutions/openstack/icehouse/debian/
7. Install the package:
dpkg –i python-mlnx-dnsmasq_2014.1.1-1_all.deb
8. Configure the file /etc/neutron/dhcp_agent.ini:
[DEFAULT] dhcp_driver = mlnx_dhcp.MlnxDnsmasq
9. Make sure that the Dnsmasq which is installed as a part of the OpenStack (not mlnx-dnsmasq) environment is of version 2.65.
10. Copy /etc/neutron/neutron.conf from the Compute node to the same path on the Network node.
11. Edit /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows:
[linux_bridge] physical_interface_mappings = <physical_network>:<eth_ipoib interface> (i.e. default:eth2)
12. Edit /etc/neutron/dhcp_agent.ini and /etc/neutron/l3_agent.ini as follows:
[DEFAULT] interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver
13. Restart the following services:
service neutron-plugin-linuxbridge-agent restart service neutron-dhcp-agent restart service neutron-l3-agent restart
Troubleshooting
- If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.
