Jump to: navigation, search

Difference between revisions of "Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand"

(→‎Ethernet Network)
m
 
(14 intermediate revisions by 2 users not shown)
Line 5: Line 5:
 
= Prerequisites =
 
= Prerequisites =
  
−
* A running OpenStack environment  installed with the ML2 plugin on top of OVS.
+
* A running OpenStack environment  installed with the ML2 plugin on top of Linux Bridge.
 
* All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119)  
 
* All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119)  
 
* Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers
 
* Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers
Line 14: Line 14:
 
= InfiniBand Network =  
 
= InfiniBand Network =  
  
−
== Controller Node ==  
+
== OpenSM Node ==
  
−
In /etc/nova/nova.conf ensure you have the following line:  
+
To configure OpenSM node:  
−
  security_group_api=nova
 
  
−
If you change, restart the nova services
+
1. Edit the file /etc/opensm/partitions.conf by adding the following line:
−
  # for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done
+
management=0xffff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
 +
Create the file if it does not exist.
  
−
== Neutron Server Node ==
+
For each VLAN value in the VLAN range used for VLAN network segmentation, add the following line in the file /etc/opensm/partitions.conf (for vlan-pkey association):
−
=== Installation ===  
+
vlan<decimal-value>=0x<hexadecimal-value>, ipoib, sl=0, defmember=full: ALL;
 +
In this manual, the VLAN range is 2-10 (mlnx_agent.conf /ml2_conf.ini). Therefore, add the following lines:
 +
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
 +
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
 +
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
 +
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
 +
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
 +
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
 +
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
 +
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
 +
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
 +
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;
 +
2. Modify the following line in the file /etc/opensm/opensm.conf from FALSE to TRUE:
 +
allow_both_pkeys TRUE
 +
3. Restart OpenSM:
 +
service opensmd restart
 +
4. Verify that all ports on all nodes have their ib interface (i.e. ib0) active, run ibdev2netdev on all hosts, and make sure that the port is up:
 +
mlx4_0 port 1 ==> ib0 (Up)
 +
If the port is down, try bringing it up by running “ifconfig ib0 up”, and make sure your switch ports are of type “InfiniBand”, and that all nodes can reach the OpenSM node by running “sminfo”.
  
−
1. Neutron server uses MySQL database. Make sure you have running MySQL database for neutron.
+
== Controller Node ==
  
−
If Neutron server is already running, you should stop it.
+
To configure the Controller node:
  
−
  #/etc/init.d/neutron-server stop
+
1. Ensure that the file /etc/nova/nova.conf includes the following line:
 +
security_group_api=nova
  
−
If you want to use a database that was created you have to drop it and create another one. For example:  
+
2. If the security group value is changed, restart the Nova services:
 +
# for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done
  
−
  mysql -u root -ppassword <<EOF
+
== Neutron Server Node ==
−
  drop database if exists neutron;
 
−
  create database neutron;
 
−
  EOF
 
  
−
If you want to create a new one:  
+
NOTE: The Neutron Server can sometimes be part of the Controller node.
  
−
  mysql -u root -ppassword <<EOF
+
To configure the Neutron Server node:
−
  drop database if exists neutron;
 
−
  create database neutron;
 
−
  GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'localhost' IDENTIFIED BY 'password';
 
−
  GRANT ALL PRIVILEGES ON neutron.* TO 'neutron' IDENTIFIED BY 'password';
 
−
  FLUSH PRIVILEGES;
 
−
  EOF
 
  
−
2. Install the required Deb for the Mellanox Neutron plugin:  
+
1. Edit /etc/neutron/neutron.conf and make sure Neutron is using ML2 plugin:
−
 
+
[DEFAULT]
−
  apt-get install -y neutron-plugin-mlnx-agent
+
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin
−
 
+
2. Edit the ML2 configuration file /etc/neutron/plugins/ml2/ml2_conf.ini as follows:
−
3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.
+
[ml2]
−
 
+
type_drivers = vlan,flat
−
4. Change /etc/default/neutron-server :
+
tenant_network_types = vlan
−
 
+
mechanism_drivers = linuxbridge,mlnx
−
  NEUTRON_PLUGIN_CONFIG="/etc/neutron/plugins/mlnx/mlnx_conf.ini"
+
 
−
 
+
[ml2_type_vlan]
−
=== Configuration ===
+
network_vlan_ranges=<physical_network>:<min_vlan>:<max_vlan> //(i.e. default:2:10)
−
 
+
−
1. Make the Mellanox plugin the current Neutron plugin by editing neutron.conf and changing core_plugin.
+
[securitygroup]
−
 
+
enable_security_group = True
−
  core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin
+
−
 
+
[eswitch]
−
2. Plugin configuration: Edit the configuration file: /etc/neutron/plugins/mlnx/mlnx_conf.ini  
+
vnic_type = hostdev
−
 
+
apply_profile_patch = True
−
  [MLNX]
+
3. Install the required Deb for the Mellanox Neutron plugin (from Ubuntu’s repository):
−
  tenant_network_type - must be set on of supported tenant network types.  
+
apt-get install neutron-plugin-mlnx-agent
−
                        Possible values: 'vlan' for Ethernet or 'ib' for Infiniband.'vlan' is the default.
+
4. Edit /etc/neutron/plugins/mlnx/mlnx_conf.ini to match your environment:
−
  network_vlan_ranges - must be configured to specify the names of the physical   
+
[mlnx]
 +
network_vlan_ranges = <physical_network>:<min_vlan:<max_vlan> //(i.e. default:2:10)
 +
tenant_network_type = vlan
 +
   
 +
[eswitch]
 +
physical_interface_mappings = <physical_network>:<ib interface>  //(i.e. default:ib0)
 +
vnic_type = hostdev
 +
 +
[agent]
 +
  support_old_agents = True
 
   
 
   
−
  networks managed by the mellanox plugin,
+
5. Restart Neutron Server:
−
  along with the ranges of VLAN IDs available on each physical network for allocation to virtual networks. 
+
  /etc/init.d/neutron-server start
−
  Possible range is vlan range is 1-4093.  
 
−
  The default is "default:1:100"
 
−
  <fabric name >:<vlan range start>:<vlan end range>
 
−
 
 
−
  [agent]
 
−
  rpc_support_old_agents = True
 
−
  Note: rpc_support_old_agents should be set to 'True' (non default).
 
−
 
 
−
=== Start Services ===
 
−
 
 
−
Start (or restart) the Neutron server
 
−
  #/etc/init.d/neutron-server start
 
  
 
== Compute Nodes ==
 
== Compute Nodes ==
  
−
=== Installation ===
+
To configure the Compute node:
−
 
 
−
1. If you didn't download Mellanox OpenStack debian files download it :
 
−
 
 
−
http://www.mellanox.com/downloads/solutions/openstack/
 
−
 
 
−
2. Install the eswitchd debian files:
 
−
  dpkg -i eswitchd_0.8-1_amd64.deb
 
−
  dpkg -i python-mlnxvif_0.6-1_all.deb
 
−
  apt-get install -f # fix dependencies
 
−
  apt-get install -y neutron-plugin-mlnx-agent
 
−
 
 
−
3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
 
−
 
 
−
=== Configuration ===
 
−
 
 
−
1. Configure /etc/eswitchd/eswitchd.conf if needed
 
−
 
 
−
Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here])
 
−
 
 
−
2. Modify /etc/nova/nova.conf
 
−
 
 
−
  compute_driver=nova.virt.libvirt.driver.LibvirtDriver
 
−
  libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver
 
−
  security_group_api=nova
 
−
  connection_type=libvirt
 
−
 
 
−
In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
 
−
 
 
−
  libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver
 
−
 
 
−
3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.
 
−
  [AGENT]
 
−
  rpc_support_old_agents = True
 
−
  [ESWITCH]
 
−
  physical_interface_mapping -  the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is:    <fabric name>:< PF name> (Only releavant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is  "default:autoeth". 
 
−
  vnic_type - type of VM network interface: 'mlnx_direct' or 'hostdev'  according to libvirt terminology.
 
−
        hostdev: this is the traditional method of assigning any generic PCI device to a guest (SR-IOV).
 
−
        mlnx_direct: this is a method to provide macvtap device on top of the PCI device (default).
 
−
        bridge:  - When using Linux Bridge Plugin on top of eIPoIB device
 
−
  daemon_endpoint - 'tcp://127.0.0.1:60001'
 
−
 
 
−
For a plugin configuration file example (Havana), please refer to Mellanox config *ini file (click [https://github.com/openstack/neutron/blob/stable/havana/etc/neutron/plugins/mlnx/mlnx_conf.ini here])
 
−
 
 
−
4. (Optional) Modify /etc/init/neutron-plugin-mlnx-agent.conf as described in LP bug #[https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1257732 1257732]
 
−
 
 
−
5. Check that ebrctl is listed somewhere in /etc/nova/rootwrap.d/*, otherwise add this to /etc/nova/rootwrap.d/compute.filters file in the [Filters] section.
 
−
  ebrctl: CommandFilter, ebrctl, root
 
−
 
 
−
=== Start Services ===
 
−
 
 
−
1. Restart Nova.
 
−
 
 
−
  #/etc/init.d/nova-compute restart
 
−
 
 
−
2. Start eswitch Daemon
 
−
 
 
−
  #/etc/init.d/eswitchd start
 
  
−
3. Start the Neutron agent  
+
1. Create the file /etc/modprobe.d/mlx4_ib.conf with this line:
 +
options mlx4_ib sm_guid_assign=0
 +
2. Download mlnxvif, eSwitch Daemon (eSwitchd), and mlnx-dnsmasq from this link:
 +
http://www.mellanox.com/downloads/solutions/openstack/icehouse/debian/
 +
3. Install the Mellanox plugin services mlnxvif and eSwitchd:
 +
dpkg –i python-mlnxvif_2014.1.1-1_all.deb
 +
dpkg –i python-eswitchd_0.10-1_all.deb
 +
4. Edit /etc/eswitchd/eswitchd.conf as follows:
 +
[DAEMON]
 +
fabrics=<physical_network>:<ib-interface> //(i.e. default:ib0)
 +
5. Edit /etc/nova/nova.conf as follows:
 +
[DEFAULT]
 +
compute_driver = libvirt.LibvirtDriver
 +
libvirt_vif_driver = mlnxvif.vif.MlxEthVIFDriver
 +
security_group_api = nova
 +
6. Check if “ebrctl” appears in one of Nova’s rootwrap files:
 +
grep 'ebrctl' /etc/nova/rootwrap.d/*
 +
If “ebrctl” does not appear in one of Nova’s rootwrap files, edit the file /etc/nova/rootwrap.d/compute.filter as follows:
 +
[Filters]
 +
ebrctl: CommandFilter, ebrctl, root
 +
7. Install the Neutron Mellanox agent (from Ubuntu repository):
 +
apt-get install neutron-plugin-mlnx-agent
 +
8. Copy /etc/neutron/plugins/mlnx/mlnx_conf.ini from the Neutron Server node to the same path on each Compute node.
  
−
  #/etc/init.d/neutron-plugin-mlnx-agent start
+
9. Restart the following services:
−
 
+
/etc/init.d/nova-compute restart
−
Note: eswitch Daemon should be running before the Neutron Agent is started.
+
/etc/init.d/eswitchd start
 +
/etc/init.d/neutron-plugin-mlnx-agent start
  
 
== Network Node ==
 
== Network Node ==
  
−
1. Install the Neutron Linux bridge plugin, the Neutron DHCP agent and the Neutron L3 agent:
+
To configure the Network node:  
−
 
 
−
  # apt-get install neutron-plugin-linuxbridge-agent neutron-dhcp-agent neutron-l3-agent
 
−
 
 
−
2. Copy /etc/neutron/neutron.conf from the Compute node.
 
−
 
 
−
3. Change the following configuration of the ini file (/etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .
 
−
  [linux_bridge] 
 
−
  physical_interface_mappings = default:eth2
 
−
 
 
−
4. Configure the DHCP agent according to the following guidelines
 
−
 
 
−
Update the following file: /etc/neutron/dhcp_agent.ini with:
 
−
  [DEFAULT]
 
−
  interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver
 
−
 
 
−
For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_dhcp_agent OpenStack documentation]
 
−
 
 
−
5. Configure the L3 agent according to the following guidelines
 
  
−
Update the following file: /etc/neutron/l3_agent.ini with:
+
1. Configure eIPoIB (for Linux bridge plugin) by editing the file /etc/infiniband/openib.conf as follows:
−
  [DEFAULT]  
+
E_IPOIB_LOAD=yes
−
  interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver
+
2. Restart OFED: 
 +
service openibd restart
 +
3. Verify that the new Ethernet interfaces now appear with ibdev2netdev:
 +
# ibdev2netdev
 +
mlx4_0 port 1 ==> ib0 (Up)
 +
mlx4_0 port 2 ==> ib1 (Down)
 +
mlx4_0 port 1 ==> eth2 (Up)
 +
mlx4_0 port 2 ==> eth1 (Down)
 +
In eIPoIB, every ib interface has a matching eth interface. The ib interface used here is ib0, and the matching one is eth2 (same card and port names: mlx4_0 port 1). By using Ethtool, you can verify that ib0 is working on ib_ipoib driver, and eth2 is working on eth_ipoib driver, as follows:
 +
# ethtool -i ib0
 +
driver: ib_ipoib
 +
version: 2.3-1.5.4_Sep 30 2014
 +
firmware-version: 2.32.5100
 +
bus-info: 0000:07:00.0# ethtool -i eth2
 +
driver: eth_ipoib
 +
version: 2.3-1.5.4 (Sep 30 2014)
 +
firmware-version: 1
 +
bus-info: ib0
 +
4. Edit /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows:
 +
[linux_bridge]  
 +
physical_interface_mappings = <physical_network>:<eth_ipoib-interface>
 +
//(i.e. default:eth2)
 +
5. Restart the Linux bridge agent:
 +
service neutron-linuxbridge-agent restart
 +
6. To allow DHCP support, download mlnx-dnsmadq from this link: http://www.mellanox.com/downloads/solutions/openstack/icehouse/debian/
  
−
For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_l3_agent OpenStack documentation]
+
7. Install the package:
 +
dpkg –i python-mlnx-dnsmasq_2014.1.1-1_all.deb
 +
8. Configure the file /etc/neutron/dhcp_agent.ini:
 +
[DEFAULT]
 +
dhcp_driver = mlnx_dhcp.MlnxDnsmasq
 +
9. Make sure that the Dnsmasq which is installed as a part of the OpenStack (not mlnx-dnsmasq) environment is of version 2.65.
  
−
6. Configure the Metadata agent according to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#nova_config_metadata OpenStack documentation]
+
10. Copy /etc/neutron/neutron.conf from the Compute node to the same path on the Network node.
  
−
7. Start the Neutron Linux Bridge, DHCP and L3 agents
+
11. Edit /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows:
−
  # restart neutron-plugin-linuxbridge-agent
+
[linux_bridge] 
−
  # restart neutron-dhcp-agent
+
physical_interface_mappings = <physical_network>:<eth_ipoib interface>
−
  # restart neutron-l3-agent
+
(i.e. default:eth2)
−
  # restart neutron-metadata-agent
+
12. Edit /etc/neutron/dhcp_agent.ini and /etc/neutron/l3_agent.ini as follows:
 +
[DEFAULT]
 +
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver
 +
13. Restart the following services:
 +
service neutron-plugin-linuxbridge-agent restart
 +
service neutron-dhcp-agent restart
 +
service neutron-l3-agent restart
  
 
== Troubleshooting ==
 
== Troubleshooting ==
  
 
* If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.
 
* If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.

Latest revision as of 09:52, 1 January 2015

Overview

This wiki page describes how to tune Mellanox OpenStack on Ubuntu 12.04 servers

Prerequisites

InfiniBand Network

OpenSM Node

To configure OpenSM node:

1. Edit the file /etc/opensm/partitions.conf by adding the following line: 

management=0xffff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;

Create the file if it does not exist.

For each VLAN value in the VLAN range used for VLAN network segmentation, add the following line in the file /etc/opensm/partitions.conf (for vlan-pkey association): 

vlan<decimal-value>=0x<hexadecimal-value>, ipoib, sl=0, defmember=full: ALL;

In this manual, the VLAN range is 2-10 (mlnx_agent.conf /ml2_conf.ini). Therefore, add the following lines: 

vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

2. Modify the following line in the file /etc/opensm/opensm.conf from FALSE to TRUE: 

allow_both_pkeys TRUE

3. Restart OpenSM: 

service opensmd restart

4. Verify that all ports on all nodes have their ib interface (i.e. ib0) active, run ibdev2netdev on all hosts, and make sure that the port is up: 

mlx4_0 port 1 ==> ib0 (Up)

If the port is down, try bringing it up by running “ifconfig ib0 up”, and make sure your switch ports are of type “InfiniBand”, and that all nodes can reach the OpenSM node by running “sminfo”.

Controller Node

To configure the Controller node:

1. Ensure that the file /etc/nova/nova.conf includes the following line: 

security_group_api=nova

2. If the security group value is changed, restart the Nova services: 

# for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done

Neutron Server Node

NOTE: The Neutron Server can sometimes be part of the Controller node.

To configure the Neutron Server node:

1. Edit /etc/neutron/neutron.conf and make sure Neutron is using ML2 plugin: 

[DEFAULT]
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin

2. Edit the ML2 configuration file /etc/neutron/plugins/ml2/ml2_conf.ini as follows: 

[ml2]
type_drivers = vlan,flat
tenant_network_types = vlan 
mechanism_drivers = linuxbridge,mlnx
 
[ml2_type_vlan]
network_vlan_ranges=<physical_network>:<min_vlan>:<max_vlan> //(i.e. default:2:10)

[securitygroup]
enable_security_group = True

[eswitch]
vnic_type = hostdev
apply_profile_patch = True

3. Install the required Deb for the Mellanox Neutron plugin (from Ubuntu’s repository): 

apt-get install neutron-plugin-mlnx-agent

4. Edit /etc/neutron/plugins/mlnx/mlnx_conf.ini to match your environment: 

[mlnx]
network_vlan_ranges = <physical_network>:<min_vlan:<max_vlan> //(i.e. default:2:10)
tenant_network_type = vlan

[eswitch]
physical_interface_mappings = <physical_network>:<ib interface>  //(i.e. default:ib0)
vnic_type = hostdev

[agent]
support_old_agents = True

5. Restart Neutron Server: 

/etc/init.d/neutron-server start

Compute Nodes

To configure the Compute node:

1. Create the file /etc/modprobe.d/mlx4_ib.conf with this line: 

options mlx4_ib sm_guid_assign=0

2. Download mlnxvif, eSwitch Daemon (eSwitchd), and mlnx-dnsmasq from this link: 

http://www.mellanox.com/downloads/solutions/openstack/icehouse/debian/

3. Install the Mellanox plugin services mlnxvif and eSwitchd: 

dpkg –i python-mlnxvif_2014.1.1-1_all.deb
dpkg –i python-eswitchd_0.10-1_all.deb

4. Edit /etc/eswitchd/eswitchd.conf as follows: 

[DAEMON]
fabrics=<physical_network>:<ib-interface> //(i.e. default:ib0)

5. Edit /etc/nova/nova.conf as follows: 

[DEFAULT]
compute_driver = libvirt.LibvirtDriver
libvirt_vif_driver = mlnxvif.vif.MlxEthVIFDriver
security_group_api = nova

6. Check if “ebrctl” appears in one of Nova’s rootwrap files: 

grep 'ebrctl' /etc/nova/rootwrap.d/*
If “ebrctl” does not appear in one of Nova’s rootwrap files, edit the file /etc/nova/rootwrap.d/compute.filter as follows:
[Filters]
ebrctl: CommandFilter, ebrctl, root

7. Install the Neutron Mellanox agent (from Ubuntu repository): 

apt-get install neutron-plugin-mlnx-agent

8. Copy /etc/neutron/plugins/mlnx/mlnx_conf.ini from the Neutron Server node to the same path on each Compute node.

9. Restart the following services: 

/etc/init.d/nova-compute restart
/etc/init.d/eswitchd start
/etc/init.d/neutron-plugin-mlnx-agent start

Network Node

To configure the Network node:

1. Configure eIPoIB (for Linux bridge plugin) by editing the file /etc/infiniband/openib.conf as follows: 

E_IPOIB_LOAD=yes

2. Restart OFED: 

service openibd restart

3. Verify that the new Ethernet interfaces now appear with ibdev2netdev: 

# ibdev2netdev
mlx4_0 port 1 ==> ib0 (Up)
mlx4_0 port 2 ==> ib1 (Down)
mlx4_0 port 1 ==> eth2 (Up)
mlx4_0 port 2 ==> eth1 (Down)

In eIPoIB, every ib interface has a matching eth interface. The ib interface used here is ib0, and the matching one is eth2 (same card and port names: mlx4_0 port 1). By using Ethtool, you can verify that ib0 is working on ib_ipoib driver, and eth2 is working on eth_ipoib driver, as follows: 

# ethtool -i ib0
driver: ib_ipoib
version: 2.3-1.5.4_Sep 30 2014
firmware-version: 2.32.5100
bus-info: 0000:07:00.0# ethtool -i eth2
driver: eth_ipoib
version: 2.3-1.5.4 (Sep 30 2014)
firmware-version: 1
bus-info: ib0

4. Edit /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows: 

[linux_bridge] 
physical_interface_mappings = <physical_network>:<eth_ipoib-interface> 
//(i.e. default:eth2)

5. Restart the Linux bridge agent: 

service neutron-linuxbridge-agent restart

6. To allow DHCP support, download mlnx-dnsmadq from this link: http://www.mellanox.com/downloads/solutions/openstack/icehouse/debian/

7. Install the package: 

dpkg –i python-mlnx-dnsmasq_2014.1.1-1_all.deb

8. Configure the file /etc/neutron/dhcp_agent.ini: 

[DEFAULT]
dhcp_driver = mlnx_dhcp.MlnxDnsmasq

9. Make sure that the Dnsmasq which is installed as a part of the OpenStack (not mlnx-dnsmasq) environment is of version 2.65.

10. Copy /etc/neutron/neutron.conf from the Compute node to the same path on the Network node.

11. Edit /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows: 

[linux_bridge]  
physical_interface_mappings = <physical_network>:<eth_ipoib interface> 
(i.e. default:eth2)

12. Edit /etc/neutron/dhcp_agent.ini and /etc/neutron/l3_agent.ini as follows: 

[DEFAULT] 
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

13. Restart the following services: 

service neutron-plugin-linuxbridge-agent restart
service neutron-dhcp-agent restart
service neutron-l3-agent restart

Troubleshooting

  • If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.
Retrieved from "https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand&oldid=71060"