Mellanox-Neutron-Icehouse-Ubuntu-Ethernet
Contents
Overview
This wiki page describes how to tune Mellanox OpenStack on Ubuntu 12.04 servers
Prerequisites
- A running OpenStack environment installed with the ML2 plugin on top of OVS.
- All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119)
- Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers
- SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community click here.
- The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes
- VLANs configured on the ports in the switch.
Ethernet Network
Controller Node
In /etc/nova/nova.conf ensure you have the following line:
security_group_api=nova
If the value is changed, restart the Nova services:
# for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done
Neutron Server Node
NOTE: The Neutron Server can sometimes be part of the Controller node.
To configure Neutron-Server node:
1. Edit /etc/neutron/neutron.conf, and make sure Neutron is using ML2 plugin:
[DEFAULT] core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin
2. Modify the ML2 configuration file (/etc/neutron/plugins/ml2/ml2_conf.ini) as follows:
[ml2] type_drivers = vlan,flat tenant_network_types = vlan mechanism_drivers = openvswitch,mlnx
[ml2_type_vlan] network_vlan_ranges=<physical_network>:<min_vlan>:<max_vlan> //(i.e. default:2:10)
[securitygroup] enable_security_group = True
[eswitch] vnic_type = hostdev apply_profile_patch = True
3. Install the required Deb for the Mellanox Neutron plugin (from Ubuntu’s repository):
apt-get install neutron-plugin-mlnx-agent
4. Edit /etc/neutron/plugins/mlnx/mlnx_conf.ini to match your environment:
[mlnx] network_vlan_ranges = <physical_network>:<min_vlan:<max_vlan> //(i.e. default:2:10) tenant_network_type = vlan
[eswitch] physical_interface_mappings = <physical_network>:<eth interface> //(i.e. default:eth0) vnic_type = hostdev
[agent] support_old_agents = True
5. Restart neutron-server:
/etc/init.d/neutron-server start
Compute Nodes
Installation
1. If you didn't download Mellanox OpenStack debian files download it :
http://www.mellanox.com/downloads/solutions/openstack/
2. Install the eswitchd debian files:
dpkg -i eswitchd_0.8-1_amd64.deb dpkg -i python-mlnxvif_0.6-1_all.deb apt-get install -f # fix dependencies apt-get install -y neutron-plugin-mlnx-agent
3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
Configuration
1. Configure /etc/eswitchd/eswitchd.conf if needed
Please Refer to Mellanox Community for the eSwitchd installation notes ( click here)
2. Modify /etc/nova/nova.conf
compute_driver=nova.virt.libvirt.driver.LibvirtDriver libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver security_group_api=nova connection_type=libvirt
In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver
3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.
[AGENT] rpc_support_old_agents = True [ESWITCH] physical_interface_mapping - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only releavant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth". vnic_type - type of VM network interface: 'mlnx_direct' or 'hostdev' according to libvirt terminology. hostdev: this is the traditional method of assigning any generic PCI device to a guest (SR-IOV). mlnx_direct: this is a method to provide macvtap device on top of the PCI device (default). bridge: - When using Linux Bridge Plugin on top of eIPoIB device daemon_endpoint - 'tcp://127.0.0.1:60001'
For a plugin configuration file example (Havana), please refer to Mellanox config *ini file (click here)
4. (Optional) Modify /etc/init/neutron-plugin-mlnx-agent.conf as described in LP bug #1257732
5. Check that ebrctl is listed somewhere in /etc/nova/rootwrap.d/*, otherwise add this to /etc/nova/rootwrap.d/compute.filters file in the [Filters] section.
ebrctl: CommandFilter, ebrctl, root
Start Services
1. Restart Nova.
#/etc/init.d/nova-compute restart
2. Start eswitch Daemon
#/etc/init.d/eswitchd start
3. Start the Neutron agent
#/etc/init.d/neutron-plugin-mlnx-agent start
Note: eswitch Daemon should be running before the Neutron Agent is started.
Network Node
1. Install the Neutron Linux bridge plugin, the Neutron DHCP agent and the Neutron L3 agent:
# apt-get install neutron-plugin-linuxbridge-agent neutron-dhcp-agent neutron-l3-agent
2. Copy /etc/neutron/neutron.conf from the Compute node.
3. Change the following configuration of the ini file (/etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .
[linux_bridge] physical_interface_mappings = default:eth2
4. Configure the DHCP agent according to the following guidelines
Update the following file: /etc/neutron/dhcp_agent.ini with:
[DEFAULT] interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver
For additional information refer to the OpenStack documentation
5. Configure the L3 agent according to the following guidelines
Update the following file: /etc/neutron/l3_agent.ini with:
[DEFAULT] interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver
For additional information refer to the OpenStack documentation
6. Configure the Metadata agent according to the OpenStack documentation
7. Start the Neutron Linux Bridge, DHCP and L3 agents
# restart neutron-plugin-linuxbridge-agent # restart neutron-dhcp-agent # restart neutron-l3-agent # restart neutron-metadata-agent
Troubleshooting
- If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.