Difference between revisions of "Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand"
(→Controller Node) |
(→Neutron Server Node) |
||
Line 54: | Line 54: | ||
== Neutron Server Node == | == Neutron Server Node == | ||
− | |||
− | + | NOTE: The Neutron Server can sometimes be part of the Controller node. | |
− | + | To configure the Neutron Server node: | |
− | + | 1. Edit /etc/neutron/neutron.conf and make sure Neutron is using ML2 plugin: | |
− | + | [DEFAULT] | |
− | + | core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin | |
− | + | 2. Edit the ML2 configuration file /etc/neutron/plugins/ml2/ml2_conf.ini as follows: | |
− | + | [ml2] | |
− | + | type_drivers = vlan,flat | |
− | + | tenant_network_types = vlan | |
− | + | mechanism_drivers = linuxbridge,mlnx | |
− | + | ||
− | + | [ml2_type_vlan] | |
− | + | network_vlan_ranges=<physical_network>:<min_vlan>:<max_vlan> //(i.e. default:2:10) | |
− | + | ||
− | + | [securitygroup] | |
− | + | enable_security_group = True | |
− | + | ||
− | + | [eswitch] | |
− | + | vnic_type = hostdev | |
− | + | apply_profile_patch = True | |
− | + | 3. Install the required Deb for the Mellanox Neutron plugin (from Ubuntu’s repository): | |
− | + | apt-get install neutron-plugin-mlnx-agent | |
− | + | 4. Edit /etc/neutron/plugins/mlnx/mlnx_conf.ini to match your environment: | |
− | + | [mlnx] | |
− | + | network_vlan_ranges = <physical_network>:<min_vlan:<max_vlan> //(i.e. default:2:10) | |
− | + | tenant_network_type = vlan | |
− | + | ||
− | + | [eswitch] | |
− | + | physical_interface_mappings = <physical_network>:<ib interface> //(i.e. default:ib0) | |
− | + | vnic_type = hostdev | |
− | + | ||
− | = | + | [agent] |
− | + | support_old_agents = True | |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | + | 5. Restart Neutron Server: | |
− | + | /etc/init.d/neutron-server start | |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
== Compute Nodes == | == Compute Nodes == |
Revision as of 16:48, 26 November 2014
Contents
Overview
This wiki page describes how to tune Mellanox OpenStack on Ubuntu 12.04 servers
Prerequisites
- A running OpenStack environment installed with the ML2 plugin on top of OVS.
- All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119)
- Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers
- SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community click here.
- The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes
- VLANs configured on the ports in the switch.
InfiniBand Network
OpenSM Node
To configure OpenSM node:
1. Edit the file /etc/opensm/partitions.conf by adding the following line:
management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;
Create the file if it does not exist.
For each VLAN value in the VLAN range used for VLAN network segmentation, add the following line in the file /etc/opensm/partitions.conf (for vlan-pkey association):
vlan<decimal-value>=0x<hexadecimal-value>, ipoib, sl=0, defmember=full: ALL;
In this manual, the VLAN range is 2-10 (mlnx_agent.conf /ml2_conf.ini). Therefore, add the following lines:
vlan1=0x1, ipoib, sl=0, defmember=full : ALL; vlan2=0x2, ipoib, sl=0, defmember=full : ALL; vlan3=0x3, ipoib, sl=0, defmember=full : ALL; vlan4=0x4, ipoib, sl=0, defmember=full : ALL; vlan5=0x5, ipoib, sl=0, defmember=full : ALL; vlan6=0x6, ipoib, sl=0, defmember=full : ALL; vlan7=0x7, ipoib, sl=0, defmember=full : ALL; vlan8=0x8, ipoib, sl=0, defmember=full : ALL; vlan9=0x9, ipoib, sl=0, defmember=full : ALL; vlan10=0xa, ipoib, sl=0, defmember=full : ALL;
2. Modify the following line in the file /etc/opensm/opensm.conf from FALSE to TRUE:
allow_both_pkeys TRUE
3. Restart OpenSM:
service opensmd restart
4. Verify that all ports on all nodes have their ib interface (i.e. ib0) active, run ibdev2netdev on all hosts, and make sure that the port is up:
mlx4_0 port 1 ==> ib0 (Up)
If the port is down, try bringing it up by running “ifconfig ib0 up”, and make sure your switch ports are of type “InfiniBand”, and that all nodes can reach the OpenSM node by running “sminfo”.
Controller Node
To configure the Controller node:
1. Ensure that the file /etc/nova/nova.conf includes the following line:
security_group_api=nova
2. If the security group value is changed, restart the Nova services:
# for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done
Neutron Server Node
NOTE: The Neutron Server can sometimes be part of the Controller node.
To configure the Neutron Server node:
1. Edit /etc/neutron/neutron.conf and make sure Neutron is using ML2 plugin:
[DEFAULT] core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin
2. Edit the ML2 configuration file /etc/neutron/plugins/ml2/ml2_conf.ini as follows:
[ml2] type_drivers = vlan,flat tenant_network_types = vlan mechanism_drivers = linuxbridge,mlnx [ml2_type_vlan] network_vlan_ranges=<physical_network>:<min_vlan>:<max_vlan> //(i.e. default:2:10) [securitygroup] enable_security_group = True [eswitch] vnic_type = hostdev apply_profile_patch = True
3. Install the required Deb for the Mellanox Neutron plugin (from Ubuntu’s repository):
apt-get install neutron-plugin-mlnx-agent
4. Edit /etc/neutron/plugins/mlnx/mlnx_conf.ini to match your environment:
[mlnx] network_vlan_ranges = <physical_network>:<min_vlan:<max_vlan> //(i.e. default:2:10) tenant_network_type = vlan [eswitch] physical_interface_mappings = <physical_network>:<ib interface> //(i.e. default:ib0) vnic_type = hostdev [agent] support_old_agents = True
5. Restart Neutron Server:
/etc/init.d/neutron-server start
Compute Nodes
Installation
1. If you didn't download Mellanox OpenStack debian files download it :
http://www.mellanox.com/downloads/solutions/openstack/
2. Install the eswitchd debian files:
dpkg -i eswitchd_0.8-1_amd64.deb dpkg -i python-mlnxvif_0.6-1_all.deb apt-get install -f # fix dependencies apt-get install -y neutron-plugin-mlnx-agent
3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)
Configuration
1. Configure /etc/eswitchd/eswitchd.conf if needed
Please Refer to Mellanox Community for the eSwitchd installation notes ( click here)
2. Modify /etc/nova/nova.conf
compute_driver=nova.virt.libvirt.driver.LibvirtDriver libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver security_group_api=nova connection_type=libvirt
In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following:
libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver
3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment.
[AGENT] rpc_support_old_agents = True [ESWITCH] physical_interface_mapping - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: <fabric name>:< PF name> (Only releavant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is "default:autoeth". vnic_type - type of VM network interface: 'mlnx_direct' or 'hostdev' according to libvirt terminology. hostdev: this is the traditional method of assigning any generic PCI device to a guest (SR-IOV). mlnx_direct: this is a method to provide macvtap device on top of the PCI device (default). bridge: - When using Linux Bridge Plugin on top of eIPoIB device daemon_endpoint - 'tcp://127.0.0.1:60001'
For a plugin configuration file example (Havana), please refer to Mellanox config *ini file (click here)
4. (Optional) Modify /etc/init/neutron-plugin-mlnx-agent.conf as described in LP bug #1257732
5. Check that ebrctl is listed somewhere in /etc/nova/rootwrap.d/*, otherwise add this to /etc/nova/rootwrap.d/compute.filters file in the [Filters] section.
ebrctl: CommandFilter, ebrctl, root
Start Services
1. Restart Nova.
#/etc/init.d/nova-compute restart
2. Start eswitch Daemon
#/etc/init.d/eswitchd start
3. Start the Neutron agent
#/etc/init.d/neutron-plugin-mlnx-agent start
Note: eswitch Daemon should be running before the Neutron Agent is started.
Network Node
1. Install the Neutron Linux bridge plugin, the Neutron DHCP agent and the Neutron L3 agent:
# apt-get install neutron-plugin-linuxbridge-agent neutron-dhcp-agent neutron-l3-agent
2. Copy /etc/neutron/neutron.conf from the Compute node.
3. Change the following configuration of the ini file (/etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini). The "default" in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini .
[linux_bridge] physical_interface_mappings = default:eth2
4. Configure the DHCP agent according to the following guidelines
Update the following file: /etc/neutron/dhcp_agent.ini with:
[DEFAULT] interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver
For additional information refer to the OpenStack documentation
5. Configure the L3 agent according to the following guidelines
Update the following file: /etc/neutron/l3_agent.ini with:
[DEFAULT] interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver
For additional information refer to the OpenStack documentation
6. Configure the Metadata agent according to the OpenStack documentation
7. Start the Neutron Linux Bridge, DHCP and L3 agents
# restart neutron-plugin-linuxbridge-agent # restart neutron-dhcp-agent # restart neutron-l3-agent # restart neutron-metadata-agent
Troubleshooting
- If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.