|
|
(5 intermediate revisions by the same user not shown) |
Line 1: |
Line 1: |
− | =Overview=
| |
| | | |
− | ==Mellanox Neutron ML2 Driver==
| |
− |
| |
− | Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API.
| |
− | This driver supports Mellanox embedded switch functionality as part of the InfiniBand HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin.
| |
− |
| |
− | Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) vnic type to allow VM network connectivity.
| |
− | For vnic type configuration API details, please refer to configuration reference guide.
| |
− | Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access).
| |
− |
| |
− | The driver supports VLAN network type to facilitate virtual networks either on InfiniBand fabrics.
| |
− |
| |
− | ===Prerequisites===
| |
− | * CentOS 7.6 / Ubuntu 18.04 or later
| |
− | * Mellanox ConnectX® Family device:
| |
− | [https://www.mellanox.com/page/products_dyn?product_family=119 ConnectX®-3/ConnectX®-3 PRO]
| |
− | [https://www.mellanox.com/page/products_dyn?product_family=201& ConnectX®-4]/[https://www.mellanox.com/page/products_dyn?product_family=214&mtag=connectx_4_lx_en_ic ConnectX®-4Lx]
| |
− | [https://www.mellanox.com/page/products_dyn?product_family=258&mtag=connectx_5_vpi_card ConnectX®-5]
| |
− | [https://www.mellanox.com/page/products_dyn?product_family=265&mtag=connectx_6_vpi_card ConnectX®-6]
| |
− | * Driver: [http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers Mellanox OFED] 4.6-1.0.1.1 or greater
| |
− | * A running OpenStack environment installed (RDO Manager or Packstack).
| |
− | * [https://community.mellanox.com/docs/DOC-1317 SR-IOV] enabled on all compute nodes.
| |
− | * The software package [http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 iproute2] installed on all Compute nodes
| |
− | * [http://www.mellanox.com/page/products_dyn?product_family=100 Mellanox UFM] greater than 5.9.5
| |
− |
| |
− | =InfiniBand Network=
| |
− | An Infiniband network relies on a software entity to manage the network.
| |
− | This entity is referred to as a Subnet Manager or SM.
| |
− | Mellanox Neutron Plugin uses InfiniBand Partitions (PKeys) to separate Networks.
| |
− |
| |
− | ==SM Node==
| |
− |
| |
− | === OpenSM Provisioning with SDN Mechanism Driver ===
| |
− |
| |
− | [https://community.mellanox.com/docs/DOC-2155 SDN Mechanism Driver] allows OpenSM dynamically assign PKs in the IB network.
| |
− |
| |
− | More details about applying SDN Mechanism Driver with [http://www.mellanox.com/page/products_dyn?product_family=220&mtag=mellanox_neo NEO] can be found [https://community.mellanox.com/docs/DOC-2251 here]
| |
− |
| |
− | === Manual OpenSM Configuration ===
| |
− |
| |
− | ====For ConnectX®-3/ConnectX®-3Pro use the following configuration====
| |
− |
| |
− | Change the following in /etc/opensm/opensm.conf:
| |
− | allow_both_pkeys TRUE
| |
− |
| |
− | ====For ConnectX®-4 or newer use the following configuration====
| |
− |
| |
− | Change the following in /etc/opensm/opensm.conf:
| |
− | virt_enabled 2
| |
− |
| |
− | ==== Partition membership configuration ====
| |
− | In an Infiniband network, network segmentation is achieved through partitions (roughly equivalent to ETH VLAN). Each partition has its own key (15bit partition key).
| |
− | This file contains configuration of partition membership for all network endpoints in the system.
| |
− |
| |
− | The following guidelines need to be followed to allow connectivity
| |
− |
| |
− | For each PKEY its required to:
| |
− | # Have OpenSM PF(physical function IB device) GUID be a member of the partition.
| |
− | # Have L3 and DHCP PF GUID be a member of the partition.
| |
− | # Have VF GUID be member of the partition (of the relevant VMs).
| |
− |
| |
− |
| |
− | There is a 1:1 mapping between VLAN assigned to a network in openstack and the translation of that VLAN to PKEY (e.g VLAN 10 translates to PKEY 10)
| |
− |
| |
− | ===== Example1: =====
| |
− |
| |
− | The following configurations corresponds to a single vlan network (vlan 3) with two VMs and a DHCP agent
| |
− |
| |
− | management=0x7fff,ipoib, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
| |
− | vlan3_vm1=0x3, indx0, ipoib, defmember=full : 0xfa163e0000c0851b;
| |
− | vlan3_vm2=0x3, indx0, ipoib, defmember=full : 0xfa163e0000df0519;
| |
− | vlan3_l3_dhcp_services=0x3, ipoib, defmember=full : 0xe41d2d030061f5fa;
| |
− | vlan3_sm=0x3, ipoib, defmember=full : SELF;
| |
− |
| |
− | Going over it line by line:
| |
− |
| |
− | # make all member of the default (management) pkey
| |
− | # VM1 port guid (VF) member of network with pkey 3
| |
− | # VM2 port guid (VF) member of network with pkey 3
| |
− | # L3/DHCP agent PF guid member of network with pkey3
| |
− | # Subnet manager member of network with pkey3
| |
− |
| |
− | ===== Example 2: =====
| |
− |
| |
− | The following configurations corresponds to a two vlan networks (vlan 3, vlan 5) with one VM on each network and a DHCP and L3 agent
| |
− |
| |
− | management=0x7fff,ipoib, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
| |
− | vlan3_vm1=0x3, indx0, ipoib, defmember=full : 0xfa163e0000c0851b;
| |
− | vlan5_vm2=0x5, indx0, ipoib, defmember=full : 0xfa163e0000df0519;
| |
− | vlan3_l3_dhcp_services=0x3, ipoib, defmember=full : 0xe41d2d030061f5fa;
| |
− | vlan5_l3_dhcp_services=0x5, ipoib, defmember=full : 0xe41d2d030061f5fa;
| |
− | vlan3_sm=0x3, ipoib, defmember=full : SELF;
| |
− | vlan5_sm=0x5, ipoib, defmember=full : SELF;
| |
− |
| |
− | Going over it line by line:
| |
− |
| |
− | 1. make all member of the default (management) pkey
| |
− | 2. VM1 port guid (VF) member of network with pkey 3
| |
− | 3. VM2 port guid (VF) member of network with pkey 5
| |
− | 4. L3/DHCP agent PF guid member of network with pkey3
| |
− | 5. L3/DHCP agent PF guid member of network with pkey5
| |
− | 6. Subnet manager member of network with pkey3
| |
− | 7. Subnet manager member of network with pkey5
| |
− |
| |
− | 4. Restart the OpenSM:
| |
− | # systemctl restart opensmd.service
| |
− |
| |
− | ==Controller Node==
| |
− |
| |
− | To configure the Controller node:
| |
− |
| |
− | 1. Install Mellanox RPMs:
| |
− | # yum install -y --nogpgcheck python-networking-mlnx
| |
− |
| |
− | ===Neutron Server===
| |
− | 1. Make sure ML2 is the current Neutron plugin by checking the core_plugin parameter in '''/etc/neutron/neutron.conf''':
| |
− | core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin
| |
− |
| |
− | 2. Make sure '''/etc/neutron/plugin.ini''' is pointing to '''/etc/neutron/plugins/ml2/ml2_conf.ini''' (symbolic link)
| |
− |
| |
− | 3. Modify '''/etc/neutron/plugins/ml2/ml2_conf.ini''' by adding the following:
| |
− | [ml2]
| |
− | type_drivers = vlan,flat
| |
− | tenant_network_types = vlan
| |
− | mechanism_drivers = mlnx_infiniband, openvswitch
| |
− | # or mechanism_drivers = mlnx_infiniband, linuxbridge
| |
− | [ml2_type_vlan]
| |
− | network_vlan_ranges = default:1:10
| |
− |
| |
− | 4. Start (or restart) the Neutron server:
| |
− | # systemctl restart neutron-server.service
| |
− |
| |
− | ===Nova Scheduler===
| |
− | Enabling PciPassthroughFilter modify /etc/nova/nova.conf
| |
− | scheduler_available_filters = nova.scheduler.filters.all_filters
| |
− | scheduler_default_filters = RetryFilter, AvailabilityZoneFilter, RamFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, PciPassthroughFilter
| |
− |
| |
− | == Network Node==
| |
− | To configure the Network node:
| |
− |
| |
− | Prerequisites:
| |
− | E_IPoIB port is configured and up
| |
− |
| |
− | 1. Make sure that eIPoIB module is up and configured in '''/etc/infiniband/openib.conf''': For more information, please refer to eIPoIB configuration in Mellanox OFED User Manual.
| |
− | E_IPOIB_LOAD=yes
| |
− |
| |
− |
| |
− | 2. Restart openibd:
| |
− | # service openibd restart
| |
− |
| |
− | 3. Modify the network bridge configuration according to the use of OpenVswitch or LinuxBridge
| |
− | * 3.1 OpenVswitch '''/etc/neutron/plugins/ml2/openvswitch_agent.ini'''
| |
− | [ovs]
| |
− | bridge_mappings = default:br-<eIPoIB interface>
| |
− | * 3.2 LinuxBridge file '''/etc/neutron/plugins/ml2/linuxbridge_agent.ini''':
| |
− | [linux_bridge]
| |
− | physical_interface_mappings = default:<eIPoIB interface>
| |
− |
| |
− |
| |
− | NOTE: In order to obtain the eIPoIB interface name, run the ethtool tool command (see below) and check that driver name is eth_ipoib:
| |
− | # ethtool -i <eIPoIB_interface>
| |
− | driver: eth_ipoib
| |
− | .....
| |
− |
| |
− |
| |
− | 4. Restart network bridge:
| |
− | * 4.1 OpenVswitch
| |
− | systemctl restart neutron-openvswitch-agent.service
| |
− | * 4.2 LinuxBridge (if using linux bridge)
| |
− | # service neutron-linuxbridge-agent restart
| |
− |
| |
− |
| |
− | NOTE: For DHCP support, the Network node should use the Mellanox Dnsmasq driver as the DHCP driver.
| |
− |
| |
− | ===DHCP Server (Usually part of the Network node)===
| |
− |
| |
− | 1. Modify '''/etc/neutron/dhcp_agent.ini''' as follows and according to OVS or Linuxbridge:
| |
− | dhcp_driver = networking_mlnx.dhcp.mlnx_dhcp.MlnxDnsmasq
| |
− | dhcp_broadcast_reply = True
| |
− |
| |
− | 1.1 For OVS
| |
− | interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
| |
− | 1.2 For Linux Bridge
| |
− | interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver
| |
− |
| |
− |
| |
− | 2. Restart DHCP server:
| |
− | # systemctl restart neutron-dhcp-agent.service
| |
− |
| |
− | ==Compute Nodes==
| |
− |
| |
− | To configure the Compute Node:
| |
− |
| |
− | 1. Install Mellanox RPMs:
| |
− | # yum install --nogpgcheck -y python-networking-mlnx
| |
− |
| |
− | 2. Create the file '''/etc/modprobe.d/mlx4_ib.conf''' and add the following:
| |
− | options mlx4_ib sm_guid_assign=0
| |
− |
| |
− | 3. Restart the driver:
| |
− | # /etc/init.d/openibd restart
| |
− |
| |
− | ===Nova Compute===
| |
− | Nova-compute needs to know which PCI devices are allowed to be passed through to the VMs.
| |
− | Also for SRIOV PCI devices it needs to know to which physical network the VF belongs.
| |
− | This is done through the pci_passthrough_whitelist parameter under the default section in '''/etc/nova/nova.conf'''.
| |
− | For example if we want to whitelist and tag the VFs by their PCI address we would use the following setting:
| |
− | [pci]
| |
− | passthrough_whitelist = {"address":"*:0a:00.*","physical_network":"default"}
| |
− | This associates any VF with address that includes ':0a:00.' in its address to the physical network default.
| |
− |
| |
− | 1. add pci passthrough_whitelist to '''/etc/nova/nova.conf'''
| |
− |
| |
− | 2. Restart Nova:
| |
− | # systemctl restart openstack-nova-compute
| |
− |
| |
− | ===Neutron MLNX Agent===
| |
− | 1. Run:
| |
− | # systemctl enable neutron-mlnx-agent.service
| |
− | # systemctl start neutron-mlnx-agent.service
| |
− |
| |
− | 2. Run:
| |
− | # systemctl daemon-reload
| |
− |
| |
− | 3. In the file '''/etc/neutron/plugins/mlnx/mlnx_conf.ini''', the parameters tenant_network_type , and network_vlan_ranges should be configured as the controllers:
| |
− | physical_interface_mappings = default:<ib_interface>(for example default:ib0)
| |
− |
| |
− | 4. Modify the file '''/etc/neutron/plugins/ml2/eswitchd.conf''' as follows:
| |
− | fabrics = default:<ib_interface> (for example default:ib0)
| |
− |
| |
− | 5. Start eSwitchd:
| |
− | # systemctl enable eswitchd.service
| |
− | # systemctl start eswitchd.service
| |
− |
| |
− | 6. Start the Neutron agent:
| |
− | # systemctl restart neutron-mlnx-agent
| |
− |
| |
− | ==Known issues and Troubleshooting==
| |
− | For known issues and troubleshooting options refer to [https://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]
| |
− |
| |
− | Issue: Missing zmq package on all nodes (Controller/Compute)
| |
− | Solution:
| |
− | <pre>
| |
− | # wget https://bootstrap.pypa.io/get-pip.py
| |
− | # sudo python get-pip.py
| |
− | # sudo pip install pyzmq
| |
− | </pre>
| |