https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Rocky-Redhat-InfiniBand&feed=atom&action=history
Mellanox-Neutron-Rocky-Redhat-InfiniBand - Revision history
2024-03-28T16:22:46Z
Revision history for this page on the wiki
MediaWiki 1.28.2
https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Rocky-Redhat-InfiniBand&diff=166892&oldid=prev
Hasanq: Created page with "=Overview= ==Mellanox Neutron ML2 Driver== Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API. This driver supports Mellanox embedded switch fun..."
2018-11-25T13:33:31Z
<p>Created page with "=Overview= ==Mellanox Neutron ML2 Driver== Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API. This driver supports Mellanox embedded switch fun..."</p>
<p><b>New page</b></p><div>=Overview=<br />
<br />
==Mellanox Neutron ML2 Driver==<br />
<br />
<br />
Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API. <br />
<br />
This driver supports Mellanox embedded switch functionality as part of the InfiniBand HCA. Mellanox ML2 Mechanism Driver provides functional parity with Mellanox Neutron plugin. <br />
<br />
Mellanox ML2 Mechanism Driver supports DIRECT (pci passthrough) vnic type. For vnic type configuration API details, please refer to configuration reference guide.<br />
Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as RDMA (remote direct memory access). <br />
<br />
The driver supports VLAN network type to facilitate virtual networks either on InfiniBand fabrics. <br />
* Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node. <br />
* Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port. <br />
<br />
===Prerequisites===<br />
* Red Hat Installation > 7.1<br />
* Mellanox [http://www.mellanox.com/page/products_dyn?product_family=119 ConnectX®-3/Mellanox ConnectX®-3 PRO] with [http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers Mellanox OFED] the latest 3.X <br />
* Mellanox [http://www.mellanox.com/page/products_dyn?product_family=201& ConnectX®-4]/[http://www.mellanox.com/page/products_dyn?product_family=214&mtag=connectx_4_lx_en_ic ConnectX®-4Lx]/[http://www.mellanox.com/page/products_dyn?product_family=258&mtag=connectx_5_vpi_card ConnectX®-5] with [http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers MLNX_OFED_LINUX] the latest 3.X .<br />
* A running OpenStack environment installed with the ML2 plugin on top of OpenVswitch or Linux Bridge (RDO Manager or Packstack).<br />
* [https://community.mellanox.com/docs/DOC-1317 SR-IOV] enabled on all compute nodes.<br />
* The software package [http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 iproute2] installed on all Compute nodes<br />
* [http://www.mellanox.com/page/products_dyn?product_family=100 Mellanox UFM] greater than 5.9.5<br />
* Add the following repositories on each node<br />
yum -y install yum-plugin-priorities<br />
cd /etc/yum.repos.d/<br />
sudo wget https://trunk.rdoproject.org/centos7-rocky/current/delorean.repo<br />
yum update -y<br />
<br />
=InfiniBand Network=<br />
The Mellanox Neutron Plugin uses InfiniBand Partitions (PKeys) to separate Networks. <br />
<br />
==SM Node==<br />
<br />
=== OpenSM Provisioning with SDN Mechanism Driver ===<br />
<br />
[https://community.mellanox.com/docs/DOC-2155 SDN Mechanism Driver] allows OpenSM dynamically assign PKs in the IB network.<br />
<br />
More details about applying SDN Mechanism Driver with [http://www.mellanox.com/page/products_dyn?product_family=220&mtag=mellanox_neo NEO] can be found [https://community.mellanox.com/docs/DOC-2251 here]<br />
<br />
=== Manual OpenSM Configuration ===<br />
<br />
All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf)<br />
(Automatic cofiguration is planned in future phase)<br />
<br />
====For ConnectX®-3/ConnectX®-3Pro use the following configuration====<br />
<br />
Add/Change the following in the partitions.conf file<br />
management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;<br />
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;<br />
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;<br />
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;<br />
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;<br />
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;<br />
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;<br />
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;<br />
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;<br />
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;<br />
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;<br />
Change the following in /etc/opensm/opensm.conf:<br />
allow_both_pkeys TRUE<br />
<br />
====For ConnectX®-4/ConnectX®-5 use the following configuration====<br />
<br />
Add/Change the following in the partitions.conf.user_ext<br />
vlan1=0x1, ipoib, sl=0, defmember=full: ALL_CAS;<br />
#Storage and management vlan should be define as follow<br />
vlan2=0x2, ipoib, sl=0, defmember=full: ALL_CAS;<br />
vlan3=0x3, ipoib, sl=0, defmember=full: ALL_CAS;<br />
#define OpenSM as a member for all OpenStack vlans. If not guest will have link down on “ibdev2netdev” and no connectivity.<br />
vlan4=0x4, ipoib, sl=0, defmember=full: SELF;<br />
vlan5=0x5, ipoib, sl=0, defmember=full: SELF;<br />
vlan6=0x6, ipoib, sl=0, defmember=full: SELF;<br />
vlan7=0x7, ipoib, sl=0, defmember=full: SELF;<br />
vlan8=0x8, ipoib, sl=0, defmember=full: SELF;<br />
vlan9=0x9, ipoib, sl=0, defmember=full: SELF;<br />
vlan10=0xa, ipoib, sl=0, defmember=full: SELF;<br />
<br />
Change the following in /etc/opensm/opensm.conf:<br />
virt_enabled 2<br />
part_enforce off<br />
allow_both_pkeys FALSE<br />
<br />
4. Restart the OpenSM: <br />
# systemctl restart opensmd.service<br />
<br />
==Controller Node==<br />
<br />
To configure the Controller node:<br />
<br />
1. Install Mellanox RPMs: <br />
# yum install -y --nogpgcheck python-networking-mlnx <br />
<br />
===Neutron Server===<br />
1. Make sure ML2 is the current Neutron plugin by checking the core_plugin parameter in '''/etc/neutron/neutron.conf''': <br />
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin<br />
<br />
2. Make sure '''/etc/neutron/plugin.ini''' is pointing to '''/etc/neutron/plugins/ml2/ml2_conf.ini''' (symbolic link) <br />
<br />
3. Modify '''/etc/neutron/plugins/ml2/ml2_conf.ini''' by adding the following: <br />
[ml2]<br />
type_drivers = vlan,flat<br />
tenant_network_types = vlan<br />
mechanism_drivers = mlnx_infiniband, openvswitch<br />
# or mechanism_drivers = mlnx_infiniband, linuxbridge<br />
[ml2_type_vlan]<br />
network_vlan_ranges = default:1:10<br />
<br />
4. Start (or restart) the Neutron server: <br />
# systemctl restart neutron-server.service<br />
<br />
===Nova Scheduler===<br />
Enabling PciPassthroughFilter modify /etc/nova/nova.conf<br />
scheduler_available_filters = nova.scheduler.filters.all_filters<br />
scheduler_default_filters = RetryFilter, AvailabilityZoneFilter, RamFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, PciPassthroughFilter<br />
<br />
== Network Node==<br />
To configure the Network node:<br />
<br />
Prerequisites:<br />
E_IPoIB port is configured and up<br />
<br />
1. Make sure that eIPoIB module is up and configured in '''/etc/infiniband/openib.conf''': For more information, please refer to eIPoIB configuration in Mellanox OFED User Manual. <br />
E_IPOIB_LOAD=yes<br />
<br />
<br />
2. Restart openibd: <br />
# service openibd restart<br />
<br />
3. Modify the network bridge configuration according to the use of OpenVswitch or LinuxBridge<br />
* 3.1 OpenVswitch '''/etc/neutron/plugins/ml2/openvswitch_agent.ini''' <br />
[ovs]<br />
bridge_mappings = default:br-<eIPoIB interface><br />
* 3.2 LinuxBridge file '''/etc/neutron/plugins/ml2/linuxbridge_agent.ini''':<br />
[linux_bridge] <br />
physical_interface_mappings = default:<eIPoIB interface><br />
<br />
<br />
NOTE: In order to obtain the eIPoIB interface name, run the ethtool tool command (see below) and check that driver name is eth_ipoib: <br />
# ethtool -i <eIPoIB_interface> <br />
driver: eth_ipoib<br />
.....<br />
<br />
<br />
4. Restart network bridge: <br />
* 4.1 OpenVswitch<br />
systemctl restart neutron-openvswitch-agent.service<br />
* 4.2 LinuxBridge (if using linux bridge)<br />
# service neutron-linuxbridge-agent restart<br />
<br />
<br />
NOTE: For DHCP support, the Network node should use the Mellanox Dnsmasq driver as the DHCP driver. <br />
<br />
===DHCP Server (Usually part of the Network node)===<br />
<br />
1. Modify '''/etc/neutron/dhcp_agent.ini''' as follows and according to OVS or Linuxbridge: <br />
dhcp_driver = networking_mlnx.dhcp.mlnx_dhcp.MlnxDnsmasq<br />
dhcp_broadcast_reply = True<br />
<br />
1.1 For OVS<br />
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver<br />
1.2 For Linux Bridge<br />
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver<br />
<br />
<br />
2. Restart DHCP server: <br />
# systemctl restart neutron-dhcp-agent.service<br />
<br />
==Compute Nodes==<br />
<br />
To configure the Compute Node: <br />
<br />
1. Install Mellanox RPMs: <br />
# yum install --nogpgcheck -y python-networking-mlnx<br />
<br />
2. Create the file '''/etc/modprobe.d/mlx4_ib.conf''' and add the following: <br />
options mlx4_ib sm_guid_assign=0<br />
<br />
3. Restart the driver: <br />
# /etc/init.d/openibd restart<br />
<br />
===Nova Compute===<br />
Nova-compute needs to know which PCI devices are allowed to be passed through to the VMs. <br />
Also for SRIOV PCI devices it needs to know to which physical network the VF belongs.<br />
This is done through the pci_passthrough_whitelist parameter under the default section in '''/etc/nova/nova.conf'''.<br />
For example if we want to whitelist and tag the VFs by their PCI address we would use the following setting:<br />
[pci]<br />
passthrough_whitelist = {"address":"*:0a:00.*","physical_network":"default"}<br />
This associates any VF with address that includes ':0a:00.' in its address to the physical network default.<br />
<br />
1. add pci passthrough_whitelist to '''/etc/nova/nova.conf'''<br />
<br />
2. Restart Nova:<br />
# systemctl restart openstack-nova-compute<br />
<br />
===Neutron MLNX Agent===<br />
1. Run: <br />
# systemctl enable neutron-mlnx-agent.service<br />
# systemctl start neutron-mlnx-agent.service<br />
<br />
2. Run: <br />
# systemctl daemon-reload<br />
<br />
3. In the file '''/etc/neutron/plugins/mlnx/mlnx_conf.ini''', the parameters tenant_network_type , and network_vlan_ranges should be configured as the controllers: <br />
physical_interface_mappings = default:<ib_interface>(for example default:ib0)<br />
<br />
4. Modify the file '''/etc/neutron/plugins/ml2/eswitchd.conf''' as follows:<br />
fabrics = default:<ib_interface> (for example default:ib0)<br />
<br />
5. Start eSwitchd: <br />
# systemctl enable eswitchd.service<br />
# systemctl start eswitchd.service<br />
<br />
6. Start the Neutron agent: <br />
# systemctl restart neutron-mlnx-agent<br />
<br />
==Known issues and Troubleshooting==<br />
For known issues and troubleshooting options refer to [https://community.mellanox.com/docs/DOC-1127 Mellanox OpenStack Troubleshooting]<br />
<br />
Issue: Missing zmq package on all nodes (Controller/Compute)<br />
Solution:<br />
<pre><br />
# wget https://bootstrap.pypa.io/get-pip.py<br />
# sudo python get-pip.py<br />
# sudo pip install pyzmq<br />
</pre></div>
Hasanq