https://wiki.openstack.org/w/api.php?action=feedcontributions&feedformat=atom&user=May 2026-07-12T03:01:04Z User contributions MediaWiki 1.28.2 https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Kilo-InfiniBand&diff=136936 2016-10-31T15:34:39Z

<p>May: </p> <hr /> <div><br /> Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API. <br /> <br /> This driver supports Mellanox embedded switch functionality as part of the InfiniBand HCA. Mellanox ML2 Mechanism Driver provides a functional parity with Mellanox Neutron plugin. <br /> <br /> Mellanox ML2 Mechanism Driver supports DIRECT (PCI passthrough) vNIC type. For vNIC type configuration API details, please refer to [https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking#VM_creation_flow_with_SR-IOV_vNIC configuration reference guide]. Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as Remote Direct Memory Access (RDMA).<br /> <br /> The driver supports VLAN network type to facilitate virtual networks on InfiniBand fabrics. <br /> * Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node. <br /> * Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and an embedded switch port.<br /> <br /> <br /> '''Prerequisites'''<br /> * All nodes equipped with Mellanox ConnectX®-3/Mellanox ConnectX®-3 PRO [http://www.mellanox.com/page/products_dyn?product_family=119 Network Adapter]<br /> * Mellanox OFED 2.4 or above installed on all nodes. Please refer to Mellanox website for the latest [http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers OFED] version<br /> <br /> <br /> For '''Kilo''' release information, please refer to the relevant OS as follows:<br /> * [[Mellanox-Neutron-Kilo-Redhat-InfiniBand | Redhat7]]<br /> * [[Mellanox-Neutron-Kilo-Ubuntu-InfiniBand | Ubuntu14.04]]</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Cinder&diff=136935 2016-10-31T15:33:20Z

<p>May: /* Known Issues */</p> <hr /> <div>= Overview = <br /> Mellanox added iSCSI Extensions for RDMA (iSER) that does not need the TCP layer. Therefore, it provides lower latency and higher performance.&lt;br /&gt;<br /> It permits data to be transferred directly into and out of SCSI computer memory buffers, which connects computers to storage devices, without intermediate data copies.&lt;br /&gt;<br /> iSER is a server network protocol that extends the Internet Small Computer System Interface (iSCSI) protocol to use Remote Direct Memory Access (RDMA).&lt;br /&gt;&lt;br /&gt;<br /> Please contact Mellanox for further information and benchmark results.<br /> <br /> == Installation ==<br /> Provisioned through standard Cinder installation.<br /> <br /> == Configuration ==<br /> In order to enable iSER, adjust these values at '''/etc/cinder/cinder.conf''':<br /> <br /> iscsi_ip_address = &lt;ipoib/roce_address&gt;<br /> iscsi_protocol = iser<br /> volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver<br /> <br /> '''iscsi_ip_address''' is required to do a &quot;discovery&quot; over the IB/RoCE interface from the initiator side.<br /> <br /> '''volume_driver''' points cinder to use the LVMVolumeDriver.<br /> <br /> Restart Cinder service<br /> <br /> #/etc/init.d/openstack-cinder-volume restart<br /> <br /> For troubleshooting issues, refer to section [https://wiki.openstack.org/wiki/Mellanox-Cinder#Known_Issues Known Issues]<br /> <br /> = Known Issues =<br /> <br /> (1) “scsi-target-utils” package:<br /> <br /> '''Note''': For RH6.4 or below, “scsi-target-utils” package version needs to be 1.0.38 and above. If you use RDO installation, you should check the version.<br /> The RPM can be found at http://www.mellanox.com/downloads/solutions/rpms/<br /> <br /> #wget http://www.mellanox.com/downloads/solutions/rpms/scsi-target-utils-1.0.39-v1.0.39.c1135a.x86_64.rpm<br /> #rpm -Uvh scsi-target-utils-1.0.39-v1.0.39.c1135a.x86_64.rpm<br /> <br /> In addition, RH6.5 includes “scsi-target-utils” package with version 1.0.24-10. This version number is sufficient and no further action is needed for RHEL6.5.<br /> <br /> (2) Boot Instance from volume fails when iSER is enabled:<br /> <br /> Steps to reproduce the bug:<br /> <br /> - Create new volume with image source.<br /> <br /> - Launch a new instance boot from the above volume using iSER transport.<br /> <br /> The bug is defined here:<br /> [https://bugs.launchpad.net/nova/+bug/1253578]<br /> <br /> A patch can be found here: [http://www.mellanox.com/downloads/solutions/openstack/havana/patches/volume.patch]<br /> <br /> To apply the patch, run the following commands on '''all nova-compute nodes''':<br /> # wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/volume.patch <br /> # patch -p1 &lt; volume.patch<br /> # service nova-compute restart <br /> <br /> (3) Flow control:<br /> <br /> In case the network is Ethernet, make sure that flow control is enabled on the switches on the relevant ports across the network.<br /> <br /> (4) Linux-IO (LIO) support limitation:<br /> <br /> Open-source implementation of SCSI target in Openstack can be managed by either TGT or LIO.&lt;br /&gt;<br /> Currently in Openstack, iSER is supported over TGT only, and iSER support over LIO is scheduled for Kilo version.&lt;br /&gt;<br /> In case LIO is the default SCSI target, TGT must be installed and Openstack Cinder configuration file must be altered to use TGT instead of LIO.<br /> <br /> = References =<br /> 1. [http://www.mellanox.com/openstack/ OpenStack solution page at Mellanox site]<br /> <br /> 2. [https://github.com/mellanox-openstack Source repository]<br /> <br /> 3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED web page]<br /> <br /> 4. [http://docs.openstack.org/havana/install-guide/install/yum/content/cinder-controller.html Cinder-controller]<br /> <br /> 5. [http://docs.openstack.org/havana/install-guide/install/yum/content/cinder-node.html Cinder-node]<br /> <br /> Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.<br /> <br /> For more details, please direct any inquiries to [mailto:openstack@mellanox.com openstack@mellanox.com].</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Liberty-InfiniBand&diff=136934 2016-10-31T15:31:38Z

<p>May: </p> <hr /> <div><br /> Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API. <br /> <br /> This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides a functional parity with Mellanox Neutron plugin. <br /> <br /> Mellanox ML2 Mechanism Driver supports DIRECT (PCI passthrough) vNIC type. For vNIC type configuration API details, please refer to [https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking#VM_creation_flow_with_SR-IOV_vNIC configuration reference guide]. Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as Remote Direct Memory Access (RDMA).<br /> <br /> The driver supports VLAN network type to facilitate virtual networks either on Ethernet or on InfiniBand fabrics. <br /> * Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node. <br /> * Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and an embedded switch port.<br /> <br /> <br /> '''Prerequisites'''<br /> * Mellanox ConnectX®-3/Mellanox ConnectX®-3 PRO [http://www.mellanox.com/page/products_dyn?product_family=119 Network Adapter] with Mellanox OFED 2.4 or greater<br /> * Mellanox ConnectX®-4 [http://www.mellanox.com/page/products_dyn?product_family=201&amp; Network Adapter] with MLNX_OFED_LINUX-3.1-1.5.7 and greater.<br /> * [http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers Mellanox OFED download page] <br /> <br /> <br /> For '''Liberty''' release Information refer to the relevant OS as follows:<br /> <br /> * [[Mellanox-Neutron-Liberty-Redhat-InfiniBand | Redhat7]]<br /> * [[Mellanox-Neutron-Liberty-Ubuntu-InfiniBand | Ubuntu14.04]]<br /> &lt;br /&gt;</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Newton-InfiniBand&diff=136932 2016-10-31T14:54:22Z

<p>May: </p> <hr /> <div>Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API. <br /> <br /> This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA.<br /> Mellanox ML2 Mechanism Driver provides a functional parity with Mellanox Neutron plugin.<br /> <br /> Mellanox ML2 Mechanism Driver supports DIRECT (PCI passthrough) vNIC type. For vNIC type configuration API details, please refer to [https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking#VM_creation_flow_with_SR-IOV_vNIC configuration reference guide].<br /> Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as Remote Direct Memory Access (RDMA). <br /> <br /> The driver supports VLAN network type to facilitate virtual networks either on Ethernet or on InfiniBand fabrics. <br /> * Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node. <br /> * Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and an embedded switch port.<br /> <br /> <br /> '''Prerequisites'''<br /> * Mellanox ConnectX®-3/Mellanox ConnectX®-3 PRO [http://www.mellanox.com/page/products_dyn?product_family=119 Network Adapter] with [http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers Mellanox OFED 2.4 or greater]<br /> * Mellanox ConnectX®-4 [http://www.mellanox.com/page/products_dyn?product_family=201&amp; Network Adapter] with [http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers MLNX_OFED_LINUX-3.1-1.5.7 or greater].<br /> <br /> <br /> For '''Newton''' release Information refer to the relevant OS as follows:<br /> <br /> * [[Mellanox-Neutron-Newton-Redhat-InfiniBand | Redhat7]]<br /> * [[Mellanox-Neutron-Newton-Ubuntu-InfiniBand | Ubuntu14.04]]<br /> &lt;br /&gt;</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Newton-InfiniBand&diff=136931 2016-10-31T14:53:15Z

<p>May: </p> <hr /> <div>Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API. <br /> <br /> This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA.<br /> Mellanox ML2 Mechanism Driver provides a functional parity with Mellanox Neutron plugin.<br /> <br /> Mellanox ML2 Mechanism Driver supports DIRECT (PCI passthrough) vNIC type. For vNIC type configuration API details, please refer to configuration reference guide.<br /> Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as Remote Direct Memory Access (RDMA). <br /> <br /> The driver supports VLAN network type to facilitate virtual networks either on Ethernet or on InfiniBand fabrics. <br /> * Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node. <br /> * Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and an embedded switch port.<br /> <br /> <br /> '''Prerequisites'''<br /> * Mellanox ConnectX®-3/Mellanox ConnectX®-3 PRO [http://www.mellanox.com/page/products_dyn?product_family=119 Network Adapter] with [http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers Mellanox OFED 2.4 or greater]<br /> * Mellanox ConnectX®-4 [http://www.mellanox.com/page/products_dyn?product_family=201&amp; Network Adapter] with [http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers MLNX_OFED_LINUX-3.1-1.5.7 or greater].<br /> <br /> <br /> For '''Newton''' release Information refer to the relevant OS as follows:<br /> <br /> * [[Mellanox-Neutron-Newton-Redhat-InfiniBand | Redhat7]]<br /> * [[Mellanox-Neutron-Newton-Ubuntu-InfiniBand | Ubuntu14.04]]<br /> &lt;br /&gt;</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Juno-InfiniBand&diff=136930 2016-10-31T14:52:15Z

<p>May: </p> <hr /> <div><br /> Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API. <br /> <br /> This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides a functional parity with Mellanox Neutron plugin. <br /> <br /> Mellanox ML2 Mechanism Driver supports DIRECT (PCI passthrough) vNIC type. For vNIC type configuration API details, please refer to [https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking#VM_creation_flow_with_SR-IOV_vNIC configuration reference guide]. Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as Remote Direct Memory Access (RDMA).<br /> <br /> The driver supports VLAN network type to facilitate virtual networks either on Ethernet or on InfiniBand fabrics. <br /> * Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node. <br /> * Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and an embedded switch port.<br /> <br /> <br /> '''Prerequisites'''<br /> * All nodes equipped with Mellanox ConnectX®-3/Mellanox ConnectX®-3 PRO [http://www.mellanox.com/page/products_dyn?product_family=119 Network Adapter]<br /> * Mellanox OFED 2.4 or greater installed on all nodes. Please refer to Mellanox website for the latest [http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers OFED] version<br /> <br /> <br /> For '''Juno''' release information, please refer to the relevant OS as follows:<br /> <br /> * [[Mellanox-Neutron-Juno-Redhat-InfiniBand|Redhat7]]<br /> * [[Mellanox-Neutron-Juno-Ubuntu-InfiniBand|Ubuntu14.04]]</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Juno-InfiniBand&diff=136929 2016-10-31T14:35:55Z

<p>May: </p> <hr /> <div><br /> Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API. <br /> <br /> This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides a functional parity with Mellanox Neutron plugin. <br /> <br /> Mellanox ML2 Mechanism Driver supports DIRECT (PCI passthrough) vNIC type. For vNIC type configuration API details, please refer to configuration reference guide (click here). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as Remote Direct Memory Access (RDMA).<br /> <br /> The driver supports VLAN network type to facilitate virtual networks either on Ethernet or on InfiniBand fabrics. <br /> * Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node. <br /> * Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and an embedded switch port.<br /> <br /> <br /> '''Prerequisites'''<br /> * All nodes equipped with Mellanox ConnectX®-3/Mellanox ConnectX®-3 PRO [http://www.mellanox.com/page/products_dyn?product_family=119 Network Adapter]<br /> * Mellanox OFED 2.4 or greater installed on all nodes. Please refer to Mellanox website for the latest [http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers OFED] version<br /> <br /> <br /> For '''Juno''' release information, please refer to the relevant OS as follows:<br /> <br /> * [[Mellanox-Neutron-Juno-Redhat-InfiniBand|Redhat7]]<br /> * [[Mellanox-Neutron-Juno-Ubuntu-InfiniBand|Ubuntu14.04]]</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Kilo-InfiniBand&diff=136928 2016-10-31T14:31:48Z

<p>May: </p> <hr /> <div><br /> Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API. <br /> <br /> This driver supports Mellanox embedded switch functionality as part of the InfiniBand HCA. Mellanox ML2 Mechanism Driver provides a functional parity with Mellanox Neutron plugin. <br /> <br /> Mellanox ML2 Mechanism Driver supports DIRECT (PCI passthrough) vNIC type. For vNIC type configuration API details, please refer to configuration reference guide. Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as Remote Direct Memory Access (RDMA).<br /> <br /> The driver supports VLAN network type to facilitate virtual networks on InfiniBand fabrics. <br /> * Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node. <br /> * Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and an embedded switch port.<br /> <br /> <br /> '''Prerequisites'''<br /> * All nodes equipped with Mellanox ConnectX®-3/Mellanox ConnectX®-3 PRO [http://www.mellanox.com/page/products_dyn?product_family=119 Network Adapter]<br /> * Mellanox OFED 2.4 or above installed on all nodes. Please refer to Mellanox website for the latest [http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers OFED] version<br /> <br /> <br /> For '''Kilo''' release information, please refer to the relevant OS as follows:<br /> * [[Mellanox-Neutron-Kilo-Redhat-InfiniBand | Redhat7]]<br /> * [[Mellanox-Neutron-Kilo-Ubuntu-InfiniBand | Ubuntu14.04]]</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Kilo-InfiniBand&diff=136927 2016-10-31T14:31:09Z

<p>May: </p> <hr /> <div><br /> Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API. <br /> <br /> This driver supports Mellanox embedded switch functionality as part of the InfiniBand HCA. Mellanox ML2 Mechanism Driver provides a functional parity with Mellanox Neutron plugin. <br /> <br /> Mellanox ML2 Mechanism Driver supports DIRECT (PCI passthrough) vNIC type. For vNIC type configuration API details, please refer to configuration reference guide. Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as Remote Direct Memory Access (RDMA).<br /> <br /> The driver supports VLAN network type to facilitate virtual networks on InfiniBand fabrics. <br /> * Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node. <br /> * Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and an embedded switch port.<br /> <br /> <br /> '''Prerequisites'''<br /> * All nodes equipped with Mellanox ConnectX®-3/Mellanox ConnectX®-3 PRO [http://www.mellanox.com/page/products_dyn?product_family=119 Network Adapter]<br /> * Mellanox OFED 2.4 or above installed on all nodes. Please refer to Mellanox website for the latest [http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers OFED]<br /> <br /> <br /> For '''Kilo''' release information, please refer to the relevant OS as follows:<br /> * [[Mellanox-Neutron-Kilo-Redhat-InfiniBand | Redhat7]]<br /> * [[Mellanox-Neutron-Kilo-Ubuntu-InfiniBand | Ubuntu14.04]]</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Juno-Ethernet&diff=136926 2016-10-31T14:28:53Z

<p>May: /* SR-IOV Networking in OpenStack Juno */</p> <hr /> <div>=Ethernet Configuration with Mellanox SR-IOV Networking in OpenStack Juno= <br /> OpenStack Juno added inbox support to request VM access to virtual network via SR-IOV NIC. With the introduction of SR-IOV based NICs, the traditional virtual bridge is no longer required. Each SR-IOV port is associated with a virtual function (VF). SR-IOV ports may be provided by Hardware-based Virtual Ethernet Bridging (HW VEB); or they may be extended to an upstream physical switch (IEEE 802.1br).<br /> There are two ways in which SR-IOV port may be connected:<br /> * Directly connected to its VF<br /> * Connected with a macvtap device that resides on the host, which is then connected to the corresponding VF<br /> <br /> ==Configuration==<br /> [[SR-IOV-Passthrough-For-Networking|Configure SR-IOV]]</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Juno-Ethernet&diff=136925 2016-10-31T14:27:56Z

<p>May: /* Ethernet Configuration with Mellanox */</p> <hr /> <div>=SR-IOV Networking in OpenStack Juno= <br /> OpenStack Juno added inbox support to request VM access to virtual network via SR-IOV NIC. With the introduction of SR-IOV based NICs, the traditional virtual bridge is no longer required. Each SR-IOV port is associated with a virtual function (VF). SR-IOV ports may be provided by Hardware-based Virtual Ethernet Bridging (HW VEB); or they may be extended to an upstream physical switch (IEEE 802.1br).<br /> There are two ways that SR-IOV port may be connected:<br /> * directly connected to its VF<br /> * connected with a macvtap device that resides on the host, which is then connected to the corresponding VF<br /> <br /> ==Configuration==<br /> [[SR-IOV-Passthrough-For-Networking|Configure SR-IOV]]</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Kilo-Ethernet&diff=136924 2016-10-31T14:25:37Z

<p>May: /* SDN Mechanism Driver */</p> <hr /> <div>=Ethernet Configuration with Mellanox SR-IOV Networking in OpenStack Kilo= <br /> OpenStack Juno added inbox support to request VM access to virtual network via SR-IOV NIC. With the introduction of SR-IOV based NICs, the traditional virtual bridge is no longer required. Each SR-IOV port is associated with a virtual function (VF). SR-IOV ports may be provided by Hardware-based Virtual Ethernet Bridging (HW VEB); or they may be extended to an upstream physical switch (IEEE 802.1br).<br /> There are two ways in which the SR-IOV port may be connected:<br /> * Directly connected to its VF<br /> * Connected with a macvtap device that resides on the host, which is then connected to the corresponding VF<br /> <br /> ==Configuration==<br /> [[SR-IOV-Passthrough-For-Networking|Configure SR-IOV]]<br /> <br /> ==SDN Mechanism Driver==<br /> [https://community.mellanox.com/docs/DOC-2155 SDN Mechanism Driver] allows NEO to dynamically configure VLANs on the Mellanox Switches.<br /> Further reading:<br /> * [http://www.mellanox.com/page/products_dyn?product_family=220&amp;mtag=mellanox_neo NEO]<br /> * [http://www.mellanox.com/page/ethernet_switch_overview Mellanox Switches]<br /> * [https://community.mellanox.com/docs/DOC-2251 SDN Mechanism Driver]<br /> <br /> ==Ceilometer==<br /> The Mellanox Ceilometer SR-IOV module allows the OpenStack Ceilometer to collect measurements of SR-IOV counters.<br /> <br /> More details about applying Mellanox Ceilometer module can be found [https://community.mellanox.com/docs/DOC-2145 here]</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Kilo-Ethernet&diff=136923 2016-10-31T14:23:47Z

<p>May: /* SR-IOV Networking in OpenStack Kilo */</p> <hr /> <div>=Ethernet Configuration with Mellanox SR-IOV Networking in OpenStack Kilo= <br /> OpenStack Juno added inbox support to request VM access to virtual network via SR-IOV NIC. With the introduction of SR-IOV based NICs, the traditional virtual bridge is no longer required. Each SR-IOV port is associated with a virtual function (VF). SR-IOV ports may be provided by Hardware-based Virtual Ethernet Bridging (HW VEB); or they may be extended to an upstream physical switch (IEEE 802.1br).<br /> There are two ways in which the SR-IOV port may be connected:<br /> * Directly connected to its VF<br /> * Connected with a macvtap device that resides on the host, which is then connected to the corresponding VF<br /> <br /> ==Configuration==<br /> [[SR-IOV-Passthrough-For-Networking|Configure SR-IOV]]<br /> <br /> ==SDN Mechanism Driver==<br /> [https://community.mellanox.com/docs/DOC-2155 SDN Mechanism Driver] allows NEO to configure dynamically VLANs on the Mellanox Switches.<br /> Further Reading<br /> * [http://www.mellanox.com/page/products_dyn?product_family=220&amp;mtag=mellanox_neo NEO]<br /> * [http://www.mellanox.com/page/ethernet_switch_overview Mellanox Switches]<br /> * [https://community.mellanox.com/docs/DOC-2251 SDN Mechanism Driver]<br /> <br /> ==Ceilometer==<br /> The Mellanox Ceilometer SR-IOV module allows the OpenStack Ceilometer to collect measurements of SR-IOV counters.<br /> <br /> More details about applying Mellanox Ceilometer module can be found [https://community.mellanox.com/docs/DOC-2145 here]</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Kilo-Ethernet&diff=136922 2016-10-31T14:22:07Z

<p>May: /* Ethernet Configuration with Mellanox */</p> <hr /> <div>=SR-IOV Networking in OpenStack Kilo= <br /> OpenStack Juno added inbox support to request VM access to virtual network via SR-IOV NIC. With the introduction of SR-IOV based NICs, the traditional virtual bridge is no longer required. Each SR-IOV port is associated with a virtual function (VF). SR-IOV ports may be provided by Hardware-based Virtual Ethernet Bridging (HW VEB); or they may be extended to an upstream physical switch (IEEE 802.1br).<br /> There are two ways in which the SR-IOV port may be connected:<br /> * Directly connected to its VF<br /> * Connected with a macvtap device that resides on the host, which is then connected to the corresponding VF<br /> <br /> ==Configuration==<br /> [[SR-IOV-Passthrough-For-Networking|Configure SR-IOV]]<br /> <br /> ==SDN Mechanism Driver==<br /> [https://community.mellanox.com/docs/DOC-2155 SDN Mechanism Driver] allows NEO to configure dynamically VLANs on the Mellanox Switches.<br /> Further Reading<br /> * [http://www.mellanox.com/page/products_dyn?product_family=220&amp;mtag=mellanox_neo NEO]<br /> * [http://www.mellanox.com/page/ethernet_switch_overview Mellanox Switches]<br /> * [https://community.mellanox.com/docs/DOC-2251 SDN Mechanism Driver]<br /> <br /> ==Ceilometer==<br /> The Mellanox Ceilometer SR-IOV module allows the OpenStack Ceilometer to collect measurements of SR-IOV counters.<br /> <br /> More details about applying Mellanox Ceilometer module can be found [https://community.mellanox.com/docs/DOC-2145 here]</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Kilo-Ethernet&diff=136921 2016-10-31T14:18:24Z

<p>May: /* SR-IOV Networking in OpenStack Kilo */</p> <hr /> <div>=Ethernet Configuration with Mellanox=<br /> =SR-IOV Networking in OpenStack Kilo= <br /> OpenStack Juno added inbox support to request VM access to virtual network via SR-IOV NIC. With the introduction of SR-IOV based NICs, the traditional virtual bridge is no longer required. Each SR-IOV port is associated with a virtual function (VF). SR-IOV ports may be provided by Hardware-based Virtual Ethernet Bridging (HW VEB); or they may be extended to an upstream physical switch (IEEE 802.1br).<br /> There are two ways in which the SR-IOV port may be connected:<br /> * Directly connected to its VF<br /> * Connected with a macvtap device that resides on the host, which is then connected to the corresponding VF<br /> <br /> ==Configuration==<br /> [[SR-IOV-Passthrough-For-Networking|Configure SR-IOV]]<br /> <br /> ==SDN Mechanism Driver==<br /> [https://community.mellanox.com/docs/DOC-2155 SDN Mechanism Driver] allows NEO to configure dynamically VLANs on the Mellanox Switches.<br /> Further Reading<br /> * [http://www.mellanox.com/page/products_dyn?product_family=220&amp;mtag=mellanox_neo NEO]<br /> * [http://www.mellanox.com/page/ethernet_switch_overview Mellanox Switches]<br /> * [https://community.mellanox.com/docs/DOC-2251 SDN Mechanism Driver]<br /> <br /> ==Ceilometer==<br /> The Mellanox Ceilometer SR-IOV module allows the OpenStack Ceilometer to collect measurements of SR-IOV counters.<br /> <br /> More details about applying Mellanox Ceilometer module can be found [https://community.mellanox.com/docs/DOC-2145 here]</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Cinder&diff=136919 2016-10-31T13:41:19Z

<p>May: /* References */</p> <hr /> <div>= Overview = <br /> Mellanox added iSCSI Extensions for RDMA (iSER) that does not need the TCP layer. Therefore, it provides lower latency and higher performance.&lt;br /&gt;<br /> It permits data to be transferred directly into and out of SCSI computer memory buffers, which connects computers to storage devices, without intermediate data copies.&lt;br /&gt;<br /> iSER is a server network protocol that extends the Internet Small Computer System Interface (iSCSI) protocol to use Remote Direct Memory Access (RDMA).&lt;br /&gt;&lt;br /&gt;<br /> Please contact Mellanox for further information and benchmark results.<br /> <br /> == Installation ==<br /> Provisioned through standard Cinder installation.<br /> <br /> == Configuration ==<br /> In order to enable iSER, adjust these values at '''/etc/cinder/cinder.conf''':<br /> <br /> iscsi_ip_address = &lt;ipoib/roce_address&gt;<br /> iscsi_protocol = iser<br /> volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver<br /> <br /> '''iscsi_ip_address''' is required to do a &quot;discovery&quot; over the IB/RoCE interface from the initiator side.<br /> <br /> '''volume_driver''' points cinder to use the LVMVolumeDriver.<br /> <br /> Restart Cinder service<br /> <br /> #/etc/init.d/openstack-cinder-volume restart<br /> <br /> For troubleshooting issues, refer to section [https://wiki.openstack.org/wiki/Mellanox-Cinder#Known_Issues Known Issues]<br /> <br /> = Known Issues =<br /> <br /> (1) “scsi-target-utils” package <br /> <br /> '''Note''': For RH6.4 or below, “scsi-target-utils” package version needs to be 1.0.38 and above. If you use RDO installation, you should check the version.<br /> The RPM can be found at http://www.mellanox.com/downloads/solutions/rpms/<br /> <br /> #wget http://www.mellanox.com/downloads/solutions/rpms/scsi-target-utils-1.0.39-v1.0.39.c1135a.x86_64.rpm<br /> #rpm -Uvh scsi-target-utils-1.0.39-v1.0.39.c1135a.x86_64.rpm<br /> <br /> In addition, RH6.5 includes “scsi-target-utils” package with version 1.0.24-10. This version number is sufficient and no further action is needed for RHEL6.5.<br /> <br /> (2) Boot Instance from volume fails when iSER is enabled.<br /> <br /> Steps to reproduce the bug:<br /> <br /> - Create new volume with image source.<br /> <br /> - Launch a new instance boot from the above volume using iSER transport.<br /> <br /> The bug is defined here:<br /> [https://bugs.launchpad.net/nova/+bug/1253578]<br /> <br /> A patch can be found here: [http://www.mellanox.com/downloads/solutions/openstack/havana/patches/volume.patch]<br /> <br /> To apply the patch, run the following commands on '''all nova-compute nodes''':<br /> # wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/volume.patch <br /> # patch -p1 &lt; volume.patch<br /> # service nova-compute restart <br /> <br /> (3) Flow control<br /> <br /> In case the network is Ethernet, make sure that flow control is enabled on the switches on the relevant ports across the network.<br /> <br /> (4) Linux-IO (LIO) support limitation.<br /> <br /> Open-source implementation of SCSI target in Openstack can be managed by either TGT or LIO.&lt;br /&gt;<br /> Currently in Openstack, iSER is supported over TGT only, and iSER support over LIO is scheduled for Kilo version.&lt;br /&gt;<br /> In case LIO is the default SCSI target, TGT must be installed and Openstack Cinder configuration file must be altered to use TGT instead of LIO.<br /> <br /> = References =<br /> 1. [http://www.mellanox.com/openstack/ OpenStack solution page at Mellanox site]<br /> <br /> 2. [https://github.com/mellanox-openstack Source repository]<br /> <br /> 3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED web page]<br /> <br /> 4. [http://docs.openstack.org/havana/install-guide/install/yum/content/cinder-controller.html Cinder-controller]<br /> <br /> 5. [http://docs.openstack.org/havana/install-guide/install/yum/content/cinder-node.html Cinder-node]<br /> <br /> Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.<br /> <br /> For more details, please direct any inquiries to [mailto:openstack@mellanox.com openstack@mellanox.com].</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Cinder&diff=136918 2016-10-31T13:40:33Z

<p>May: /* References */</p> <hr /> <div>= Overview = <br /> Mellanox added iSCSI Extensions for RDMA (iSER) that does not need the TCP layer. Therefore, it provides lower latency and higher performance.&lt;br /&gt;<br /> It permits data to be transferred directly into and out of SCSI computer memory buffers, which connects computers to storage devices, without intermediate data copies.&lt;br /&gt;<br /> iSER is a server network protocol that extends the Internet Small Computer System Interface (iSCSI) protocol to use Remote Direct Memory Access (RDMA).&lt;br /&gt;&lt;br /&gt;<br /> Please contact Mellanox for further information and benchmark results.<br /> <br /> == Installation ==<br /> Provisioned through standard Cinder installation.<br /> <br /> == Configuration ==<br /> In order to enable iSER, adjust these values at '''/etc/cinder/cinder.conf''':<br /> <br /> iscsi_ip_address = &lt;ipoib/roce_address&gt;<br /> iscsi_protocol = iser<br /> volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver<br /> <br /> '''iscsi_ip_address''' is required to do a &quot;discovery&quot; over the IB/RoCE interface from the initiator side.<br /> <br /> '''volume_driver''' points cinder to use the LVMVolumeDriver.<br /> <br /> Restart Cinder service<br /> <br /> #/etc/init.d/openstack-cinder-volume restart<br /> <br /> For troubleshooting issues, refer to section [https://wiki.openstack.org/wiki/Mellanox-Cinder#Known_Issues Known Issues]<br /> <br /> = Known Issues =<br /> <br /> (1) “scsi-target-utils” package <br /> <br /> '''Note''': For RH6.4 or below, “scsi-target-utils” package version needs to be 1.0.38 and above. If you use RDO installation, you should check the version.<br /> The RPM can be found at http://www.mellanox.com/downloads/solutions/rpms/<br /> <br /> #wget http://www.mellanox.com/downloads/solutions/rpms/scsi-target-utils-1.0.39-v1.0.39.c1135a.x86_64.rpm<br /> #rpm -Uvh scsi-target-utils-1.0.39-v1.0.39.c1135a.x86_64.rpm<br /> <br /> In addition, RH6.5 includes “scsi-target-utils” package with version 1.0.24-10. This version number is sufficient and no further action is needed for RHEL6.5.<br /> <br /> (2) Boot Instance from volume fails when iSER is enabled.<br /> <br /> Steps to reproduce the bug:<br /> <br /> - Create new volume with image source.<br /> <br /> - Launch a new instance boot from the above volume using iSER transport.<br /> <br /> The bug is defined here:<br /> [https://bugs.launchpad.net/nova/+bug/1253578]<br /> <br /> A patch can be found here: [http://www.mellanox.com/downloads/solutions/openstack/havana/patches/volume.patch]<br /> <br /> To apply the patch, run the following commands on '''all nova-compute nodes''':<br /> # wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/volume.patch <br /> # patch -p1 &lt; volume.patch<br /> # service nova-compute restart <br /> <br /> (3) Flow control<br /> <br /> In case the network is Ethernet, make sure that flow control is enabled on the switches on the relevant ports across the network.<br /> <br /> (4) Linux-IO (LIO) support limitation.<br /> <br /> Open-source implementation of SCSI target in Openstack can be managed by either TGT or LIO.&lt;br /&gt;<br /> Currently in Openstack, iSER is supported over TGT only, and iSER support over LIO is scheduled for Kilo version.&lt;br /&gt;<br /> In case LIO is the default SCSI target, TGT must be installed and Openstack Cinder configuration file must be altered to use TGT instead of LIO.<br /> <br /> = References =<br /> 1. [http://www.mellanox.com/openstack/ OpenStack solution page at Mellanox site]<br /> <br /> 2. [https://github.com/mellanox-openstack Source repository]<br /> <br /> 3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED web page]<br /> <br /> 4. [http://docs.openstack.org/havana/install-guide/install/yum/content/cinder-controller.html Cinder-controller]<br /> <br /> 5. [http://docs.openstack.org/havana/install-guide/install/yum/content/cinder-node.html Cinder-node]<br /> <br /> Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.<br /> <br /> For more details, please address any inquiries to [mailto:openstack@mellanox.com openstack@mellanox.com].</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Cinder&diff=136917 2016-10-31T13:39:54Z

<p>May: /* Known Issues */</p> <hr /> <div>= Overview = <br /> Mellanox added iSCSI Extensions for RDMA (iSER) that does not need the TCP layer. Therefore, it provides lower latency and higher performance.&lt;br /&gt;<br /> It permits data to be transferred directly into and out of SCSI computer memory buffers, which connects computers to storage devices, without intermediate data copies.&lt;br /&gt;<br /> iSER is a server network protocol that extends the Internet Small Computer System Interface (iSCSI) protocol to use Remote Direct Memory Access (RDMA).&lt;br /&gt;&lt;br /&gt;<br /> Please contact Mellanox for further information and benchmark results.<br /> <br /> == Installation ==<br /> Provisioned through standard Cinder installation.<br /> <br /> == Configuration ==<br /> In order to enable iSER, adjust these values at '''/etc/cinder/cinder.conf''':<br /> <br /> iscsi_ip_address = &lt;ipoib/roce_address&gt;<br /> iscsi_protocol = iser<br /> volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver<br /> <br /> '''iscsi_ip_address''' is required to do a &quot;discovery&quot; over the IB/RoCE interface from the initiator side.<br /> <br /> '''volume_driver''' points cinder to use the LVMVolumeDriver.<br /> <br /> Restart Cinder service<br /> <br /> #/etc/init.d/openstack-cinder-volume restart<br /> <br /> For troubleshooting issues, refer to section [https://wiki.openstack.org/wiki/Mellanox-Cinder#Known_Issues Known Issues]<br /> <br /> = Known Issues =<br /> <br /> (1) “scsi-target-utils” package <br /> <br /> '''Note''': For RH6.4 or below, “scsi-target-utils” package version needs to be 1.0.38 and above. If you use RDO installation, you should check the version.<br /> The RPM can be found at http://www.mellanox.com/downloads/solutions/rpms/<br /> <br /> #wget http://www.mellanox.com/downloads/solutions/rpms/scsi-target-utils-1.0.39-v1.0.39.c1135a.x86_64.rpm<br /> #rpm -Uvh scsi-target-utils-1.0.39-v1.0.39.c1135a.x86_64.rpm<br /> <br /> In addition, RH6.5 includes “scsi-target-utils” package with version 1.0.24-10. This version number is sufficient and no further action is needed for RHEL6.5.<br /> <br /> (2) Boot Instance from volume fails when iSER is enabled.<br /> <br /> Steps to reproduce the bug:<br /> <br /> - Create new volume with image source.<br /> <br /> - Launch a new instance boot from the above volume using iSER transport.<br /> <br /> The bug is defined here:<br /> [https://bugs.launchpad.net/nova/+bug/1253578]<br /> <br /> A patch can be found here: [http://www.mellanox.com/downloads/solutions/openstack/havana/patches/volume.patch]<br /> <br /> To apply the patch, run the following commands on '''all nova-compute nodes''':<br /> # wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/volume.patch <br /> # patch -p1 &lt; volume.patch<br /> # service nova-compute restart <br /> <br /> (3) Flow control<br /> <br /> In case the network is Ethernet, make sure that flow control is enabled on the switches on the relevant ports across the network.<br /> <br /> (4) Linux-IO (LIO) support limitation.<br /> <br /> Open-source implementation of SCSI target in Openstack can be managed by either TGT or LIO.&lt;br /&gt;<br /> Currently in Openstack, iSER is supported over TGT only, and iSER support over LIO is scheduled for Kilo version.&lt;br /&gt;<br /> In case LIO is the default SCSI target, TGT must be installed and Openstack Cinder configuration file must be altered to use TGT instead of LIO.<br /> <br /> = References =<br /> 1. [http://www.mellanox.com/openstack/ OpenStack solution page at Mellanox site]<br /> <br /> 2. [https://github.com/mellanox-openstack Source repository]<br /> <br /> 3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED web page]<br /> <br /> 4. [http://docs.openstack.org/havana/install-guide/install/yum/content/cinder-controller.html Cinder-controller]<br /> <br /> 5. [http://docs.openstack.org/havana/install-guide/install/yum/content/cinder-node.html Cinder-node]<br /> <br /> Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.<br /> <br /> For more details, please refer any inquiries to [mailto:openstack@mellanox.com openstack@mellanox.com].</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Cinder&diff=136904 2016-10-31T12:54:26Z

<p>May: /* Known Issues */</p> <hr /> <div>= Overview = <br /> Mellanox added iSCSI Extensions for RDMA (iSER) that does not need the TCP layer. Therefore, it provides lower latency and higher performance.&lt;br /&gt;<br /> It permits data to be transferred directly into and out of SCSI computer memory buffers, which connects computers to storage devices, without intermediate data copies.&lt;br /&gt;<br /> iSER is a server network protocol that extends the Internet Small Computer System Interface (iSCSI) protocol to use Remote Direct Memory Access (RDMA).&lt;br /&gt;&lt;br /&gt;<br /> Please contact Mellanox for further information and benchmark results.<br /> <br /> == Installation ==<br /> Provisioned through standard Cinder installation.<br /> <br /> == Configuration ==<br /> In order to enable iSER, adjust these values at '''/etc/cinder/cinder.conf''':<br /> <br /> iscsi_ip_address = &lt;ipoib/roce_address&gt;<br /> iscsi_protocol = iser<br /> volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver<br /> <br /> '''iscsi_ip_address''' is required to do a &quot;discovery&quot; over the IB/RoCE interface from the initiator side.<br /> <br /> '''volume_driver''' points cinder to use the LVMVolumeDriver.<br /> <br /> Restart Cinder service<br /> <br /> #/etc/init.d/openstack-cinder-volume restart<br /> <br /> For troubleshooting issues, refer to section [https://wiki.openstack.org/wiki/Mellanox-Cinder#Known_Issues Known Issues]<br /> <br /> = Known Issues =<br /> <br /> (1) “scsi-target-utils” package <br /> <br /> '''Note''': For RH6.4 or below, “scsi-target-utils” package version needs to be 1.0.38 and above. If you use RDO installation, you should check the version.<br /> The RPM can be found at http://www.mellanox.com/downloads/solutions/rpms/<br /> <br /> #wget http://www.mellanox.com/downloads/solutions/rpms/scsi-target-utils-1.0.39-v1.0.39.c1135a.x86_64.rpm<br /> #rpm -Uvh scsi-target-utils-1.0.39-v1.0.39.c1135a.x86_64.rpm<br /> <br /> In addition, RH6.5 includes “scsi-target-utils” package with version 1.0.24-10. This version number is sufficient and no further action is needed for RHEL6.5.<br /> <br /> (2) Boot Instance from volume fails when iSER is enabled.<br /> <br /> Steps to reproduce the bug:<br /> <br /> - Create new volume with image source.<br /> <br /> - Launch a new instance boot from the above volume using iSER transport.<br /> <br /> The bug is defined here:<br /> [https://bugs.launchpad.net/nova/+bug/1253578]<br /> <br /> A patch can be found here: [http://www.mellanox.com/downloads/solutions/openstack/havana/patches/volume.patch]<br /> <br /> To apply the patch, run the following commands on '''all nova-compute nodes''':<br /> # wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/volume.patch <br /> # patch -p1 &lt; volume.patch<br /> # service nova-compute restart <br /> <br /> (3) Flow control<br /> <br /> In case the network is Ethernet, make sure that flow control is enabled on the switches on the relevant ports across the network.<br /> <br /> (4) Linux-IO (LIO) support limitation.<br /> <br /> Open-source implementation of SCSI target in Openstack can be managed by either TGT or LIO.&lt;br /&gt;<br /> Currently in Openstack, iSER is supported over TGT only, and iSER support over LIO is scheduled for Kilo version.&lt;br /&gt;<br /> In case LIO is the default SCSI target, one must one install TGT and alter Openstack Cinder configuration file to use TGT instead of LIO.<br /> <br /> = References =<br /> 1. [http://www.mellanox.com/openstack/ OpenStack solution page at Mellanox site]<br /> <br /> 2. [https://github.com/mellanox-openstack Source repository]<br /> <br /> 3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED web page]<br /> <br /> 4. [http://docs.openstack.org/havana/install-guide/install/yum/content/cinder-controller.html Cinder-controller]<br /> <br /> 5. [http://docs.openstack.org/havana/install-guide/install/yum/content/cinder-node.html Cinder-node]<br /> <br /> Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.<br /> <br /> For more details, please refer any inquiries to [mailto:openstack@mellanox.com openstack@mellanox.com].</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Cinder&diff=136903 2016-10-31T12:53:22Z

<p>May: /* Known Issues */</p> <hr /> <div>= Overview = <br /> Mellanox added iSCSI Extensions for RDMA (iSER) that does not need the TCP layer. Therefore, it provides lower latency and higher performance.&lt;br /&gt;<br /> It permits data to be transferred directly into and out of SCSI computer memory buffers, which connects computers to storage devices, without intermediate data copies.&lt;br /&gt;<br /> iSER is a server network protocol that extends the Internet Small Computer System Interface (iSCSI) protocol to use Remote Direct Memory Access (RDMA).&lt;br /&gt;&lt;br /&gt;<br /> Please contact Mellanox for further information and benchmark results.<br /> <br /> == Installation ==<br /> Provisioned through standard Cinder installation.<br /> <br /> == Configuration ==<br /> In order to enable iSER, adjust these values at '''/etc/cinder/cinder.conf''':<br /> <br /> iscsi_ip_address = &lt;ipoib/roce_address&gt;<br /> iscsi_protocol = iser<br /> volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver<br /> <br /> '''iscsi_ip_address''' is required to do a &quot;discovery&quot; over the IB/RoCE interface from the initiator side.<br /> <br /> '''volume_driver''' points cinder to use the LVMVolumeDriver.<br /> <br /> Restart Cinder service<br /> <br /> #/etc/init.d/openstack-cinder-volume restart<br /> <br /> For troubleshooting issues, refer to section [https://wiki.openstack.org/wiki/Mellanox-Cinder#Known_Issues Known Issues]<br /> <br /> = Known Issues =<br /> <br /> (1) “scsi-target-utils” package <br /> <br /> '''Note''': For RH6.4 or below, “scsi-target-utils” package version needs to be 1.0.38 and above. If you use RDO installation, you should check the version.<br /> The RPM can be found at http://www.mellanox.com/downloads/solutions/rpms/<br /> <br /> #wget http://www.mellanox.com/downloads/solutions/rpms/scsi-target-utils-1.0.39-v1.0.39.c1135a.x86_64.rpm<br /> #rpm -Uvh scsi-target-utils-1.0.39-v1.0.39.c1135a.x86_64.rpm<br /> <br /> In addition, RH6.5 includes “scsi-target-utils” package with version 1.0.24-10. This version number is sufficient and no further action is needed for RHEL6.5.<br /> <br /> (2) Boot Instance from volume fails when iSER is enabled.<br /> <br /> Steps to reproduce the bug:<br /> <br /> - Create new volume with image source.<br /> <br /> - Launch a new instance boot from the above volume using iSER transport.<br /> <br /> The bug is defined here:<br /> [https://bugs.launchpad.net/nova/+bug/1253578]<br /> <br /> A patch can be found here: [http://www.mellanox.com/downloads/solutions/openstack/havana/patches/volume.patch]<br /> <br /> To apply the patch, run the following commands on '''all nova-compute nodes''':<br /> # wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/volume.patch <br /> # patch -p1 &lt; volume.patch<br /> # service nova-compute restart <br /> <br /> (3) Flow control<br /> <br /> In case the network is Ethernet, make sure that flow control is enabled on the switches on the relevant ports across the network.<br /> <br /> (4) Linux-IO (LIO) support limitation.<br /> <br /> Open-source implementation of SCSI target in Openstack can be managed by either TGT or LIO.&lt;br /&gt;<br /> Currently in Openstack, iSER is supported over TGT only, and iSER support over LIO is scheduled for Kilo version.&lt;br /&gt;<br /> In case LIO is the default SCSI target, one must one install TGT and alter Openstack Cinder conf file to use TGT instead of lio.<br /> <br /> = References =<br /> 1. [http://www.mellanox.com/openstack/ OpenStack solution page at Mellanox site]<br /> <br /> 2. [https://github.com/mellanox-openstack Source repository]<br /> <br /> 3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED web page]<br /> <br /> 4. [http://docs.openstack.org/havana/install-guide/install/yum/content/cinder-controller.html Cinder-controller]<br /> <br /> 5. [http://docs.openstack.org/havana/install-guide/install/yum/content/cinder-node.html Cinder-node]<br /> <br /> Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.<br /> <br /> For more details, please refer any inquiries to [mailto:openstack@mellanox.com openstack@mellanox.com].</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Cinder&diff=136902 2016-10-31T12:05:43Z

<p>May: /* Known Issues */</p> <hr /> <div>= Overview = <br /> Mellanox added iSCSI Extensions for RDMA (iSER) that does not need the TCP layer. Therefore, it provides lower latency and higher performance.&lt;br /&gt;<br /> It permits data to be transferred directly into and out of SCSI computer memory buffers, which connects computers to storage devices, without intermediate data copies.&lt;br /&gt;<br /> iSER is a server network protocol that extends the Internet Small Computer System Interface (iSCSI) protocol to use Remote Direct Memory Access (RDMA).&lt;br /&gt;&lt;br /&gt;<br /> Please contact Mellanox for further information and benchmark results.<br /> <br /> == Installation ==<br /> Provisioned through standard Cinder installation.<br /> <br /> == Configuration ==<br /> In order to enable iSER, adjust these values at '''/etc/cinder/cinder.conf''':<br /> <br /> iscsi_ip_address = &lt;ipoib/roce_address&gt;<br /> iscsi_protocol = iser<br /> volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver<br /> <br /> '''iscsi_ip_address''' is required to do a &quot;discovery&quot; over the IB/RoCE interface from the initiator side.<br /> <br /> '''volume_driver''' points cinder to use the LVMVolumeDriver.<br /> <br /> Restart Cinder service<br /> <br /> #/etc/init.d/openstack-cinder-volume restart<br /> <br /> For troubleshooting issues, refer to section [https://wiki.openstack.org/wiki/Mellanox-Cinder#Known_Issues Known Issues]<br /> <br /> = Known Issues =<br /> <br /> (1) “scsi-target-utils” package <br /> <br /> '''Note''': For RH6.4 or below: “scsi-target-utils” package needs to be 1.0.38 and up. If you use RDO installation you should check the version.<br /> The RPM can be found in http://www.mellanox.com/downloads/solutions/rpms/<br /> <br /> #wget http://www.mellanox.com/downloads/solutions/rpms/scsi-target-utils-1.0.39-v1.0.39.c1135a.x86_64.rpm<br /> #rpm -Uvh scsi-target-utils-1.0.39-v1.0.39.c1135a.x86_64.rpm<br /> <br /> In addition, RH6.5 has “scsi-target-utils” package with version 1.0.24-10, which is good enough, no further action is needed for RHEL6.5<br /> <br /> (2) Boot Instance from volume fails when iSER is enabled<br /> <br /> Steps to reproduce the bug:<br /> <br /> - Create new volume with image source.<br /> <br /> - Launch a new instance boot from the above volume using iSER transport.<br /> <br /> The bug is defined here:<br /> [https://bugs.launchpad.net/nova/+bug/1253578]<br /> <br /> A patch can be found in here: [http://www.mellanox.com/downloads/solutions/openstack/havana/patches/volume.patch]<br /> <br /> To apply the patch run the following commands on '''all nova-compute nodes''':<br /> # wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/volume.patch <br /> # patch -p1 &lt; volume.patch<br /> # service nova-compute restart <br /> <br /> (3) Flow control<br /> <br /> In case the network is Ethernet, Make sure that flow control is enabled on the switches on the relevant ports across the network.<br /> <br /> (4) lio support limitation.<br /> <br /> Open-source implementation of SCSI target in Openstack can be managed by either TGT or LIO.&lt;br /&gt;<br /> Currently in Openstack, iSER is supported over TGT only, and iSER support over LIO is scheduled for Kilo version.&lt;br /&gt;<br /> In case LIO is the default SCSI target, one must one install TGT and alter Openstack Cinder conf file to use TGT instead of lio.<br /> <br /> = References =<br /> 1. [http://www.mellanox.com/openstack/ OpenStack solution page at Mellanox site]<br /> <br /> 2. [https://github.com/mellanox-openstack Source repository]<br /> <br /> 3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED web page]<br /> <br /> 4. [http://docs.openstack.org/havana/install-guide/install/yum/content/cinder-controller.html Cinder-controller]<br /> <br /> 5. [http://docs.openstack.org/havana/install-guide/install/yum/content/cinder-node.html Cinder-node]<br /> <br /> Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.<br /> <br /> For more details, please refer any inquiries to [mailto:openstack@mellanox.com openstack@mellanox.com].</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Cinder&diff=136901 2016-10-31T11:47:25Z

<p>May: /* Overview */</p> <hr /> <div>= Overview = <br /> Mellanox added iSCSI Extensions for RDMA (iSER) that does not need the TCP layer. Therefore, it provides lower latency and higher performance.&lt;br /&gt;<br /> It permits data to be transferred directly into and out of SCSI computer memory buffers, which connects computers to storage devices, without intermediate data copies.&lt;br /&gt;<br /> iSER is a server network protocol that extends the Internet Small Computer System Interface (iSCSI) protocol to use Remote Direct Memory Access (RDMA).&lt;br /&gt;&lt;br /&gt;<br /> Please contact Mellanox for further information and benchmark results.<br /> <br /> == Installation ==<br /> Provisioned through standard Cinder installation.<br /> <br /> == Configuration ==<br /> In order to enable iSER, adjust these values at '''/etc/cinder/cinder.conf''':<br /> <br /> iscsi_ip_address = &lt;ipoib/roce_address&gt;<br /> iscsi_protocol = iser<br /> volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver<br /> <br /> '''iscsi_ip_address''' is required to do a &quot;discovery&quot; over the IB/RoCE interface from the initiator side.<br /> <br /> '''volume_driver''' points cinder to use the LVMVolumeDriver.<br /> <br /> Restart Cinder service<br /> <br /> #/etc/init.d/openstack-cinder-volume restart<br /> <br /> For troubleshooting issues, refer to section [https://wiki.openstack.org/wiki/Mellanox-Cinder#Known_Issues Known Issues]<br /> <br /> = Known Issues =<br /> <br /> (1) “scsi-target-utils” package <br /> <br /> Note: for RH6.4 or below: “scsi-target-utils” package needs to be 1.0.38 and up, if you use RDO installation you should check the version.<br /> The RPM can be found in http://www.mellanox.com/downloads/solutions/rpms/<br /> <br /> #wget http://www.mellanox.com/downloads/solutions/rpms/scsi-target-utils-1.0.39-v1.0.39.c1135a.x86_64.rpm<br /> #rpm -Uvh scsi-target-utils-1.0.39-v1.0.39.c1135a.x86_64.rpm<br /> <br /> In addition, RH6.5 has “scsi-target-utils” package with version 1.0.24-10, which is good enough, no further action is needed for RHEL6.5<br /> <br /> (2) Boot Instance from volume fails when iSER is enabled<br /> <br /> Steps to reproduce the bug:<br /> <br /> - Create new volume with image source.<br /> <br /> - Launch a new instance boot from the above volume using iSER transport.<br /> <br /> The bug is defined here:<br /> [https://bugs.launchpad.net/nova/+bug/1253578]<br /> <br /> A patch can be found in here: [http://www.mellanox.com/downloads/solutions/openstack/havana/patches/volume.patch]<br /> <br /> To apply the patch run the following commands on '''all nova-compute nodes''':<br /> # wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/volume.patch <br /> # patch -p1 &lt; volume.patch<br /> # service nova-compute restart <br /> <br /> (3) Flow control<br /> <br /> In case the network is Ethernet, Make sure that flow control is enabled on the switches on the relevant ports across the network.<br /> <br /> (4) lio support limitation.<br /> <br /> Open-source implementation of SCSI target in Openstack can be managed by either TGT or LIO.&lt;br /&gt;<br /> Currently in Openstack, iSER is supported over TGT only, and iSER support over LIO is scheduled for Kilo version.&lt;br /&gt;<br /> In case LIO is the default SCSI target, one must one install TGT and alter Openstack Cinder conf file to use TGT instead of lio. <br /> <br /> = References =<br /> 1. [http://www.mellanox.com/openstack/ OpenStack solution page at Mellanox site]<br /> <br /> 2. [https://github.com/mellanox-openstack Source repository]<br /> <br /> 3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED web page]<br /> <br /> 4. [http://docs.openstack.org/havana/install-guide/install/yum/content/cinder-controller.html Cinder-controller]<br /> <br /> 5. [http://docs.openstack.org/havana/install-guide/install/yum/content/cinder-node.html Cinder-node]<br /> <br /> Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.<br /> <br /> For more details, please refer any inquiries to [mailto:openstack@mellanox.com openstack@mellanox.com].</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Newton-InfiniBand&diff=136900 2016-10-31T09:53:52Z

<p>May: </p> <hr /> <div>Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API. <br /> <br /> This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA.<br /> Mellanox ML2 Mechanism Driver provides a functional parity with Mellanox Neutron plugin.<br /> <br /> Mellanox ML2 Mechanism Driver supports DIRECT (PCI passthrough) vNIC type. For vNIC type configuration API details, please refer to configuration reference guide.<br /> Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as Remote Direct Memory Access (RDMA). <br /> <br /> The driver supports VLAN network type to facilitate virtual networks either on Ethernet or on InfiniBand fabrics. <br /> * Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node. <br /> * Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and embedded switch port.<br /> <br /> <br /> '''Prerequisites'''<br /> * Mellanox ConnectX®-3/Mellanox ConnectX®-3 PRO [http://www.mellanox.com/page/products_dyn?product_family=119 Network Adapter] with [http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers Mellanox OFED 2.4 or greater]<br /> * Mellanox ConnectX®-4 [http://www.mellanox.com/page/products_dyn?product_family=201&amp; Network Adapter] with [http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers MLNX_OFED_LINUX-3.1-1.5.7 or greater].<br /> <br /> <br /> For '''Newton''' release Information refer to the relevant OS as follows:<br /> <br /> * [[Mellanox-Neutron-Newton-Redhat-InfiniBand | Redhat7]]<br /> * [[Mellanox-Neutron-Newton-Ubuntu-InfiniBand | Ubuntu14.04]]<br /> &lt;br /&gt;</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Mitaka-InfiniBand&diff=136899 2016-10-31T09:53:21Z

<p>May: </p> <hr /> <div>Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API. <br /> <br /> This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides a functional parity with Mellanox Neutron plugin. <br /> <br /> Mellanox ML2 Mechanism Driver supports DIRECT (PCI passthrough) vNIC type. Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as Remote Direct Memory Access (RDMA) or [http://docs.openstack.org/liberty/networking-guide/adv-config-sriov.html SR-IOV]<br /> <br /> The driver supports VLAN network type to facilitate virtual networks either on Ethernet or on InfiniBand fabrics. <br /> * Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node. <br /> * Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and an embedded switch port.<br /> <br /> <br /> '''Prerequisites'''<br /> * Mellanox ConnectX®-3/Mellanox ConnectX®-3 PRO [http://www.mellanox.com/page/products_dyn?product_family=119 Network Adapter] with [http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers Mellanox OFED 2.4] or greater.<br /> * Mellanox ConnectX®-4 [http://www.mellanox.com/page/products_dyn?product_family=201&amp; Network Adapter] with [[http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers Mellanox OFED 3.1-1.5.7] or greater.<br /> <br /> <br /> For '''Mitaka''' release Information refer to the relevant OS as follows:<br /> <br /> * [[Mellanox-Neutron-Mitaka-Redhat-InfiniBand | Redhat7]]<br /> * [[Mellanox-Neutron-Mitaka-Ubuntu-InfiniBand | Ubuntu14.04]] ( Under review )<br /> * [[Mellanox-Neutron-Mitaka-Ubuntu-InfiniBand-16 | Ubuntu 16.04 LTS]] ( Under review )<br /> &lt;br /&gt;</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Liberty-InfiniBand&diff=136898 2016-10-31T09:52:51Z

<p>May: </p> <hr /> <div><br /> Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API. <br /> <br /> This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides a functional parity with Mellanox Neutron plugin. <br /> <br /> Mellanox ML2 Mechanism Driver supports DIRECT (PCI passthrough) vNIC type. For vNIC type configuration API details, please refer to configuration reference guide (click here). Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as Remote Direct Memory Access (RDMA).<br /> <br /> The driver supports VLAN network type to facilitate virtual networks either on Ethernet or on InfiniBand fabrics. <br /> * Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node. <br /> * Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and an embedded switch port.<br /> <br /> <br /> '''Prerequisites'''<br /> * Mellanox ConnectX®-3/Mellanox ConnectX®-3 PRO [http://www.mellanox.com/page/products_dyn?product_family=119 Network Adapter] with Mellanox OFED 2.4 or greater<br /> * Mellanox ConnectX®-4 [http://www.mellanox.com/page/products_dyn?product_family=201&amp; Network Adapter] with MLNX_OFED_LINUX-3.1-1.5.7 and greater.<br /> * [http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers Mellanox OFED download page] <br /> <br /> <br /> For '''Liberty''' release Information refer to the relevant OS as follows:<br /> <br /> * [[Mellanox-Neutron-Liberty-Redhat-InfiniBand | Redhat7]]<br /> * [[Mellanox-Neutron-Liberty-Ubuntu-InfiniBand | Ubuntu14.04]]<br /> &lt;br /&gt;</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Mitaka-InfiniBand&diff=136897 2016-10-31T09:50:45Z

<p>May: </p> <hr /> <div>Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API. <br /> <br /> This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA. Mellanox ML2 Mechanism Driver provides a functional parity with Mellanox Neutron plugin. <br /> <br /> Mellanox ML2 Mechanism Driver supports DIRECT (PCI passthrough) vNIC type. Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as Remote Direct Memory Access (RDMA) or [http://docs.openstack.org/liberty/networking-guide/adv-config-sriov.html SR-IOV]<br /> <br /> The driver supports VLAN network type to facilitate virtual networks either on Ethernet or on InfiniBand fabrics. <br /> * Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node. <br /> * Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and an Embedded Switch port.<br /> <br /> <br /> '''Prerequisites'''<br /> * Mellanox ConnectX®-3/Mellanox ConnectX®-3 PRO [http://www.mellanox.com/page/products_dyn?product_family=119 Network Adapter] with [http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers Mellanox OFED 2.4] or greater.<br /> * Mellanox ConnectX®-4 [http://www.mellanox.com/page/products_dyn?product_family=201&amp; Network Adapter] with [[http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers Mellanox OFED 3.1-1.5.7] or greater.<br /> <br /> <br /> For '''Mitaka''' release Information refer to the relevant OS as follows:<br /> <br /> * [[Mellanox-Neutron-Mitaka-Redhat-InfiniBand | Redhat7]]<br /> * [[Mellanox-Neutron-Mitaka-Ubuntu-InfiniBand | Ubuntu14.04]] ( Under review )<br /> * [[Mellanox-Neutron-Mitaka-Ubuntu-InfiniBand-16 | Ubuntu 16.04 LTS]] ( Under review )<br /> &lt;br /&gt;</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Newton-InfiniBand&diff=136894 2016-10-31T08:43:48Z

<p>May: </p> <hr /> <div>Mellanox ML2 Mechanism Driver implements the ML2 Plugin Mechanism Driver API. <br /> <br /> This driver supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA.<br /> Mellanox ML2 Mechanism Driver provides a functional parity with Mellanox Neutron plugin.<br /> <br /> Mellanox ML2 Mechanism Driver supports DIRECT (PCI passthrough) vNIC type. For vNIC type configuration API details, please refer to configuration reference guide.<br /> Hardware vNICs mapped to the guest VMs allow higher performance and advanced features such as Remote Direct Memory Access (RDMA). <br /> <br /> The driver supports VLAN network type to facilitate virtual networks either on Ethernet or on InfiniBand fabrics. <br /> * Mellanox OpenStack Neutron Agent (L2 Agent) runs on each compute node. <br /> * Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.<br /> <br /> <br /> '''Prerequisites'''<br /> * Mellanox ConnectX®-3/Mellanox ConnectX®-3 PRO [http://www.mellanox.com/page/products_dyn?product_family=119 Network Adapter] with [http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers Mellanox OFED 2.4 or greater]<br /> * Mellanox ConnectX®-4 [http://www.mellanox.com/page/products_dyn?product_family=201&amp; Network Adapter] with [http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers MLNX_OFED_LINUX-3.1-1.5.7 or greater].<br /> <br /> <br /> For '''Newton''' release Information refer to the relevant OS as follows:<br /> <br /> * [[Mellanox-Neutron-Newton-Redhat-InfiniBand | Redhat7]]<br /> * [[Mellanox-Neutron-Newton-Ubuntu-InfiniBand | Ubuntu14.04]]<br /> &lt;br /&gt;</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Liberty-Ethernet&diff=136878 2016-10-30T13:40:06Z

<p>May: /* SDN Mechanism Driver */</p> <hr /> <div><br /> ==Configuration==<br /> [https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking Configure SR-IOV]<br /> <br /> ==SDN Mechanism Driver==<br /> [https://community.mellanox.com/docs/DOC-2155 SDN Mechanism Driver] allows NEO to dynamically configure VLANs on the Mellanox Switches.<br /> Further reading:<br /> * [http://www.mellanox.com/page/products_dyn?product_family=220&amp;mtag=mellanox_neo NEO]<br /> * [http://www.mellanox.com/page/ethernet_switch_overview Mellanox Switches]<br /> * [https://community.mellanox.com/docs/DOC-2251 SDN Mechanism Driver]<br /> <br /> ==Ceilometer==<br /> The Mellanox Ceilometer SR-IOV module allows the OpenStack Ceilometer to collect measurements of SR-IOV counters.<br /> <br /> More details about applying Mellanox Ceilometer module can be found [https://community.mellanox.com/docs/DOC-2145 here]</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Newton-Ethernet&diff=136877 2016-10-30T13:38:07Z

<p>May: /* SDN Mechanism Driver */</p> <hr /> <div><br /> ==Configuration==<br /> [https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking Configure SR-IOV]<br /> <br /> ==SDN Mechanism Driver==<br /> [https://community.mellanox.com/docs/DOC-2155 SDN Mechanism Driver] allows NEO to dynamically configure VLANs on the Mellanox Switches.<br /> Further reading:<br /> * [http://www.mellanox.com/page/products_dyn?product_family=220&amp;mtag=mellanox_neo NEO]<br /> * [http://www.mellanox.com/page/ethernet_switch_overview Mellanox Switches]<br /> * [https://community.mellanox.com/docs/DOC-2251 SDN Mechanism Driver]<br /> <br /> ==Ceilometer==<br /> The Mellanox Ceilometer SR-IOV module allows the OpenStack Ceilometer to collect measurements of SR-IOV counters.<br /> <br /> More details about applying Mellanox Ceilometer module can be found [https://community.mellanox.com/docs/DOC-2145 here]</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Newton-Ethernet&diff=136876 2016-10-30T11:59:54Z

<p>May: /* SDN Mechanism Driver */</p> <hr /> <div><br /> ==Configuration==<br /> [https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking Configure SR-IOV]<br /> <br /> ==SDN Mechanism Driver==<br /> [https://community.mellanox.com/docs/DOC-2155 SDN Mechanism Driver] allows NEO to dynamically configure VLANs on the Mellanox Switches.<br /> Further Reading<br /> * [http://www.mellanox.com/page/products_dyn?product_family=220&amp;mtag=mellanox_neo NEO]<br /> * [http://www.mellanox.com/page/ethernet_switch_overview Mellanox Switches]<br /> * [https://community.mellanox.com/docs/DOC-2251 SDN Mechanism Driver]<br /> <br /> ==Ceilometer==<br /> The Mellanox Ceilometer SR-IOV module allows the OpenStack Ceilometer to collect measurements of SR-IOV counters.<br /> <br /> More details about applying Mellanox Ceilometer module can be found [https://community.mellanox.com/docs/DOC-2145 here]</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-OpenStack&diff=136875 2016-10-30T11:55:45Z

<p>May: /* Mellanox Neutron Plugin */</p> <hr /> <div><br /> = Mellanox Neutron Plugin =<br /> Mellanox supports the OpenStack Neutron releases with open source networking components through Neutron. It delivers higher computing and storage performance and additional functionalities, such as NIC-based switching to provide better security and isolation for virtual cloud environments.<br /> <br /> === Ethernet SR-IOV ===<br /> <br /> {| class=&quot;wikitable&quot;<br /> |-<br /> ! Ocata !! Newton !! Mitaka !! Liberty !! Kilo !! Juno<br /> |-<br /> | || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Newton-Ethernet SR-IOV] || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Mitaka-Ethernet SR-IOV] || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Liberty-Ethernet SR-IOV] || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Kilo-Ethernet SR-IOV] || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Juno-Ethernet SR-IOV]<br /> |-<br /> | || [http://docs.openstack.org/liberty/networking-guide/adv-config-qos.html QOS] || || || || <br /> |-<br /> | || [https://community.mellanox.com/docs/DOC-2612 VF LAG] || || || || <br /> |-<br /> | || [https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking#FDB_L2_Agent_Extension FDB] || || || || <br /> |-<br /> <br /> |}<br /> <br /> ===Infiniband SR-IOV===<br /> <br /> <br /> <br /> <br /> {| class=&quot;wikitable&quot;<br /> |-<br /> ! Ocata !! Newton !! Mitaka !! Liberty !! Kilo !! Juno<br /> |-<br /> | || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Newton-InfiniBand SR-IOV] || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Mitaka-InfiniBand SR-IOV] || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Liberty-InfiniBand SR-IOV] || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Kilo-InfiniBand SR-IOV] || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Juno-InfiniBand SR-IOV] <br /> |-<br /> | || [https://community.mellanox.com/docs/DOC-2155 NEO SDN Mechanism] || [https://community.mellanox.com/docs/DOC-2155 NEO SDN Mechanism] || [https://community.mellanox.com/docs/DOC-2155 NEO SDN Mechanism] |||| <br /> |-<br /> |}<br /> <br /> = Mellanox Cinder Plugin =<br /> Mellanox provides an OpenStack Storage plugin that enables running storage volume traffic over native RDMA transport, providing five-fold faster throughput over standard TCP/IP, while significantly increasing infrastructure efficiency by using less CPU resources.<br /> <br /> For additional information about Mellanox Cinder, refer to [https://wiki.openstack.org/wiki/Mellanox-Cinder Mellanox Cinder] wiki page.<br /> <br /> = Mellanox Ironic =<br /> <br /> TBD<br /> <br /> = References =<br /> 1. [http://www.mellanox.com/openstack/ OpenStack solution page at Mellanox site]<br /> <br /> 2. [https://github.com/mellanox-openstack Source repository]<br /> <br /> 3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED web page]<br /> <br /> 4. [http://community.mellanox.com/community/solutions Mellanox Community]<br /> <br /> 5. [https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking SR-IOV Passthrough for Networking]<br /> <br /> For more details, please refer your question to [mailto:support@mellanox.com support@mellanox.com]</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-OpenStack&diff=136874 2016-10-30T11:54:06Z

<p>May: /* Mellanox Cinder Plugin */</p> <hr /> <div><br /> = Mellanox Neutron Plugin =<br /> Mellanox supports the OpenStack Neutron releases with open source networking components through Neutron. It delivers higher compute and storage performance and additional functionalities, such as NIC based switching to provide better security and isolation for virtual cloud environments.<br /> <br /> === Ethernet SR-IOV ===<br /> <br /> {| class=&quot;wikitable&quot;<br /> |-<br /> ! Ocata !! Newton !! Mitaka !! Liberty !! Kilo !! Juno<br /> |-<br /> | || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Newton-Ethernet SR-IOV] || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Mitaka-Ethernet SR-IOV] || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Liberty-Ethernet SR-IOV] || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Kilo-Ethernet SR-IOV] || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Juno-Ethernet SR-IOV]<br /> |-<br /> | || [http://docs.openstack.org/liberty/networking-guide/adv-config-qos.html QOS] || || || || <br /> |-<br /> | || [https://community.mellanox.com/docs/DOC-2612 VF LAG] || || || || <br /> |-<br /> | || [https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking#FDB_L2_Agent_Extension FDB] || || || || <br /> |-<br /> <br /> |}<br /> <br /> ===Infiniband SR-IOV===<br /> <br /> <br /> <br /> <br /> {| class=&quot;wikitable&quot;<br /> |-<br /> ! Ocata !! Newton !! Mitaka !! Liberty !! Kilo !! Juno<br /> |-<br /> | || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Newton-InfiniBand SR-IOV] || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Mitaka-InfiniBand SR-IOV] || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Liberty-InfiniBand SR-IOV] || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Kilo-InfiniBand SR-IOV] || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Juno-InfiniBand SR-IOV] <br /> |-<br /> | || [https://community.mellanox.com/docs/DOC-2155 NEO SDN Mechanism] || [https://community.mellanox.com/docs/DOC-2155 NEO SDN Mechanism] || [https://community.mellanox.com/docs/DOC-2155 NEO SDN Mechanism] |||| <br /> |-<br /> |}<br /> <br /> = Mellanox Cinder Plugin =<br /> Mellanox provides an OpenStack Storage plugin that enables running storage volume traffic over native RDMA transport, providing five-fold faster throughput over standard TCP/IP, while significantly increasing infrastructure efficiency by using less CPU resources.<br /> <br /> For additional information about Mellanox Cinder, refer to [https://wiki.openstack.org/wiki/Mellanox-Cinder Mellanox Cinder] wiki page.<br /> <br /> = Mellanox Ironic =<br /> <br /> TBD<br /> <br /> = References =<br /> 1. [http://www.mellanox.com/openstack/ OpenStack solution page at Mellanox site]<br /> <br /> 2. [https://github.com/mellanox-openstack Source repository]<br /> <br /> 3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED web page]<br /> <br /> 4. [http://community.mellanox.com/community/solutions Mellanox Community]<br /> <br /> 5. [https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking SR-IOV Passthrough for Networking]<br /> <br /> For more details, please refer your question to [mailto:support@mellanox.com support@mellanox.com]</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-OpenStack&diff=136873 2016-10-30T11:34:33Z

<p>May: /* Mellanox Neutron Plugin */</p> <hr /> <div><br /> = Mellanox Neutron Plugin =<br /> Mellanox supports the OpenStack Neutron releases with open source networking components through Neutron. It delivers higher compute and storage performance and additional functionalities, such as NIC based switching to provide better security and isolation for virtual cloud environments.<br /> <br /> === Ethernet SR-IOV ===<br /> <br /> {| class=&quot;wikitable&quot;<br /> |-<br /> ! Ocata !! Newton !! Mitaka !! Liberty !! Kilo !! Juno<br /> |-<br /> | || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Newton-Ethernet SR-IOV] || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Mitaka-Ethernet SR-IOV] || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Liberty-Ethernet SR-IOV] || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Kilo-Ethernet SR-IOV] || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Juno-Ethernet SR-IOV]<br /> |-<br /> | || [http://docs.openstack.org/liberty/networking-guide/adv-config-qos.html QOS] || || || || <br /> |-<br /> | || [https://community.mellanox.com/docs/DOC-2612 VF LAG] || || || || <br /> |-<br /> | || [https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking#FDB_L2_Agent_Extension FDB] || || || || <br /> |-<br /> <br /> |}<br /> <br /> ===Infiniband SR-IOV===<br /> <br /> <br /> <br /> <br /> {| class=&quot;wikitable&quot;<br /> |-<br /> ! Ocata !! Newton !! Mitaka !! Liberty !! Kilo !! Juno<br /> |-<br /> | || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Newton-InfiniBand SR-IOV] || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Mitaka-InfiniBand SR-IOV] || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Liberty-InfiniBand SR-IOV] || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Kilo-InfiniBand SR-IOV] || [https://wiki.openstack.org/wiki/Mellanox-Neutron-Juno-InfiniBand SR-IOV] <br /> |-<br /> | || [https://community.mellanox.com/docs/DOC-2155 NEO SDN Mechanism] || [https://community.mellanox.com/docs/DOC-2155 NEO SDN Mechanism] || [https://community.mellanox.com/docs/DOC-2155 NEO SDN Mechanism] |||| <br /> |-<br /> |}<br /> <br /> = Mellanox Cinder Plugin =<br /> Mellanox provides an OpenStack Storage plugin that enables running storage volume traffic over native RDMA transport, providing 5X faster throughput over standard TCP/IP, while significantly increasing infrastructure efficiency by using less CPU resources.<br /> <br /> For additional information about Mellanox Cinder refer to [https://wiki.openstack.org/wiki/Mellanox-Cinder Mellanox Cinder] wiki page.<br /> <br /> = Mellanox Ironic =<br /> <br /> TBD<br /> <br /> = References =<br /> 1. [http://www.mellanox.com/openstack/ OpenStack solution page at Mellanox site]<br /> <br /> 2. [https://github.com/mellanox-openstack Source repository]<br /> <br /> 3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED web page]<br /> <br /> 4. [http://community.mellanox.com/community/solutions Mellanox Community]<br /> <br /> 5. [https://wiki.openstack.org/wiki/SR-IOV-Passthrough-For-Networking SR-IOV Passthrough for Networking]<br /> <br /> For more details, please refer your question to [mailto:support@mellanox.com support@mellanox.com]</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand&diff=69188 2014-11-30T08:37:56Z

<p>May: /* Compute Nodes */</p> <hr /> <div>= Overview = <br /> <br /> This wiki page describes how to tune Mellanox OpenStack on Ubuntu 12.04 servers<br /> <br /> = Prerequisites =<br /> <br /> * A running OpenStack environment installed with the ML2 plugin on top of Linux Bridge.<br /> * All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119) <br /> * Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers<br /> * SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community [http://community.mellanox.com/docs/DOC-1317 click here].<br /> * The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes<br /> * VLANs configured on the ports in the switch.<br /> <br /> = InfiniBand Network = <br /> <br /> == OpenSM Node ==<br /> <br /> To configure OpenSM node: <br /> <br /> 1. Edit the file /etc/opensm/partitions.conf by adding the following line:<br /> management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;<br /> Create the file if it does not exist. <br /> <br /> For each VLAN value in the VLAN range used for VLAN network segmentation, add the following line in the file /etc/opensm/partitions.conf (for vlan-pkey association):<br /> vlan&lt;decimal-value&gt;=0x&lt;hexadecimal-value&gt;, ipoib, sl=0, defmember=full: ALL;<br /> In this manual, the VLAN range is 2-10 (mlnx_agent.conf /ml2_conf.ini). Therefore, add the following lines:<br /> vlan1=0x1, ipoib, sl=0, defmember=full : ALL;<br /> vlan2=0x2, ipoib, sl=0, defmember=full : ALL;<br /> vlan3=0x3, ipoib, sl=0, defmember=full : ALL;<br /> vlan4=0x4, ipoib, sl=0, defmember=full : ALL;<br /> vlan5=0x5, ipoib, sl=0, defmember=full : ALL;<br /> vlan6=0x6, ipoib, sl=0, defmember=full : ALL;<br /> vlan7=0x7, ipoib, sl=0, defmember=full : ALL;<br /> vlan8=0x8, ipoib, sl=0, defmember=full : ALL;<br /> vlan9=0x9, ipoib, sl=0, defmember=full : ALL;<br /> vlan10=0xa, ipoib, sl=0, defmember=full : ALL;<br /> 2. Modify the following line in the file /etc/opensm/opensm.conf from FALSE to TRUE:<br /> allow_both_pkeys TRUE<br /> 3. Restart OpenSM:<br /> service opensmd restart<br /> 4. Verify that all ports on all nodes have their ib interface (i.e. ib0) active, run ibdev2netdev on all hosts, and make sure that the port is up:<br /> mlx4_0 port 1 ==&gt; ib0 (Up)<br /> If the port is down, try bringing it up by running “ifconfig ib0 up”, and make sure your switch ports are of type “InfiniBand”, and that all nodes can reach the OpenSM node by running “sminfo”.<br /> <br /> == Controller Node == <br /> <br /> To configure the Controller node: <br /> <br /> 1. Ensure that the file /etc/nova/nova.conf includes the following line:<br /> security_group_api=nova<br /> <br /> 2. If the security group value is changed, restart the Nova services:<br /> # for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done<br /> <br /> == Neutron Server Node ==<br /> <br /> NOTE: The Neutron Server can sometimes be part of the Controller node.<br /> <br /> To configure the Neutron Server node: <br /> <br /> 1. Edit /etc/neutron/neutron.conf and make sure Neutron is using ML2 plugin:<br /> [DEFAULT]<br /> core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin<br /> 2. Edit the ML2 configuration file /etc/neutron/plugins/ml2/ml2_conf.ini as follows:<br /> [ml2]<br /> type_drivers = vlan,flat<br /> tenant_network_types = vlan <br /> mechanism_drivers = linuxbridge,mlnx<br /> <br /> [ml2_type_vlan]<br /> network_vlan_ranges=&lt;physical_network&gt;:&lt;min_vlan&gt;:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> <br /> [securitygroup]<br /> enable_security_group = True<br /> <br /> [eswitch]<br /> vnic_type = hostdev<br /> apply_profile_patch = True<br /> 3. Install the required Deb for the Mellanox Neutron plugin (from Ubuntu’s repository):<br /> apt-get install neutron-plugin-mlnx-agent<br /> 4. Edit /etc/neutron/plugins/mlnx/mlnx_conf.ini to match your environment:<br /> [mlnx]<br /> network_vlan_ranges = &lt;physical_network&gt;:&lt;min_vlan:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> tenant_network_type = vlan<br /> <br /> [eswitch]<br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;ib interface&gt; //(i.e. default:ib0)<br /> vnic_type = hostdev<br /> <br /> [agent]<br /> support_old_agents = True<br /> <br /> 5. Restart Neutron Server: <br /> /etc/init.d/neutron-server start<br /> <br /> == Compute Nodes ==<br /> <br /> To configure the Compute node:<br /> <br /> 1. Create the file /etc/modprobe.d/mlx4_ib.conf with this line:<br /> options mlx4_ib sm_guid_assign=0<br /> 2. Download mlnxvif, eSwitch Daemon (eSwitchd), and mlnx-dnsmasq from this link:<br /> http://www.mellanox.com/downloads/solutions/openstack/icehouse/debian/<br /> 3. Install the Mellanox plugin services mlnxvif and eSwitchd:<br /> dpkg –i python-mlnxvif_2014.1.1-1_all.deb<br /> dpkg –i python-eswitchd_0.10-1_all.deb<br /> 4. Edit /etc/eswitchd/eswitchd.conf as follows:<br /> [DAEMON]<br /> fabrics=&lt;physical_network&gt;:&lt;ib-interface&gt; //(i.e. default:ib0)<br /> 5. Edit /etc/nova/nova.conf as follows:<br /> [DEFAULT]<br /> compute_driver = libvirt.LibvirtDriver<br /> libvirt_vif_driver = mlnxvif.vif.MlxEthVIFDriver<br /> security_group_api = nova<br /> 6. Check if “ebrctl” appears in one of Nova’s rootwrap files:<br /> grep 'ebrctl' /etc/nova/rootwrap.d/*<br /> If “ebrctl” does not appear in one of Nova’s rootwrap files, edit the file /etc/nova/rootwrap.d/compute.filter as follows:<br /> [Filters]<br /> ebrctl: CommandFilter, ebrctl, root<br /> 7. Install the Neutron Mellanox agent (from Ubuntu repository):<br /> apt-get install neutron-plugin-mlnx-agent<br /> 8. Copy /etc/neutron/plugins/mlnx/mlnx_conf.ini from the Neutron Server node to the same path on each Compute node.<br /> <br /> 9. Restart the following services:<br /> /etc/init.d/nova-compute restart<br /> /etc/init.d/eswitchd start<br /> /etc/init.d/neutron-plugin-mlnx-agent start<br /> <br /> == Network Node ==<br /> <br /> To configure the Network node: <br /> <br /> 1. Configure eIPoIB (for Linux bridge plugin) by editing the file /etc/infiniband/openib.conf as follows:<br /> E_IPOIB_LOAD=yes<br /> 2. Restart OFED: <br /> service openibd restart<br /> 3. Verify that the new Ethernet interfaces now appear with ibdev2netdev:<br /> # ibdev2netdev<br /> mlx4_0 port 1 ==&gt; ib0 (Up)<br /> mlx4_0 port 2 ==&gt; ib1 (Down)<br /> mlx4_0 port 1 ==&gt; eth2 (Up)<br /> mlx4_0 port 2 ==&gt; eth1 (Down)<br /> In eIPoIB, every ib interface has a matching eth interface. The ib interface used here is ib0, and the matching one is eth2 (same card and port names: mlx4_0 port 1). By using Ethtool, you can verify that ib0 is working on ib_ipoib driver, and eth2 is working on eth_ipoib driver, as follows:<br /> # ethtool -i ib0<br /> driver: ib_ipoib<br /> version: 2.3-1.5.4_Sep 30 2014<br /> firmware-version: 2.32.5100<br /> bus-info: 0000:07:00.0# ethtool -i eth2<br /> driver: eth_ipoib<br /> version: 2.3-1.5.4 (Sep 30 2014)<br /> firmware-version: 1<br /> bus-info: ib0<br /> 4. Edit /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows:<br /> [linux_bridge] <br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;eth_ipoib-interface&gt; <br /> //(i.e. default:eth2)<br /> 5. Restart the Linux bridge agent:<br /> service neutron-linuxbridge-agent restart<br /> 6. To allow DHCP support, download mlnx-dnsmadq from this link: http://www.mellanox.com/downloads/solutions/openstack/icehouse/debian/<br /> <br /> 7. Install the package:<br /> dpkg –i python-mlnx-dnsmasq_2014.1.1-1_all.deb<br /> 8. Configure the file /etc/neutron/dhcp_agent.ini:<br /> [DEFAULT]<br /> dhcp_driver = mlnx_dhcp.MlnxDnsmasq<br /> 9. Make sure that the Dnsmasq which is installed as a part of the OpenStack (not mlnx-dnsmasq) environment is of version 2.65.<br /> <br /> 10. Copy /etc/neutron/neutron.conf from the Compute node to the same path on the Network node.<br /> <br /> 11. Edit /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows:<br /> [linux_bridge] <br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;eth_ipoib interface&gt; <br /> (i.e. default:eth2)<br /> 12. Edit /etc/neutron/dhcp_agent.ini and /etc/neutron/l3_agent.ini as follows: <br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver<br /> 13. Restart the following services:<br /> service neutron-plugin-linuxbridge-agent restart<br /> service neutron-dhcp-agent restart<br /> service neutron-l3-agent restart<br /> <br /> == Troubleshooting ==<br /> <br /> * If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand&diff=69187 2014-11-30T08:36:27Z

<p>May: /* Compute Nodes */</p> <hr /> <div>= Overview = <br /> <br /> This wiki page describes how to tune Mellanox OpenStack on Ubuntu 12.04 servers<br /> <br /> = Prerequisites =<br /> <br /> * A running OpenStack environment installed with the ML2 plugin on top of Linux Bridge.<br /> * All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119) <br /> * Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers<br /> * SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community [http://community.mellanox.com/docs/DOC-1317 click here].<br /> * The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes<br /> * VLANs configured on the ports in the switch.<br /> <br /> = InfiniBand Network = <br /> <br /> == OpenSM Node ==<br /> <br /> To configure OpenSM node: <br /> <br /> 1. Edit the file /etc/opensm/partitions.conf by adding the following line:<br /> management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;<br /> Create the file if it does not exist. <br /> <br /> For each VLAN value in the VLAN range used for VLAN network segmentation, add the following line in the file /etc/opensm/partitions.conf (for vlan-pkey association):<br /> vlan&lt;decimal-value&gt;=0x&lt;hexadecimal-value&gt;, ipoib, sl=0, defmember=full: ALL;<br /> In this manual, the VLAN range is 2-10 (mlnx_agent.conf /ml2_conf.ini). Therefore, add the following lines:<br /> vlan1=0x1, ipoib, sl=0, defmember=full : ALL;<br /> vlan2=0x2, ipoib, sl=0, defmember=full : ALL;<br /> vlan3=0x3, ipoib, sl=0, defmember=full : ALL;<br /> vlan4=0x4, ipoib, sl=0, defmember=full : ALL;<br /> vlan5=0x5, ipoib, sl=0, defmember=full : ALL;<br /> vlan6=0x6, ipoib, sl=0, defmember=full : ALL;<br /> vlan7=0x7, ipoib, sl=0, defmember=full : ALL;<br /> vlan8=0x8, ipoib, sl=0, defmember=full : ALL;<br /> vlan9=0x9, ipoib, sl=0, defmember=full : ALL;<br /> vlan10=0xa, ipoib, sl=0, defmember=full : ALL;<br /> 2. Modify the following line in the file /etc/opensm/opensm.conf from FALSE to TRUE:<br /> allow_both_pkeys TRUE<br /> 3. Restart OpenSM:<br /> service opensmd restart<br /> 4. Verify that all ports on all nodes have their ib interface (i.e. ib0) active, run ibdev2netdev on all hosts, and make sure that the port is up:<br /> mlx4_0 port 1 ==&gt; ib0 (Up)<br /> If the port is down, try bringing it up by running “ifconfig ib0 up”, and make sure your switch ports are of type “InfiniBand”, and that all nodes can reach the OpenSM node by running “sminfo”.<br /> <br /> == Controller Node == <br /> <br /> To configure the Controller node: <br /> <br /> 1. Ensure that the file /etc/nova/nova.conf includes the following line:<br /> security_group_api=nova<br /> <br /> 2. If the security group value is changed, restart the Nova services:<br /> # for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done<br /> <br /> == Neutron Server Node ==<br /> <br /> NOTE: The Neutron Server can sometimes be part of the Controller node.<br /> <br /> To configure the Neutron Server node: <br /> <br /> 1. Edit /etc/neutron/neutron.conf and make sure Neutron is using ML2 plugin:<br /> [DEFAULT]<br /> core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin<br /> 2. Edit the ML2 configuration file /etc/neutron/plugins/ml2/ml2_conf.ini as follows:<br /> [ml2]<br /> type_drivers = vlan,flat<br /> tenant_network_types = vlan <br /> mechanism_drivers = linuxbridge,mlnx<br /> <br /> [ml2_type_vlan]<br /> network_vlan_ranges=&lt;physical_network&gt;:&lt;min_vlan&gt;:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> <br /> [securitygroup]<br /> enable_security_group = True<br /> <br /> [eswitch]<br /> vnic_type = hostdev<br /> apply_profile_patch = True<br /> 3. Install the required Deb for the Mellanox Neutron plugin (from Ubuntu’s repository):<br /> apt-get install neutron-plugin-mlnx-agent<br /> 4. Edit /etc/neutron/plugins/mlnx/mlnx_conf.ini to match your environment:<br /> [mlnx]<br /> network_vlan_ranges = &lt;physical_network&gt;:&lt;min_vlan:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> tenant_network_type = vlan<br /> <br /> [eswitch]<br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;ib interface&gt; //(i.e. default:ib0)<br /> vnic_type = hostdev<br /> <br /> [agent]<br /> support_old_agents = True<br /> <br /> 5. Restart Neutron Server: <br /> /etc/init.d/neutron-server start<br /> <br /> == Compute Nodes ==<br /> <br /> To configure the Compute node:<br /> 1. Create the file /etc/modprobe.d/mlx4_ib.conf with this line:<br /> options mlx4_ib sm_guid_assign=0<br /> 2. Download mlnxvif, eSwitch Daemon (eSwitchd), and mlnx-dnsmasq from this link:<br /> http://www.mellanox.com/downloads/solutions/openstack/icehouse/debian/<br /> 3. Install the Mellanox plugin services mlnxvif and eSwitchd:<br /> dpkg –i python-mlnxvif_2014.1.1-1_all.deb<br /> dpkg –i python-eswitchd_0.10-1_all.deb<br /> 4. Edit /etc/eswitchd/eswitchd.conf as follows:<br /> [DAEMON]<br /> fabrics=&lt;physical_network&gt;:&lt;ib-interface&gt; //(i.e. default:ib0)<br /> 5. Edit /etc/nova/nova.conf as follows:<br /> [DEFAULT]<br /> compute_driver = libvirt.LibvirtDriver<br /> libvirt_vif_driver = mlnxvif.vif.MlxEthVIFDriver<br /> security_group_api = nova<br /> 6. Check if “ebrctl” appears in one of Nova’s rootwrap files:<br /> grep 'ebrctl' /etc/nova/rootwrap.d/*<br /> If “ebrctl” does not appear in one of Nova’s rootwrap files, edit the file /etc/nova/rootwrap.d/compute.filter as follows:<br /> [Filters]<br /> ebrctl: CommandFilter, ebrctl, root<br /> 7. Install the Neutron Mellanox agent (from Ubuntu repository):<br /> apt-get install neutron-plugin-mlnx-agent<br /> 8. Copy /etc/neutron/plugins/mlnx/mlnx_conf.ini from the Neutron Server node to the same path on each Compute node.<br /> 9. Restart the following services:<br /> /etc/init.d/nova-compute restart<br /> /etc/init.d/eswitchd start<br /> /etc/init.d/neutron-plugin-mlnx-agent start<br /> <br /> == Network Node ==<br /> <br /> To configure the Network node: <br /> <br /> 1. Configure eIPoIB (for Linux bridge plugin) by editing the file /etc/infiniband/openib.conf as follows:<br /> E_IPOIB_LOAD=yes<br /> 2. Restart OFED: <br /> service openibd restart<br /> 3. Verify that the new Ethernet interfaces now appear with ibdev2netdev:<br /> # ibdev2netdev<br /> mlx4_0 port 1 ==&gt; ib0 (Up)<br /> mlx4_0 port 2 ==&gt; ib1 (Down)<br /> mlx4_0 port 1 ==&gt; eth2 (Up)<br /> mlx4_0 port 2 ==&gt; eth1 (Down)<br /> In eIPoIB, every ib interface has a matching eth interface. The ib interface used here is ib0, and the matching one is eth2 (same card and port names: mlx4_0 port 1). By using Ethtool, you can verify that ib0 is working on ib_ipoib driver, and eth2 is working on eth_ipoib driver, as follows:<br /> # ethtool -i ib0<br /> driver: ib_ipoib<br /> version: 2.3-1.5.4_Sep 30 2014<br /> firmware-version: 2.32.5100<br /> bus-info: 0000:07:00.0# ethtool -i eth2<br /> driver: eth_ipoib<br /> version: 2.3-1.5.4 (Sep 30 2014)<br /> firmware-version: 1<br /> bus-info: ib0<br /> 4. Edit /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows:<br /> [linux_bridge] <br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;eth_ipoib-interface&gt; <br /> //(i.e. default:eth2)<br /> 5. Restart the Linux bridge agent:<br /> service neutron-linuxbridge-agent restart<br /> 6. To allow DHCP support, download mlnx-dnsmadq from this link: http://www.mellanox.com/downloads/solutions/openstack/icehouse/debian/<br /> <br /> 7. Install the package:<br /> dpkg –i python-mlnx-dnsmasq_2014.1.1-1_all.deb<br /> 8. Configure the file /etc/neutron/dhcp_agent.ini:<br /> [DEFAULT]<br /> dhcp_driver = mlnx_dhcp.MlnxDnsmasq<br /> 9. Make sure that the Dnsmasq which is installed as a part of the OpenStack (not mlnx-dnsmasq) environment is of version 2.65.<br /> <br /> 10. Copy /etc/neutron/neutron.conf from the Compute node to the same path on the Network node.<br /> <br /> 11. Edit /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows:<br /> [linux_bridge] <br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;eth_ipoib interface&gt; <br /> (i.e. default:eth2)<br /> 12. Edit /etc/neutron/dhcp_agent.ini and /etc/neutron/l3_agent.ini as follows: <br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver<br /> 13. Restart the following services:<br /> service neutron-plugin-linuxbridge-agent restart<br /> service neutron-dhcp-agent restart<br /> service neutron-l3-agent restart<br /> <br /> == Troubleshooting ==<br /> <br /> * If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand&diff=69186 2014-11-30T08:34:56Z

<p>May: /* Neutron Server Node */</p> <hr /> <div>= Overview = <br /> <br /> This wiki page describes how to tune Mellanox OpenStack on Ubuntu 12.04 servers<br /> <br /> = Prerequisites =<br /> <br /> * A running OpenStack environment installed with the ML2 plugin on top of Linux Bridge.<br /> * All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119) <br /> * Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers<br /> * SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community [http://community.mellanox.com/docs/DOC-1317 click here].<br /> * The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes<br /> * VLANs configured on the ports in the switch.<br /> <br /> = InfiniBand Network = <br /> <br /> == OpenSM Node ==<br /> <br /> To configure OpenSM node: <br /> <br /> 1. Edit the file /etc/opensm/partitions.conf by adding the following line:<br /> management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;<br /> Create the file if it does not exist. <br /> <br /> For each VLAN value in the VLAN range used for VLAN network segmentation, add the following line in the file /etc/opensm/partitions.conf (for vlan-pkey association):<br /> vlan&lt;decimal-value&gt;=0x&lt;hexadecimal-value&gt;, ipoib, sl=0, defmember=full: ALL;<br /> In this manual, the VLAN range is 2-10 (mlnx_agent.conf /ml2_conf.ini). Therefore, add the following lines:<br /> vlan1=0x1, ipoib, sl=0, defmember=full : ALL;<br /> vlan2=0x2, ipoib, sl=0, defmember=full : ALL;<br /> vlan3=0x3, ipoib, sl=0, defmember=full : ALL;<br /> vlan4=0x4, ipoib, sl=0, defmember=full : ALL;<br /> vlan5=0x5, ipoib, sl=0, defmember=full : ALL;<br /> vlan6=0x6, ipoib, sl=0, defmember=full : ALL;<br /> vlan7=0x7, ipoib, sl=0, defmember=full : ALL;<br /> vlan8=0x8, ipoib, sl=0, defmember=full : ALL;<br /> vlan9=0x9, ipoib, sl=0, defmember=full : ALL;<br /> vlan10=0xa, ipoib, sl=0, defmember=full : ALL;<br /> 2. Modify the following line in the file /etc/opensm/opensm.conf from FALSE to TRUE:<br /> allow_both_pkeys TRUE<br /> 3. Restart OpenSM:<br /> service opensmd restart<br /> 4. Verify that all ports on all nodes have their ib interface (i.e. ib0) active, run ibdev2netdev on all hosts, and make sure that the port is up:<br /> mlx4_0 port 1 ==&gt; ib0 (Up)<br /> If the port is down, try bringing it up by running “ifconfig ib0 up”, and make sure your switch ports are of type “InfiniBand”, and that all nodes can reach the OpenSM node by running “sminfo”.<br /> <br /> == Controller Node == <br /> <br /> To configure the Controller node: <br /> <br /> 1. Ensure that the file /etc/nova/nova.conf includes the following line:<br /> security_group_api=nova<br /> <br /> 2. If the security group value is changed, restart the Nova services:<br /> # for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done<br /> <br /> == Neutron Server Node ==<br /> <br /> NOTE: The Neutron Server can sometimes be part of the Controller node.<br /> <br /> To configure the Neutron Server node: <br /> <br /> 1. Edit /etc/neutron/neutron.conf and make sure Neutron is using ML2 plugin:<br /> [DEFAULT]<br /> core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin<br /> 2. Edit the ML2 configuration file /etc/neutron/plugins/ml2/ml2_conf.ini as follows:<br /> [ml2]<br /> type_drivers = vlan,flat<br /> tenant_network_types = vlan <br /> mechanism_drivers = linuxbridge,mlnx<br /> <br /> [ml2_type_vlan]<br /> network_vlan_ranges=&lt;physical_network&gt;:&lt;min_vlan&gt;:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> <br /> [securitygroup]<br /> enable_security_group = True<br /> <br /> [eswitch]<br /> vnic_type = hostdev<br /> apply_profile_patch = True<br /> 3. Install the required Deb for the Mellanox Neutron plugin (from Ubuntu’s repository):<br /> apt-get install neutron-plugin-mlnx-agent<br /> 4. Edit /etc/neutron/plugins/mlnx/mlnx_conf.ini to match your environment:<br /> [mlnx]<br /> network_vlan_ranges = &lt;physical_network&gt;:&lt;min_vlan:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> tenant_network_type = vlan<br /> <br /> [eswitch]<br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;ib interface&gt; //(i.e. default:ib0)<br /> vnic_type = hostdev<br /> <br /> [agent]<br /> support_old_agents = True<br /> <br /> 5. Restart Neutron Server: <br /> /etc/init.d/neutron-server start<br /> <br /> == Compute Nodes ==<br /> <br /> NOTE: The Neutron Server can sometimes be part of the Controller node.<br /> <br /> To configure the Neutron Server node: <br /> <br /> 1. Edit /etc/neutron/neutron.conf and make sure Neutron is using ML2 plugin:<br /> [DEFAULT]<br /> core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin<br /> 2. Edit the ML2 configuration file /etc/neutron/plugins/ml2/ml2_conf.ini as follows:<br /> [ml2]<br /> type_drivers = vlan,flat<br /> tenant_network_types = vlan<br /> mechanism_drivers = linuxbridge,mlnx<br /> <br /> [ml2_type_vlan]<br /> network_vlan_ranges=&lt;physical_network&gt;:&lt;min_vlan&gt;:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> <br /> [securitygroup]<br /> enable_security_group = True<br /> <br /> [eswitch]<br /> vnic_type = hostdev<br /> apply_profile_patch = True<br /> 3. Install the required Deb for the Mellanox Neutron plugin (from Ubuntu’s repository):<br /> apt-get install neutron-plugin-mlnx-agent<br /> 4. Edit /etc/neutron/plugins/mlnx/mlnx_conf.ini to match your environment:<br /> [mlnx]<br /> network_vlan_ranges = &lt;physical_network&gt;:&lt;min_vlan:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> tenant_network_type = vlan<br /> <br /> [eswitch]<br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;ib interface&gt; //(i.e. default:ib0)<br /> vnic_type = hostdev<br /> <br /> [agent]<br /> support_old_agents = True<br /> <br /> 5. Restart Neutron Server: <br /> /etc/init.d/neutron-server start<br /> <br /> == Network Node ==<br /> <br /> To configure the Network node: <br /> <br /> 1. Configure eIPoIB (for Linux bridge plugin) by editing the file /etc/infiniband/openib.conf as follows:<br /> E_IPOIB_LOAD=yes<br /> 2. Restart OFED: <br /> service openibd restart<br /> 3. Verify that the new Ethernet interfaces now appear with ibdev2netdev:<br /> # ibdev2netdev<br /> mlx4_0 port 1 ==&gt; ib0 (Up)<br /> mlx4_0 port 2 ==&gt; ib1 (Down)<br /> mlx4_0 port 1 ==&gt; eth2 (Up)<br /> mlx4_0 port 2 ==&gt; eth1 (Down)<br /> In eIPoIB, every ib interface has a matching eth interface. The ib interface used here is ib0, and the matching one is eth2 (same card and port names: mlx4_0 port 1). By using Ethtool, you can verify that ib0 is working on ib_ipoib driver, and eth2 is working on eth_ipoib driver, as follows:<br /> # ethtool -i ib0<br /> driver: ib_ipoib<br /> version: 2.3-1.5.4_Sep 30 2014<br /> firmware-version: 2.32.5100<br /> bus-info: 0000:07:00.0# ethtool -i eth2<br /> driver: eth_ipoib<br /> version: 2.3-1.5.4 (Sep 30 2014)<br /> firmware-version: 1<br /> bus-info: ib0<br /> 4. Edit /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows:<br /> [linux_bridge] <br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;eth_ipoib-interface&gt; <br /> //(i.e. default:eth2)<br /> 5. Restart the Linux bridge agent:<br /> service neutron-linuxbridge-agent restart<br /> 6. To allow DHCP support, download mlnx-dnsmadq from this link: http://www.mellanox.com/downloads/solutions/openstack/icehouse/debian/<br /> <br /> 7. Install the package:<br /> dpkg –i python-mlnx-dnsmasq_2014.1.1-1_all.deb<br /> 8. Configure the file /etc/neutron/dhcp_agent.ini:<br /> [DEFAULT]<br /> dhcp_driver = mlnx_dhcp.MlnxDnsmasq<br /> 9. Make sure that the Dnsmasq which is installed as a part of the OpenStack (not mlnx-dnsmasq) environment is of version 2.65.<br /> <br /> 10. Copy /etc/neutron/neutron.conf from the Compute node to the same path on the Network node.<br /> <br /> 11. Edit /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows:<br /> [linux_bridge] <br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;eth_ipoib interface&gt; <br /> (i.e. default:eth2)<br /> 12. Edit /etc/neutron/dhcp_agent.ini and /etc/neutron/l3_agent.ini as follows: <br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver<br /> 13. Restart the following services:<br /> service neutron-plugin-linuxbridge-agent restart<br /> service neutron-dhcp-agent restart<br /> service neutron-l3-agent restart<br /> <br /> == Troubleshooting ==<br /> <br /> * If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand&diff=69079 2014-11-26T16:54:20Z

<p>May: /* Network Node */</p> <hr /> <div>= Overview = <br /> <br /> This wiki page describes how to tune Mellanox OpenStack on Ubuntu 12.04 servers<br /> <br /> = Prerequisites =<br /> <br /> * A running OpenStack environment installed with the ML2 plugin on top of OVS.<br /> * All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119) <br /> * Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers<br /> * SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community [http://community.mellanox.com/docs/DOC-1317 click here].<br /> * The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes<br /> * VLANs configured on the ports in the switch.<br /> <br /> = InfiniBand Network = <br /> <br /> == OpenSM Node ==<br /> <br /> To configure OpenSM node: <br /> <br /> 1. Edit the file /etc/opensm/partitions.conf by adding the following line:<br /> management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;<br /> Create the file if it does not exist. <br /> <br /> For each VLAN value in the VLAN range used for VLAN network segmentation, add the following line in the file /etc/opensm/partitions.conf (for vlan-pkey association):<br /> vlan&lt;decimal-value&gt;=0x&lt;hexadecimal-value&gt;, ipoib, sl=0, defmember=full: ALL;<br /> In this manual, the VLAN range is 2-10 (mlnx_agent.conf /ml2_conf.ini). Therefore, add the following lines:<br /> vlan1=0x1, ipoib, sl=0, defmember=full : ALL;<br /> vlan2=0x2, ipoib, sl=0, defmember=full : ALL;<br /> vlan3=0x3, ipoib, sl=0, defmember=full : ALL;<br /> vlan4=0x4, ipoib, sl=0, defmember=full : ALL;<br /> vlan5=0x5, ipoib, sl=0, defmember=full : ALL;<br /> vlan6=0x6, ipoib, sl=0, defmember=full : ALL;<br /> vlan7=0x7, ipoib, sl=0, defmember=full : ALL;<br /> vlan8=0x8, ipoib, sl=0, defmember=full : ALL;<br /> vlan9=0x9, ipoib, sl=0, defmember=full : ALL;<br /> vlan10=0xa, ipoib, sl=0, defmember=full : ALL;<br /> 2. Modify the following line in the file /etc/opensm/opensm.conf from FALSE to TRUE:<br /> allow_both_pkeys TRUE<br /> 3. Restart OpenSM:<br /> service opensmd restart<br /> 4. Verify that all ports on all nodes have their ib interface (i.e. ib0) active, run ibdev2netdev on all hosts, and make sure that the port is up:<br /> mlx4_0 port 1 ==&gt; ib0 (Up)<br /> If the port is down, try bringing it up by running “ifconfig ib0 up”, and make sure your switch ports are of type “InfiniBand”, and that all nodes can reach the OpenSM node by running “sminfo”.<br /> <br /> == Controller Node == <br /> <br /> To configure the Controller node: <br /> <br /> 1. Ensure that the file /etc/nova/nova.conf includes the following line:<br /> security_group_api=nova<br /> <br /> 2. If the security group value is changed, restart the Nova services:<br /> # for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done<br /> <br /> === Neutron Server Node ===<br /> <br /> NOTE: The Neutron Server can sometimes be part of the Controller node.<br /> <br /> To configure the Neutron Server node: <br /> <br /> 1. Edit /etc/neutron/neutron.conf and make sure Neutron is using ML2 plugin:<br /> [DEFAULT]<br /> core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin<br /> 2. Edit the ML2 configuration file /etc/neutron/plugins/ml2/ml2_conf.ini as follows:<br /> [ml2]<br /> type_drivers = vlan,flat<br /> tenant_network_types = vlan <br /> mechanism_drivers = linuxbridge,mlnx<br /> <br /> [ml2_type_vlan]<br /> network_vlan_ranges=&lt;physical_network&gt;:&lt;min_vlan&gt;:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> <br /> [securitygroup]<br /> enable_security_group = True<br /> <br /> [eswitch]<br /> vnic_type = hostdev<br /> apply_profile_patch = True<br /> 3. Install the required Deb for the Mellanox Neutron plugin (from Ubuntu’s repository):<br /> apt-get install neutron-plugin-mlnx-agent<br /> 4. Edit /etc/neutron/plugins/mlnx/mlnx_conf.ini to match your environment:<br /> [mlnx]<br /> network_vlan_ranges = &lt;physical_network&gt;:&lt;min_vlan:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> tenant_network_type = vlan<br /> <br /> [eswitch]<br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;ib interface&gt; //(i.e. default:ib0)<br /> vnic_type = hostdev<br /> <br /> [agent]<br /> support_old_agents = True<br /> <br /> 5. Restart Neutron Server: <br /> /etc/init.d/neutron-server start<br /> <br /> == Compute Nodes ==<br /> <br /> NOTE: The Neutron Server can sometimes be part of the Controller node.<br /> <br /> To configure the Neutron Server node: <br /> <br /> 1. Edit /etc/neutron/neutron.conf and make sure Neutron is using ML2 plugin:<br /> [DEFAULT]<br /> core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin<br /> 2. Edit the ML2 configuration file /etc/neutron/plugins/ml2/ml2_conf.ini as follows:<br /> [ml2]<br /> type_drivers = vlan,flat<br /> tenant_network_types = vlan<br /> mechanism_drivers = linuxbridge,mlnx<br /> <br /> [ml2_type_vlan]<br /> network_vlan_ranges=&lt;physical_network&gt;:&lt;min_vlan&gt;:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> <br /> [securitygroup]<br /> enable_security_group = True<br /> <br /> [eswitch]<br /> vnic_type = hostdev<br /> apply_profile_patch = True<br /> 3. Install the required Deb for the Mellanox Neutron plugin (from Ubuntu’s repository):<br /> apt-get install neutron-plugin-mlnx-agent<br /> 4. Edit /etc/neutron/plugins/mlnx/mlnx_conf.ini to match your environment:<br /> [mlnx]<br /> network_vlan_ranges = &lt;physical_network&gt;:&lt;min_vlan:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> tenant_network_type = vlan<br /> <br /> [eswitch]<br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;ib interface&gt; //(i.e. default:ib0)<br /> vnic_type = hostdev<br /> <br /> [agent]<br /> support_old_agents = True<br /> <br /> 5. Restart Neutron Server: <br /> /etc/init.d/neutron-server start<br /> <br /> == Network Node ==<br /> <br /> To configure the Network node: <br /> <br /> 1. Configure eIPoIB (for Linux bridge plugin) by editing the file /etc/infiniband/openib.conf as follows:<br /> E_IPOIB_LOAD=yes<br /> 2. Restart OFED: <br /> service openibd restart<br /> 3. Verify that the new Ethernet interfaces now appear with ibdev2netdev:<br /> # ibdev2netdev<br /> mlx4_0 port 1 ==&gt; ib0 (Up)<br /> mlx4_0 port 2 ==&gt; ib1 (Down)<br /> mlx4_0 port 1 ==&gt; eth2 (Up)<br /> mlx4_0 port 2 ==&gt; eth1 (Down)<br /> In eIPoIB, every ib interface has a matching eth interface. The ib interface used here is ib0, and the matching one is eth2 (same card and port names: mlx4_0 port 1). By using Ethtool, you can verify that ib0 is working on ib_ipoib driver, and eth2 is working on eth_ipoib driver, as follows:<br /> # ethtool -i ib0<br /> driver: ib_ipoib<br /> version: 2.3-1.5.4_Sep 30 2014<br /> firmware-version: 2.32.5100<br /> bus-info: 0000:07:00.0# ethtool -i eth2<br /> driver: eth_ipoib<br /> version: 2.3-1.5.4 (Sep 30 2014)<br /> firmware-version: 1<br /> bus-info: ib0<br /> 4. Edit /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows:<br /> [linux_bridge] <br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;eth_ipoib-interface&gt; <br /> //(i.e. default:eth2)<br /> 5. Restart the Linux bridge agent:<br /> service neutron-linuxbridge-agent restart<br /> 6. To allow DHCP support, download mlnx-dnsmadq from this link: http://www.mellanox.com/downloads/solutions/openstack/icehouse/debian/<br /> <br /> 7. Install the package:<br /> dpkg –i python-mlnx-dnsmasq_2014.1.1-1_all.deb<br /> 8. Configure the file /etc/neutron/dhcp_agent.ini:<br /> [DEFAULT]<br /> dhcp_driver = mlnx_dhcp.MlnxDnsmasq<br /> 9. Make sure that the Dnsmasq which is installed as a part of the OpenStack (not mlnx-dnsmasq) environment is of version 2.65.<br /> <br /> 10. Copy /etc/neutron/neutron.conf from the Compute node to the same path on the Network node.<br /> <br /> 11. Edit /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows:<br /> [linux_bridge] <br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;eth_ipoib interface&gt; <br /> (i.e. default:eth2)<br /> 12. Edit /etc/neutron/dhcp_agent.ini and /etc/neutron/l3_agent.ini as follows: <br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver<br /> 13. Restart the following services:<br /> service neutron-plugin-linuxbridge-agent restart<br /> service neutron-dhcp-agent restart<br /> service neutron-l3-agent restart<br /> <br /> == Troubleshooting ==<br /> <br /> * If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand&diff=69078 2014-11-26T16:53:45Z

<p>May: /* Network Node */</p> <hr /> <div>= Overview = <br /> <br /> This wiki page describes how to tune Mellanox OpenStack on Ubuntu 12.04 servers<br /> <br /> = Prerequisites =<br /> <br /> * A running OpenStack environment installed with the ML2 plugin on top of OVS.<br /> * All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119) <br /> * Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers<br /> * SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community [http://community.mellanox.com/docs/DOC-1317 click here].<br /> * The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes<br /> * VLANs configured on the ports in the switch.<br /> <br /> = InfiniBand Network = <br /> <br /> == OpenSM Node ==<br /> <br /> To configure OpenSM node: <br /> <br /> 1. Edit the file /etc/opensm/partitions.conf by adding the following line:<br /> management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;<br /> Create the file if it does not exist. <br /> <br /> For each VLAN value in the VLAN range used for VLAN network segmentation, add the following line in the file /etc/opensm/partitions.conf (for vlan-pkey association):<br /> vlan&lt;decimal-value&gt;=0x&lt;hexadecimal-value&gt;, ipoib, sl=0, defmember=full: ALL;<br /> In this manual, the VLAN range is 2-10 (mlnx_agent.conf /ml2_conf.ini). Therefore, add the following lines:<br /> vlan1=0x1, ipoib, sl=0, defmember=full : ALL;<br /> vlan2=0x2, ipoib, sl=0, defmember=full : ALL;<br /> vlan3=0x3, ipoib, sl=0, defmember=full : ALL;<br /> vlan4=0x4, ipoib, sl=0, defmember=full : ALL;<br /> vlan5=0x5, ipoib, sl=0, defmember=full : ALL;<br /> vlan6=0x6, ipoib, sl=0, defmember=full : ALL;<br /> vlan7=0x7, ipoib, sl=0, defmember=full : ALL;<br /> vlan8=0x8, ipoib, sl=0, defmember=full : ALL;<br /> vlan9=0x9, ipoib, sl=0, defmember=full : ALL;<br /> vlan10=0xa, ipoib, sl=0, defmember=full : ALL;<br /> 2. Modify the following line in the file /etc/opensm/opensm.conf from FALSE to TRUE:<br /> allow_both_pkeys TRUE<br /> 3. Restart OpenSM:<br /> service opensmd restart<br /> 4. Verify that all ports on all nodes have their ib interface (i.e. ib0) active, run ibdev2netdev on all hosts, and make sure that the port is up:<br /> mlx4_0 port 1 ==&gt; ib0 (Up)<br /> If the port is down, try bringing it up by running “ifconfig ib0 up”, and make sure your switch ports are of type “InfiniBand”, and that all nodes can reach the OpenSM node by running “sminfo”.<br /> <br /> == Controller Node == <br /> <br /> To configure the Controller node: <br /> <br /> 1. Ensure that the file /etc/nova/nova.conf includes the following line:<br /> security_group_api=nova<br /> <br /> 2. If the security group value is changed, restart the Nova services:<br /> # for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done<br /> <br /> === Neutron Server Node ===<br /> <br /> NOTE: The Neutron Server can sometimes be part of the Controller node.<br /> <br /> To configure the Neutron Server node: <br /> <br /> 1. Edit /etc/neutron/neutron.conf and make sure Neutron is using ML2 plugin:<br /> [DEFAULT]<br /> core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin<br /> 2. Edit the ML2 configuration file /etc/neutron/plugins/ml2/ml2_conf.ini as follows:<br /> [ml2]<br /> type_drivers = vlan,flat<br /> tenant_network_types = vlan <br /> mechanism_drivers = linuxbridge,mlnx<br /> <br /> [ml2_type_vlan]<br /> network_vlan_ranges=&lt;physical_network&gt;:&lt;min_vlan&gt;:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> <br /> [securitygroup]<br /> enable_security_group = True<br /> <br /> [eswitch]<br /> vnic_type = hostdev<br /> apply_profile_patch = True<br /> 3. Install the required Deb for the Mellanox Neutron plugin (from Ubuntu’s repository):<br /> apt-get install neutron-plugin-mlnx-agent<br /> 4. Edit /etc/neutron/plugins/mlnx/mlnx_conf.ini to match your environment:<br /> [mlnx]<br /> network_vlan_ranges = &lt;physical_network&gt;:&lt;min_vlan:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> tenant_network_type = vlan<br /> <br /> [eswitch]<br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;ib interface&gt; //(i.e. default:ib0)<br /> vnic_type = hostdev<br /> <br /> [agent]<br /> support_old_agents = True<br /> <br /> 5. Restart Neutron Server: <br /> /etc/init.d/neutron-server start<br /> <br /> == Compute Nodes ==<br /> <br /> NOTE: The Neutron Server can sometimes be part of the Controller node.<br /> <br /> To configure the Neutron Server node: <br /> <br /> 1. Edit /etc/neutron/neutron.conf and make sure Neutron is using ML2 plugin:<br /> [DEFAULT]<br /> core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin<br /> 2. Edit the ML2 configuration file /etc/neutron/plugins/ml2/ml2_conf.ini as follows:<br /> [ml2]<br /> type_drivers = vlan,flat<br /> tenant_network_types = vlan<br /> mechanism_drivers = linuxbridge,mlnx<br /> <br /> [ml2_type_vlan]<br /> network_vlan_ranges=&lt;physical_network&gt;:&lt;min_vlan&gt;:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> <br /> [securitygroup]<br /> enable_security_group = True<br /> <br /> [eswitch]<br /> vnic_type = hostdev<br /> apply_profile_patch = True<br /> 3. Install the required Deb for the Mellanox Neutron plugin (from Ubuntu’s repository):<br /> apt-get install neutron-plugin-mlnx-agent<br /> 4. Edit /etc/neutron/plugins/mlnx/mlnx_conf.ini to match your environment:<br /> [mlnx]<br /> network_vlan_ranges = &lt;physical_network&gt;:&lt;min_vlan:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> tenant_network_type = vlan<br /> <br /> [eswitch]<br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;ib interface&gt; //(i.e. default:ib0)<br /> vnic_type = hostdev<br /> <br /> [agent]<br /> support_old_agents = True<br /> <br /> 5. Restart Neutron Server: <br /> /etc/init.d/neutron-server start<br /> <br /> == Network Node ==<br /> <br /> To configure the Network node: <br /> <br /> 1. Configure eIPoIB (for Linux bridge plugin) by editing the file /etc/infiniband/openib.conf as follows:<br /> E_IPOIB_LOAD=yes<br /> 2. Restart OFED: <br /> service openibd restart<br /> 3. Verify that the new Ethernet interfaces now appear with ibdev2netdev:<br /> # ibdev2netdev<br /> mlx4_0 port 1 ==&gt; ib0 (Up)<br /> mlx4_0 port 2 ==&gt; ib1 (Down)<br /> mlx4_0 port 1 ==&gt; eth2 (Up)<br /> mlx4_0 port 2 ==&gt; eth1 (Down)<br /> In eIPoIB, every ib interface has a matching eth interface. The ib interface used here is ib0, and the matching one is eth2 (same card and port names: mlx4_0 port 1). By using Ethtool, you can verify that ib0 is working on ib_ipoib driver, and eth2 is working on eth_ipoib driver, as follows:<br /> # ethtool -i ib0<br /> driver: ib_ipoib<br /> version: 2.3-1.5.4_Sep 30 2014<br /> firmware-version: 2.32.5100<br /> bus-info: 0000:07:00.0# ethtool -i eth2<br /> driver: eth_ipoib<br /> version: 2.3-1.5.4 (Sep 30 2014)<br /> firmware-version: 1<br /> bus-info: ib0<br /> 4. Edit /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows:<br /> [linux_bridge] <br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;eth_ipoib-interface&gt; <br /> //(i.e. default:eth2)<br /> 5. Restart the Linux bridge agent:<br /> service neutron-linuxbridge-agent restart<br /> 6. To allow DHCP support, download mlnx-dnsmadq from this link: http://www.mellanox.com/downloads/solutions/openstack/icehouse/debian/<br /> <br /> 7. Install the package:<br /> dpkg –i python-mlnx-dnsmasq_2014.1.1-1_all.deb<br /> 8. Configure the file /etc/neutron/dhcp_agent.ini:<br /> [DEFAULT]<br /> dhcp_driver = mlnx_dhcp.MlnxDnsmasq<br /> 9. Make sure that the Dnsmasq which is installed as a part of the OpenStack (not mlnx-dnsmasq) environment is of version 2.65.<br /> 10. Copy /etc/neutron/neutron.conf from the Compute node to the same path on the Network node.<br /> 11. Edit /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows:<br /> [linux_bridge] <br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;eth_ipoib interface&gt; <br /> (i.e. default:eth2)<br /> 12. Edit /etc/neutron/dhcp_agent.ini and /etc/neutron/l3_agent.ini as follows: <br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver<br /> 13. Restart the following services:<br /> service neutron-plugin-linuxbridge-agent restart<br /> service neutron-dhcp-agent restart<br /> service neutron-l3-agent restart<br /> <br /> == Troubleshooting ==<br /> <br /> * If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand&diff=69077 2014-11-26T16:52:48Z

<p>May: /* Network Node */</p> <hr /> <div>= Overview = <br /> <br /> This wiki page describes how to tune Mellanox OpenStack on Ubuntu 12.04 servers<br /> <br /> = Prerequisites =<br /> <br /> * A running OpenStack environment installed with the ML2 plugin on top of OVS.<br /> * All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119) <br /> * Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers<br /> * SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community [http://community.mellanox.com/docs/DOC-1317 click here].<br /> * The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes<br /> * VLANs configured on the ports in the switch.<br /> <br /> = InfiniBand Network = <br /> <br /> == OpenSM Node ==<br /> <br /> To configure OpenSM node: <br /> <br /> 1. Edit the file /etc/opensm/partitions.conf by adding the following line:<br /> management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;<br /> Create the file if it does not exist. <br /> <br /> For each VLAN value in the VLAN range used for VLAN network segmentation, add the following line in the file /etc/opensm/partitions.conf (for vlan-pkey association):<br /> vlan&lt;decimal-value&gt;=0x&lt;hexadecimal-value&gt;, ipoib, sl=0, defmember=full: ALL;<br /> In this manual, the VLAN range is 2-10 (mlnx_agent.conf /ml2_conf.ini). Therefore, add the following lines:<br /> vlan1=0x1, ipoib, sl=0, defmember=full : ALL;<br /> vlan2=0x2, ipoib, sl=0, defmember=full : ALL;<br /> vlan3=0x3, ipoib, sl=0, defmember=full : ALL;<br /> vlan4=0x4, ipoib, sl=0, defmember=full : ALL;<br /> vlan5=0x5, ipoib, sl=0, defmember=full : ALL;<br /> vlan6=0x6, ipoib, sl=0, defmember=full : ALL;<br /> vlan7=0x7, ipoib, sl=0, defmember=full : ALL;<br /> vlan8=0x8, ipoib, sl=0, defmember=full : ALL;<br /> vlan9=0x9, ipoib, sl=0, defmember=full : ALL;<br /> vlan10=0xa, ipoib, sl=0, defmember=full : ALL;<br /> 2. Modify the following line in the file /etc/opensm/opensm.conf from FALSE to TRUE:<br /> allow_both_pkeys TRUE<br /> 3. Restart OpenSM:<br /> service opensmd restart<br /> 4. Verify that all ports on all nodes have their ib interface (i.e. ib0) active, run ibdev2netdev on all hosts, and make sure that the port is up:<br /> mlx4_0 port 1 ==&gt; ib0 (Up)<br /> If the port is down, try bringing it up by running “ifconfig ib0 up”, and make sure your switch ports are of type “InfiniBand”, and that all nodes can reach the OpenSM node by running “sminfo”.<br /> <br /> == Controller Node == <br /> <br /> To configure the Controller node: <br /> <br /> 1. Ensure that the file /etc/nova/nova.conf includes the following line:<br /> security_group_api=nova<br /> <br /> 2. If the security group value is changed, restart the Nova services:<br /> # for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done<br /> <br /> === Neutron Server Node ===<br /> <br /> NOTE: The Neutron Server can sometimes be part of the Controller node.<br /> <br /> To configure the Neutron Server node: <br /> <br /> 1. Edit /etc/neutron/neutron.conf and make sure Neutron is using ML2 plugin:<br /> [DEFAULT]<br /> core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin<br /> 2. Edit the ML2 configuration file /etc/neutron/plugins/ml2/ml2_conf.ini as follows:<br /> [ml2]<br /> type_drivers = vlan,flat<br /> tenant_network_types = vlan <br /> mechanism_drivers = linuxbridge,mlnx<br /> <br /> [ml2_type_vlan]<br /> network_vlan_ranges=&lt;physical_network&gt;:&lt;min_vlan&gt;:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> <br /> [securitygroup]<br /> enable_security_group = True<br /> <br /> [eswitch]<br /> vnic_type = hostdev<br /> apply_profile_patch = True<br /> 3. Install the required Deb for the Mellanox Neutron plugin (from Ubuntu’s repository):<br /> apt-get install neutron-plugin-mlnx-agent<br /> 4. Edit /etc/neutron/plugins/mlnx/mlnx_conf.ini to match your environment:<br /> [mlnx]<br /> network_vlan_ranges = &lt;physical_network&gt;:&lt;min_vlan:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> tenant_network_type = vlan<br /> <br /> [eswitch]<br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;ib interface&gt; //(i.e. default:ib0)<br /> vnic_type = hostdev<br /> <br /> [agent]<br /> support_old_agents = True<br /> <br /> 5. Restart Neutron Server: <br /> /etc/init.d/neutron-server start<br /> <br /> == Compute Nodes ==<br /> <br /> NOTE: The Neutron Server can sometimes be part of the Controller node.<br /> <br /> To configure the Neutron Server node: <br /> <br /> 1. Edit /etc/neutron/neutron.conf and make sure Neutron is using ML2 plugin:<br /> [DEFAULT]<br /> core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin<br /> 2. Edit the ML2 configuration file /etc/neutron/plugins/ml2/ml2_conf.ini as follows:<br /> [ml2]<br /> type_drivers = vlan,flat<br /> tenant_network_types = vlan<br /> mechanism_drivers = linuxbridge,mlnx<br /> <br /> [ml2_type_vlan]<br /> network_vlan_ranges=&lt;physical_network&gt;:&lt;min_vlan&gt;:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> <br /> [securitygroup]<br /> enable_security_group = True<br /> <br /> [eswitch]<br /> vnic_type = hostdev<br /> apply_profile_patch = True<br /> 3. Install the required Deb for the Mellanox Neutron plugin (from Ubuntu’s repository):<br /> apt-get install neutron-plugin-mlnx-agent<br /> 4. Edit /etc/neutron/plugins/mlnx/mlnx_conf.ini to match your environment:<br /> [mlnx]<br /> network_vlan_ranges = &lt;physical_network&gt;:&lt;min_vlan:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> tenant_network_type = vlan<br /> <br /> [eswitch]<br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;ib interface&gt; //(i.e. default:ib0)<br /> vnic_type = hostdev<br /> <br /> [agent]<br /> support_old_agents = True<br /> <br /> 5. Restart Neutron Server: <br /> /etc/init.d/neutron-server start<br /> <br /> == Network Node ==<br /> <br /> To configure the Network node: <br /> <br /> 1. Configure eIPoIB (for Linux bridge plugin) by editing the file /etc/infiniband/openib.conf as follows:<br /> E_IPOIB_LOAD=yes<br /> 2. Restart OFED: <br /> service openibd restart<br /> 3. Verify that the new Ethernet interfaces now appear with ibdev2netdev:<br /> # ibdev2netdev<br /> mlx4_0 port 1 ==&gt; ib0 (Up)<br /> mlx4_0 port 2 ==&gt; ib1 (Down)<br /> mlx4_0 port 1 ==&gt; eth2 (Up)<br /> mlx4_0 port 2 ==&gt; eth1 (Down)<br /> In eIPoIB, every ib interface has a matching eth interface. The ib interface used here is ib0, and the matching one is eth2 (same card and port names: mlx4_0 port 1). By using Ethtool, you can verify that ib0 is working on ib_ipoib driver, and eth2 is working on eth_ipoib driver, as follows:<br /> # ethtool -i ib0<br /> driver: ib_ipoib<br /> version: 2.3-1.5.4_Sep 30 2014<br /> firmware-version: 2.32.5100<br /> bus-info: 0000:07:00.0# ethtool -i eth2<br /> driver: eth_ipoib<br /> version: 2.3-1.5.4 (Sep 30 2014)<br /> firmware-version: 1<br /> bus-info: ib0<br /> 4. Edit /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows:<br /> [linux_bridge] <br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;eth_ipoib-interface&gt; <br /> //(i.e. default:eth2)<br /> 5. Restart the Linux bridge agent:<br /> service neutron-linuxbridge-agent restart<br /> 6. To allow DHCP support, download mlnx-dnsmadq from this link: http://www.mellanox.com/downloads/solutions/openstack/icehouse/debian/<br /> 7. Install the package:<br /> dpkg –i python-mlnx-dnsmasq_2014.1.1-1_all.deb<br /> 8. Configure the file /etc/neutron/dhcp_agent.ini:<br /> [DEFAULT]<br /> dhcp_driver = mlnx_dhcp.MlnxDnsmasq<br /> 9. Make sure that the Dnsmasq which is installed as a part of the OpenStack (not mlnx-dnsmasq) environment is of version 2.65.<br /> 10. Copy /etc/neutron/neutron.conf from the Compute node to the same path on the Network node.<br /> 11. Edit /etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini as follows:<br /> [linux_bridge] <br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;eth_ipoib interface&gt; <br /> (i.e. default:eth2)<br /> 12. Edit /etc/neutron/dhcp_agent.ini and /etc/neutron/l3_agent.ini as follows: <br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver<br /> 13. Restart the following services:<br /> service neutron-plugin-linuxbridge-agent restart<br /> service neutron-dhcp-agent restart<br /> service neutron-l3-agent restart<br /> <br /> == Troubleshooting ==<br /> <br /> * If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand&diff=69076 2014-11-26T16:50:51Z

<p>May: /* Compute Nodes */</p> <hr /> <div>= Overview = <br /> <br /> This wiki page describes how to tune Mellanox OpenStack on Ubuntu 12.04 servers<br /> <br /> = Prerequisites =<br /> <br /> * A running OpenStack environment installed with the ML2 plugin on top of OVS.<br /> * All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119) <br /> * Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers<br /> * SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community [http://community.mellanox.com/docs/DOC-1317 click here].<br /> * The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes<br /> * VLANs configured on the ports in the switch.<br /> <br /> = InfiniBand Network = <br /> <br /> == OpenSM Node ==<br /> <br /> To configure OpenSM node: <br /> <br /> 1. Edit the file /etc/opensm/partitions.conf by adding the following line:<br /> management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;<br /> Create the file if it does not exist. <br /> <br /> For each VLAN value in the VLAN range used for VLAN network segmentation, add the following line in the file /etc/opensm/partitions.conf (for vlan-pkey association):<br /> vlan&lt;decimal-value&gt;=0x&lt;hexadecimal-value&gt;, ipoib, sl=0, defmember=full: ALL;<br /> In this manual, the VLAN range is 2-10 (mlnx_agent.conf /ml2_conf.ini). Therefore, add the following lines:<br /> vlan1=0x1, ipoib, sl=0, defmember=full : ALL;<br /> vlan2=0x2, ipoib, sl=0, defmember=full : ALL;<br /> vlan3=0x3, ipoib, sl=0, defmember=full : ALL;<br /> vlan4=0x4, ipoib, sl=0, defmember=full : ALL;<br /> vlan5=0x5, ipoib, sl=0, defmember=full : ALL;<br /> vlan6=0x6, ipoib, sl=0, defmember=full : ALL;<br /> vlan7=0x7, ipoib, sl=0, defmember=full : ALL;<br /> vlan8=0x8, ipoib, sl=0, defmember=full : ALL;<br /> vlan9=0x9, ipoib, sl=0, defmember=full : ALL;<br /> vlan10=0xa, ipoib, sl=0, defmember=full : ALL;<br /> 2. Modify the following line in the file /etc/opensm/opensm.conf from FALSE to TRUE:<br /> allow_both_pkeys TRUE<br /> 3. Restart OpenSM:<br /> service opensmd restart<br /> 4. Verify that all ports on all nodes have their ib interface (i.e. ib0) active, run ibdev2netdev on all hosts, and make sure that the port is up:<br /> mlx4_0 port 1 ==&gt; ib0 (Up)<br /> If the port is down, try bringing it up by running “ifconfig ib0 up”, and make sure your switch ports are of type “InfiniBand”, and that all nodes can reach the OpenSM node by running “sminfo”.<br /> <br /> == Controller Node == <br /> <br /> To configure the Controller node: <br /> <br /> 1. Ensure that the file /etc/nova/nova.conf includes the following line:<br /> security_group_api=nova<br /> <br /> 2. If the security group value is changed, restart the Nova services:<br /> # for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done<br /> <br /> === Neutron Server Node ===<br /> <br /> NOTE: The Neutron Server can sometimes be part of the Controller node.<br /> <br /> To configure the Neutron Server node: <br /> <br /> 1. Edit /etc/neutron/neutron.conf and make sure Neutron is using ML2 plugin:<br /> [DEFAULT]<br /> core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin<br /> 2. Edit the ML2 configuration file /etc/neutron/plugins/ml2/ml2_conf.ini as follows:<br /> [ml2]<br /> type_drivers = vlan,flat<br /> tenant_network_types = vlan <br /> mechanism_drivers = linuxbridge,mlnx<br /> <br /> [ml2_type_vlan]<br /> network_vlan_ranges=&lt;physical_network&gt;:&lt;min_vlan&gt;:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> <br /> [securitygroup]<br /> enable_security_group = True<br /> <br /> [eswitch]<br /> vnic_type = hostdev<br /> apply_profile_patch = True<br /> 3. Install the required Deb for the Mellanox Neutron plugin (from Ubuntu’s repository):<br /> apt-get install neutron-plugin-mlnx-agent<br /> 4. Edit /etc/neutron/plugins/mlnx/mlnx_conf.ini to match your environment:<br /> [mlnx]<br /> network_vlan_ranges = &lt;physical_network&gt;:&lt;min_vlan:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> tenant_network_type = vlan<br /> <br /> [eswitch]<br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;ib interface&gt; //(i.e. default:ib0)<br /> vnic_type = hostdev<br /> <br /> [agent]<br /> support_old_agents = True<br /> <br /> 5. Restart Neutron Server: <br /> /etc/init.d/neutron-server start<br /> <br /> == Compute Nodes ==<br /> <br /> NOTE: The Neutron Server can sometimes be part of the Controller node.<br /> <br /> To configure the Neutron Server node: <br /> <br /> 1. Edit /etc/neutron/neutron.conf and make sure Neutron is using ML2 plugin:<br /> [DEFAULT]<br /> core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin<br /> 2. Edit the ML2 configuration file /etc/neutron/plugins/ml2/ml2_conf.ini as follows:<br /> [ml2]<br /> type_drivers = vlan,flat<br /> tenant_network_types = vlan<br /> mechanism_drivers = linuxbridge,mlnx<br /> <br /> [ml2_type_vlan]<br /> network_vlan_ranges=&lt;physical_network&gt;:&lt;min_vlan&gt;:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> <br /> [securitygroup]<br /> enable_security_group = True<br /> <br /> [eswitch]<br /> vnic_type = hostdev<br /> apply_profile_patch = True<br /> 3. Install the required Deb for the Mellanox Neutron plugin (from Ubuntu’s repository):<br /> apt-get install neutron-plugin-mlnx-agent<br /> 4. Edit /etc/neutron/plugins/mlnx/mlnx_conf.ini to match your environment:<br /> [mlnx]<br /> network_vlan_ranges = &lt;physical_network&gt;:&lt;min_vlan:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> tenant_network_type = vlan<br /> <br /> [eswitch]<br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;ib interface&gt; //(i.e. default:ib0)<br /> vnic_type = hostdev<br /> <br /> [agent]<br /> support_old_agents = True<br /> <br /> 5. Restart Neutron Server: <br /> /etc/init.d/neutron-server start<br /> <br /> == Network Node ==<br /> <br /> 1. Install the Neutron Linux bridge plugin, the Neutron DHCP agent and the Neutron L3 agent:<br /> <br /> # apt-get install neutron-plugin-linuxbridge-agent neutron-dhcp-agent neutron-l3-agent<br /> <br /> 2. Copy /etc/neutron/neutron.conf from the Compute node.<br /> <br /> 3. Change the following configuration of the ini file (/etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini). The &quot;default&quot; in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini . <br /> [linux_bridge] <br /> physical_interface_mappings = default:eth2 <br /> <br /> 4. Configure the DHCP agent according to the following guidelines <br /> <br /> Update the following file: /etc/neutron/dhcp_agent.ini with:<br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver <br /> <br /> For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_dhcp_agent OpenStack documentation]<br /> <br /> 5. Configure the L3 agent according to the following guidelines <br /> <br /> Update the following file: /etc/neutron/l3_agent.ini with:<br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver <br /> <br /> For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_l3_agent OpenStack documentation]<br /> <br /> 6. Configure the Metadata agent according to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#nova_config_metadata OpenStack documentation]<br /> <br /> 7. Start the Neutron Linux Bridge, DHCP and L3 agents <br /> # restart neutron-plugin-linuxbridge-agent<br /> # restart neutron-dhcp-agent<br /> # restart neutron-l3-agent<br /> # restart neutron-metadata-agent<br /> <br /> == Troubleshooting ==<br /> <br /> * If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand&diff=69075 2014-11-26T16:50:38Z

<p>May: /* Compute Nodes */</p> <hr /> <div>= Overview = <br /> <br /> This wiki page describes how to tune Mellanox OpenStack on Ubuntu 12.04 servers<br /> <br /> = Prerequisites =<br /> <br /> * A running OpenStack environment installed with the ML2 plugin on top of OVS.<br /> * All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119) <br /> * Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers<br /> * SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community [http://community.mellanox.com/docs/DOC-1317 click here].<br /> * The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes<br /> * VLANs configured on the ports in the switch.<br /> <br /> = InfiniBand Network = <br /> <br /> == OpenSM Node ==<br /> <br /> To configure OpenSM node: <br /> <br /> 1. Edit the file /etc/opensm/partitions.conf by adding the following line:<br /> management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;<br /> Create the file if it does not exist. <br /> <br /> For each VLAN value in the VLAN range used for VLAN network segmentation, add the following line in the file /etc/opensm/partitions.conf (for vlan-pkey association):<br /> vlan&lt;decimal-value&gt;=0x&lt;hexadecimal-value&gt;, ipoib, sl=0, defmember=full: ALL;<br /> In this manual, the VLAN range is 2-10 (mlnx_agent.conf /ml2_conf.ini). Therefore, add the following lines:<br /> vlan1=0x1, ipoib, sl=0, defmember=full : ALL;<br /> vlan2=0x2, ipoib, sl=0, defmember=full : ALL;<br /> vlan3=0x3, ipoib, sl=0, defmember=full : ALL;<br /> vlan4=0x4, ipoib, sl=0, defmember=full : ALL;<br /> vlan5=0x5, ipoib, sl=0, defmember=full : ALL;<br /> vlan6=0x6, ipoib, sl=0, defmember=full : ALL;<br /> vlan7=0x7, ipoib, sl=0, defmember=full : ALL;<br /> vlan8=0x8, ipoib, sl=0, defmember=full : ALL;<br /> vlan9=0x9, ipoib, sl=0, defmember=full : ALL;<br /> vlan10=0xa, ipoib, sl=0, defmember=full : ALL;<br /> 2. Modify the following line in the file /etc/opensm/opensm.conf from FALSE to TRUE:<br /> allow_both_pkeys TRUE<br /> 3. Restart OpenSM:<br /> service opensmd restart<br /> 4. Verify that all ports on all nodes have their ib interface (i.e. ib0) active, run ibdev2netdev on all hosts, and make sure that the port is up:<br /> mlx4_0 port 1 ==&gt; ib0 (Up)<br /> If the port is down, try bringing it up by running “ifconfig ib0 up”, and make sure your switch ports are of type “InfiniBand”, and that all nodes can reach the OpenSM node by running “sminfo”.<br /> <br /> == Controller Node == <br /> <br /> To configure the Controller node: <br /> <br /> 1. Ensure that the file /etc/nova/nova.conf includes the following line:<br /> security_group_api=nova<br /> <br /> 2. If the security group value is changed, restart the Nova services:<br /> # for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done<br /> <br /> === Neutron Server Node ===<br /> <br /> NOTE: The Neutron Server can sometimes be part of the Controller node.<br /> <br /> To configure the Neutron Server node: <br /> <br /> 1. Edit /etc/neutron/neutron.conf and make sure Neutron is using ML2 plugin:<br /> [DEFAULT]<br /> core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin<br /> 2. Edit the ML2 configuration file /etc/neutron/plugins/ml2/ml2_conf.ini as follows:<br /> [ml2]<br /> type_drivers = vlan,flat<br /> tenant_network_types = vlan <br /> mechanism_drivers = linuxbridge,mlnx<br /> <br /> [ml2_type_vlan]<br /> network_vlan_ranges=&lt;physical_network&gt;:&lt;min_vlan&gt;:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> <br /> [securitygroup]<br /> enable_security_group = True<br /> <br /> [eswitch]<br /> vnic_type = hostdev<br /> apply_profile_patch = True<br /> 3. Install the required Deb for the Mellanox Neutron plugin (from Ubuntu’s repository):<br /> apt-get install neutron-plugin-mlnx-agent<br /> 4. Edit /etc/neutron/plugins/mlnx/mlnx_conf.ini to match your environment:<br /> [mlnx]<br /> network_vlan_ranges = &lt;physical_network&gt;:&lt;min_vlan:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> tenant_network_type = vlan<br /> <br /> [eswitch]<br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;ib interface&gt; //(i.e. default:ib0)<br /> vnic_type = hostdev<br /> <br /> [agent]<br /> support_old_agents = True<br /> <br /> 5. Restart Neutron Server: <br /> /etc/init.d/neutron-server start<br /> <br /> == Compute Nodes ==<br /> <br /> NOTE: The Neutron Server can sometimes be part of the Controller node.<br /> <br /> To configure the Neutron Server node: <br /> 1. Edit /etc/neutron/neutron.conf and make sure Neutron is using ML2 plugin:<br /> [DEFAULT]<br /> core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin<br /> 2. Edit the ML2 configuration file /etc/neutron/plugins/ml2/ml2_conf.ini as follows:<br /> [ml2]<br /> type_drivers = vlan,flat<br /> tenant_network_types = vlan<br /> mechanism_drivers = linuxbridge,mlnx<br /> <br /> [ml2_type_vlan]<br /> network_vlan_ranges=&lt;physical_network&gt;:&lt;min_vlan&gt;:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> <br /> [securitygroup]<br /> enable_security_group = True<br /> <br /> [eswitch]<br /> vnic_type = hostdev<br /> apply_profile_patch = True<br /> 3. Install the required Deb for the Mellanox Neutron plugin (from Ubuntu’s repository):<br /> apt-get install neutron-plugin-mlnx-agent<br /> 4. Edit /etc/neutron/plugins/mlnx/mlnx_conf.ini to match your environment:<br /> [mlnx]<br /> network_vlan_ranges = &lt;physical_network&gt;:&lt;min_vlan:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> tenant_network_type = vlan<br /> <br /> [eswitch]<br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;ib interface&gt; //(i.e. default:ib0)<br /> vnic_type = hostdev<br /> <br /> [agent]<br /> support_old_agents = True<br /> <br /> 5. Restart Neutron Server: <br /> /etc/init.d/neutron-server start<br /> <br /> == Network Node ==<br /> <br /> 1. Install the Neutron Linux bridge plugin, the Neutron DHCP agent and the Neutron L3 agent:<br /> <br /> # apt-get install neutron-plugin-linuxbridge-agent neutron-dhcp-agent neutron-l3-agent<br /> <br /> 2. Copy /etc/neutron/neutron.conf from the Compute node.<br /> <br /> 3. Change the following configuration of the ini file (/etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini). The &quot;default&quot; in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini . <br /> [linux_bridge] <br /> physical_interface_mappings = default:eth2 <br /> <br /> 4. Configure the DHCP agent according to the following guidelines <br /> <br /> Update the following file: /etc/neutron/dhcp_agent.ini with:<br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver <br /> <br /> For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_dhcp_agent OpenStack documentation]<br /> <br /> 5. Configure the L3 agent according to the following guidelines <br /> <br /> Update the following file: /etc/neutron/l3_agent.ini with:<br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver <br /> <br /> For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_l3_agent OpenStack documentation]<br /> <br /> 6. Configure the Metadata agent according to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#nova_config_metadata OpenStack documentation]<br /> <br /> 7. Start the Neutron Linux Bridge, DHCP and L3 agents <br /> # restart neutron-plugin-linuxbridge-agent<br /> # restart neutron-dhcp-agent<br /> # restart neutron-l3-agent<br /> # restart neutron-metadata-agent<br /> <br /> == Troubleshooting ==<br /> <br /> * If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand&diff=69074 2014-11-26T16:48:56Z

<p>May: /* Neutron Server Node */</p> <hr /> <div>= Overview = <br /> <br /> This wiki page describes how to tune Mellanox OpenStack on Ubuntu 12.04 servers<br /> <br /> = Prerequisites =<br /> <br /> * A running OpenStack environment installed with the ML2 plugin on top of OVS.<br /> * All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119) <br /> * Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers<br /> * SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community [http://community.mellanox.com/docs/DOC-1317 click here].<br /> * The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes<br /> * VLANs configured on the ports in the switch.<br /> <br /> = InfiniBand Network = <br /> <br /> == OpenSM Node ==<br /> <br /> To configure OpenSM node: <br /> <br /> 1. Edit the file /etc/opensm/partitions.conf by adding the following line:<br /> management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;<br /> Create the file if it does not exist. <br /> <br /> For each VLAN value in the VLAN range used for VLAN network segmentation, add the following line in the file /etc/opensm/partitions.conf (for vlan-pkey association):<br /> vlan&lt;decimal-value&gt;=0x&lt;hexadecimal-value&gt;, ipoib, sl=0, defmember=full: ALL;<br /> In this manual, the VLAN range is 2-10 (mlnx_agent.conf /ml2_conf.ini). Therefore, add the following lines:<br /> vlan1=0x1, ipoib, sl=0, defmember=full : ALL;<br /> vlan2=0x2, ipoib, sl=0, defmember=full : ALL;<br /> vlan3=0x3, ipoib, sl=0, defmember=full : ALL;<br /> vlan4=0x4, ipoib, sl=0, defmember=full : ALL;<br /> vlan5=0x5, ipoib, sl=0, defmember=full : ALL;<br /> vlan6=0x6, ipoib, sl=0, defmember=full : ALL;<br /> vlan7=0x7, ipoib, sl=0, defmember=full : ALL;<br /> vlan8=0x8, ipoib, sl=0, defmember=full : ALL;<br /> vlan9=0x9, ipoib, sl=0, defmember=full : ALL;<br /> vlan10=0xa, ipoib, sl=0, defmember=full : ALL;<br /> 2. Modify the following line in the file /etc/opensm/opensm.conf from FALSE to TRUE:<br /> allow_both_pkeys TRUE<br /> 3. Restart OpenSM:<br /> service opensmd restart<br /> 4. Verify that all ports on all nodes have their ib interface (i.e. ib0) active, run ibdev2netdev on all hosts, and make sure that the port is up:<br /> mlx4_0 port 1 ==&gt; ib0 (Up)<br /> If the port is down, try bringing it up by running “ifconfig ib0 up”, and make sure your switch ports are of type “InfiniBand”, and that all nodes can reach the OpenSM node by running “sminfo”.<br /> <br /> == Controller Node == <br /> <br /> To configure the Controller node: <br /> <br /> 1. Ensure that the file /etc/nova/nova.conf includes the following line:<br /> security_group_api=nova<br /> <br /> 2. If the security group value is changed, restart the Nova services:<br /> # for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done<br /> <br /> === Neutron Server Node ===<br /> <br /> NOTE: The Neutron Server can sometimes be part of the Controller node.<br /> <br /> To configure the Neutron Server node: <br /> <br /> 1. Edit /etc/neutron/neutron.conf and make sure Neutron is using ML2 plugin:<br /> [DEFAULT]<br /> core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin<br /> 2. Edit the ML2 configuration file /etc/neutron/plugins/ml2/ml2_conf.ini as follows:<br /> [ml2]<br /> type_drivers = vlan,flat<br /> tenant_network_types = vlan <br /> mechanism_drivers = linuxbridge,mlnx<br /> <br /> [ml2_type_vlan]<br /> network_vlan_ranges=&lt;physical_network&gt;:&lt;min_vlan&gt;:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> <br /> [securitygroup]<br /> enable_security_group = True<br /> <br /> [eswitch]<br /> vnic_type = hostdev<br /> apply_profile_patch = True<br /> 3. Install the required Deb for the Mellanox Neutron plugin (from Ubuntu’s repository):<br /> apt-get install neutron-plugin-mlnx-agent<br /> 4. Edit /etc/neutron/plugins/mlnx/mlnx_conf.ini to match your environment:<br /> [mlnx]<br /> network_vlan_ranges = &lt;physical_network&gt;:&lt;min_vlan:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> tenant_network_type = vlan<br /> <br /> [eswitch]<br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;ib interface&gt; //(i.e. default:ib0)<br /> vnic_type = hostdev<br /> <br /> [agent]<br /> support_old_agents = True<br /> <br /> 5. Restart Neutron Server: <br /> /etc/init.d/neutron-server start<br /> <br /> == Compute Nodes ==<br /> <br /> === Installation ===<br /> <br /> 1. If you didn't download Mellanox OpenStack debian files download it : <br /> <br /> http://www.mellanox.com/downloads/solutions/openstack/<br /> <br /> 2. Install the eswitchd debian files: <br /> dpkg -i eswitchd_0.8-1_amd64.deb<br /> dpkg -i python-mlnxvif_0.6-1_all.deb<br /> apt-get install -f # fix dependencies<br /> apt-get install -y neutron-plugin-mlnx-agent<br /> <br /> 3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)<br /> <br /> === Configuration === <br /> <br /> 1. Configure /etc/eswitchd/eswitchd.conf if needed <br /> <br /> Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here]) <br /> <br /> 2. Modify /etc/nova/nova.conf <br /> <br /> compute_driver=nova.virt.libvirt.driver.LibvirtDriver<br /> libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver<br /> security_group_api=nova<br /> connection_type=libvirt<br /> <br /> In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following: <br /> <br /> libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver<br /> <br /> 3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment. <br /> [AGENT]<br /> rpc_support_old_agents = True<br /> [ESWITCH]<br /> physical_interface_mapping - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: &lt;fabric name&gt;:&lt; PF name&gt; (Only releavant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is &quot;default:autoeth&quot;. <br /> vnic_type - type of VM network interface: 'mlnx_direct' or 'hostdev' according to libvirt terminology.<br /> hostdev: this is the traditional method of assigning any generic PCI device to a guest (SR-IOV).<br /> mlnx_direct: this is a method to provide macvtap device on top of the PCI device (default).<br /> bridge: - When using Linux Bridge Plugin on top of eIPoIB device<br /> daemon_endpoint - 'tcp://127.0.0.1:60001'<br /> <br /> For a plugin configuration file example (Havana), please refer to Mellanox config *ini file (click [https://github.com/openstack/neutron/blob/stable/havana/etc/neutron/plugins/mlnx/mlnx_conf.ini here])<br /> <br /> 4. (Optional) Modify /etc/init/neutron-plugin-mlnx-agent.conf as described in LP bug #[https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1257732 1257732]<br /> <br /> 5. Check that ebrctl is listed somewhere in /etc/nova/rootwrap.d/*, otherwise add this to /etc/nova/rootwrap.d/compute.filters file in the [Filters] section.<br /> ebrctl: CommandFilter, ebrctl, root<br /> <br /> === Start Services ===<br /> <br /> 1. Restart Nova. <br /> <br /> #/etc/init.d/nova-compute restart<br /> <br /> 2. Start eswitch Daemon <br /> <br /> #/etc/init.d/eswitchd start<br /> <br /> 3. Start the Neutron agent <br /> <br /> #/etc/init.d/neutron-plugin-mlnx-agent start<br /> <br /> Note: eswitch Daemon should be running before the Neutron Agent is started.<br /> <br /> == Network Node ==<br /> <br /> 1. Install the Neutron Linux bridge plugin, the Neutron DHCP agent and the Neutron L3 agent:<br /> <br /> # apt-get install neutron-plugin-linuxbridge-agent neutron-dhcp-agent neutron-l3-agent<br /> <br /> 2. Copy /etc/neutron/neutron.conf from the Compute node.<br /> <br /> 3. Change the following configuration of the ini file (/etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini). The &quot;default&quot; in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini . <br /> [linux_bridge] <br /> physical_interface_mappings = default:eth2 <br /> <br /> 4. Configure the DHCP agent according to the following guidelines <br /> <br /> Update the following file: /etc/neutron/dhcp_agent.ini with:<br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver <br /> <br /> For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_dhcp_agent OpenStack documentation]<br /> <br /> 5. Configure the L3 agent according to the following guidelines <br /> <br /> Update the following file: /etc/neutron/l3_agent.ini with:<br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver <br /> <br /> For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_l3_agent OpenStack documentation]<br /> <br /> 6. Configure the Metadata agent according to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#nova_config_metadata OpenStack documentation]<br /> <br /> 7. Start the Neutron Linux Bridge, DHCP and L3 agents <br /> # restart neutron-plugin-linuxbridge-agent<br /> # restart neutron-dhcp-agent<br /> # restart neutron-l3-agent<br /> # restart neutron-metadata-agent<br /> <br /> == Troubleshooting ==<br /> <br /> * If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand&diff=69073 2014-11-26T16:48:24Z

<p>May: /* Neutron Server Node */</p> <hr /> <div>= Overview = <br /> <br /> This wiki page describes how to tune Mellanox OpenStack on Ubuntu 12.04 servers<br /> <br /> = Prerequisites =<br /> <br /> * A running OpenStack environment installed with the ML2 plugin on top of OVS.<br /> * All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119) <br /> * Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers<br /> * SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community [http://community.mellanox.com/docs/DOC-1317 click here].<br /> * The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes<br /> * VLANs configured on the ports in the switch.<br /> <br /> = InfiniBand Network = <br /> <br /> == OpenSM Node ==<br /> <br /> To configure OpenSM node: <br /> <br /> 1. Edit the file /etc/opensm/partitions.conf by adding the following line:<br /> management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;<br /> Create the file if it does not exist. <br /> <br /> For each VLAN value in the VLAN range used for VLAN network segmentation, add the following line in the file /etc/opensm/partitions.conf (for vlan-pkey association):<br /> vlan&lt;decimal-value&gt;=0x&lt;hexadecimal-value&gt;, ipoib, sl=0, defmember=full: ALL;<br /> In this manual, the VLAN range is 2-10 (mlnx_agent.conf /ml2_conf.ini). Therefore, add the following lines:<br /> vlan1=0x1, ipoib, sl=0, defmember=full : ALL;<br /> vlan2=0x2, ipoib, sl=0, defmember=full : ALL;<br /> vlan3=0x3, ipoib, sl=0, defmember=full : ALL;<br /> vlan4=0x4, ipoib, sl=0, defmember=full : ALL;<br /> vlan5=0x5, ipoib, sl=0, defmember=full : ALL;<br /> vlan6=0x6, ipoib, sl=0, defmember=full : ALL;<br /> vlan7=0x7, ipoib, sl=0, defmember=full : ALL;<br /> vlan8=0x8, ipoib, sl=0, defmember=full : ALL;<br /> vlan9=0x9, ipoib, sl=0, defmember=full : ALL;<br /> vlan10=0xa, ipoib, sl=0, defmember=full : ALL;<br /> 2. Modify the following line in the file /etc/opensm/opensm.conf from FALSE to TRUE:<br /> allow_both_pkeys TRUE<br /> 3. Restart OpenSM:<br /> service opensmd restart<br /> 4. Verify that all ports on all nodes have their ib interface (i.e. ib0) active, run ibdev2netdev on all hosts, and make sure that the port is up:<br /> mlx4_0 port 1 ==&gt; ib0 (Up)<br /> If the port is down, try bringing it up by running “ifconfig ib0 up”, and make sure your switch ports are of type “InfiniBand”, and that all nodes can reach the OpenSM node by running “sminfo”.<br /> <br /> == Controller Node == <br /> <br /> To configure the Controller node: <br /> <br /> 1. Ensure that the file /etc/nova/nova.conf includes the following line:<br /> security_group_api=nova<br /> <br /> 2. If the security group value is changed, restart the Nova services:<br /> # for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done<br /> <br /> == Neutron Server Node ==<br /> <br /> NOTE: The Neutron Server can sometimes be part of the Controller node.<br /> <br /> To configure the Neutron Server node: <br /> <br /> 1. Edit /etc/neutron/neutron.conf and make sure Neutron is using ML2 plugin:<br /> [DEFAULT]<br /> core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin<br /> 2. Edit the ML2 configuration file /etc/neutron/plugins/ml2/ml2_conf.ini as follows:<br /> [ml2]<br /> type_drivers = vlan,flat<br /> tenant_network_types = vlan <br /> mechanism_drivers = linuxbridge,mlnx<br /> <br /> [ml2_type_vlan]<br /> network_vlan_ranges=&lt;physical_network&gt;:&lt;min_vlan&gt;:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> <br /> [securitygroup]<br /> enable_security_group = True<br /> <br /> [eswitch]<br /> vnic_type = hostdev<br /> apply_profile_patch = True<br /> 3. Install the required Deb for the Mellanox Neutron plugin (from Ubuntu’s repository):<br /> apt-get install neutron-plugin-mlnx-agent<br /> 4. Edit /etc/neutron/plugins/mlnx/mlnx_conf.ini to match your environment:<br /> [mlnx]<br /> network_vlan_ranges = &lt;physical_network&gt;:&lt;min_vlan:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> tenant_network_type = vlan<br /> <br /> [eswitch]<br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;ib interface&gt; //(i.e. default:ib0)<br /> vnic_type = hostdev<br /> <br /> [agent]<br /> support_old_agents = True<br /> <br /> 5. Restart Neutron Server: <br /> /etc/init.d/neutron-server start<br /> <br /> == Compute Nodes ==<br /> <br /> === Installation ===<br /> <br /> 1. If you didn't download Mellanox OpenStack debian files download it : <br /> <br /> http://www.mellanox.com/downloads/solutions/openstack/<br /> <br /> 2. Install the eswitchd debian files: <br /> dpkg -i eswitchd_0.8-1_amd64.deb<br /> dpkg -i python-mlnxvif_0.6-1_all.deb<br /> apt-get install -f # fix dependencies<br /> apt-get install -y neutron-plugin-mlnx-agent<br /> <br /> 3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)<br /> <br /> === Configuration === <br /> <br /> 1. Configure /etc/eswitchd/eswitchd.conf if needed <br /> <br /> Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here]) <br /> <br /> 2. Modify /etc/nova/nova.conf <br /> <br /> compute_driver=nova.virt.libvirt.driver.LibvirtDriver<br /> libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver<br /> security_group_api=nova<br /> connection_type=libvirt<br /> <br /> In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following: <br /> <br /> libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver<br /> <br /> 3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment. <br /> [AGENT]<br /> rpc_support_old_agents = True<br /> [ESWITCH]<br /> physical_interface_mapping - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: &lt;fabric name&gt;:&lt; PF name&gt; (Only releavant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is &quot;default:autoeth&quot;. <br /> vnic_type - type of VM network interface: 'mlnx_direct' or 'hostdev' according to libvirt terminology.<br /> hostdev: this is the traditional method of assigning any generic PCI device to a guest (SR-IOV).<br /> mlnx_direct: this is a method to provide macvtap device on top of the PCI device (default).<br /> bridge: - When using Linux Bridge Plugin on top of eIPoIB device<br /> daemon_endpoint - 'tcp://127.0.0.1:60001'<br /> <br /> For a plugin configuration file example (Havana), please refer to Mellanox config *ini file (click [https://github.com/openstack/neutron/blob/stable/havana/etc/neutron/plugins/mlnx/mlnx_conf.ini here])<br /> <br /> 4. (Optional) Modify /etc/init/neutron-plugin-mlnx-agent.conf as described in LP bug #[https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1257732 1257732]<br /> <br /> 5. Check that ebrctl is listed somewhere in /etc/nova/rootwrap.d/*, otherwise add this to /etc/nova/rootwrap.d/compute.filters file in the [Filters] section.<br /> ebrctl: CommandFilter, ebrctl, root<br /> <br /> === Start Services ===<br /> <br /> 1. Restart Nova. <br /> <br /> #/etc/init.d/nova-compute restart<br /> <br /> 2. Start eswitch Daemon <br /> <br /> #/etc/init.d/eswitchd start<br /> <br /> 3. Start the Neutron agent <br /> <br /> #/etc/init.d/neutron-plugin-mlnx-agent start<br /> <br /> Note: eswitch Daemon should be running before the Neutron Agent is started.<br /> <br /> == Network Node ==<br /> <br /> 1. Install the Neutron Linux bridge plugin, the Neutron DHCP agent and the Neutron L3 agent:<br /> <br /> # apt-get install neutron-plugin-linuxbridge-agent neutron-dhcp-agent neutron-l3-agent<br /> <br /> 2. Copy /etc/neutron/neutron.conf from the Compute node.<br /> <br /> 3. Change the following configuration of the ini file (/etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini). The &quot;default&quot; in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini . <br /> [linux_bridge] <br /> physical_interface_mappings = default:eth2 <br /> <br /> 4. Configure the DHCP agent according to the following guidelines <br /> <br /> Update the following file: /etc/neutron/dhcp_agent.ini with:<br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver <br /> <br /> For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_dhcp_agent OpenStack documentation]<br /> <br /> 5. Configure the L3 agent according to the following guidelines <br /> <br /> Update the following file: /etc/neutron/l3_agent.ini with:<br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver <br /> <br /> For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_l3_agent OpenStack documentation]<br /> <br /> 6. Configure the Metadata agent according to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#nova_config_metadata OpenStack documentation]<br /> <br /> 7. Start the Neutron Linux Bridge, DHCP and L3 agents <br /> # restart neutron-plugin-linuxbridge-agent<br /> # restart neutron-dhcp-agent<br /> # restart neutron-l3-agent<br /> # restart neutron-metadata-agent<br /> <br /> == Troubleshooting ==<br /> <br /> * If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand&diff=69072 2014-11-26T16:46:39Z

<p>May: /* Controller Node */</p> <hr /> <div>= Overview = <br /> <br /> This wiki page describes how to tune Mellanox OpenStack on Ubuntu 12.04 servers<br /> <br /> = Prerequisites =<br /> <br /> * A running OpenStack environment installed with the ML2 plugin on top of OVS.<br /> * All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119) <br /> * Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers<br /> * SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community [http://community.mellanox.com/docs/DOC-1317 click here].<br /> * The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes<br /> * VLANs configured on the ports in the switch.<br /> <br /> = InfiniBand Network = <br /> <br /> == OpenSM Node ==<br /> <br /> To configure OpenSM node: <br /> <br /> 1. Edit the file /etc/opensm/partitions.conf by adding the following line:<br /> management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;<br /> Create the file if it does not exist. <br /> <br /> For each VLAN value in the VLAN range used for VLAN network segmentation, add the following line in the file /etc/opensm/partitions.conf (for vlan-pkey association):<br /> vlan&lt;decimal-value&gt;=0x&lt;hexadecimal-value&gt;, ipoib, sl=0, defmember=full: ALL;<br /> In this manual, the VLAN range is 2-10 (mlnx_agent.conf /ml2_conf.ini). Therefore, add the following lines:<br /> vlan1=0x1, ipoib, sl=0, defmember=full : ALL;<br /> vlan2=0x2, ipoib, sl=0, defmember=full : ALL;<br /> vlan3=0x3, ipoib, sl=0, defmember=full : ALL;<br /> vlan4=0x4, ipoib, sl=0, defmember=full : ALL;<br /> vlan5=0x5, ipoib, sl=0, defmember=full : ALL;<br /> vlan6=0x6, ipoib, sl=0, defmember=full : ALL;<br /> vlan7=0x7, ipoib, sl=0, defmember=full : ALL;<br /> vlan8=0x8, ipoib, sl=0, defmember=full : ALL;<br /> vlan9=0x9, ipoib, sl=0, defmember=full : ALL;<br /> vlan10=0xa, ipoib, sl=0, defmember=full : ALL;<br /> 2. Modify the following line in the file /etc/opensm/opensm.conf from FALSE to TRUE:<br /> allow_both_pkeys TRUE<br /> 3. Restart OpenSM:<br /> service opensmd restart<br /> 4. Verify that all ports on all nodes have their ib interface (i.e. ib0) active, run ibdev2netdev on all hosts, and make sure that the port is up:<br /> mlx4_0 port 1 ==&gt; ib0 (Up)<br /> If the port is down, try bringing it up by running “ifconfig ib0 up”, and make sure your switch ports are of type “InfiniBand”, and that all nodes can reach the OpenSM node by running “sminfo”.<br /> <br /> == Controller Node == <br /> <br /> To configure the Controller node: <br /> <br /> 1. Ensure that the file /etc/nova/nova.conf includes the following line:<br /> security_group_api=nova<br /> <br /> 2. If the security group value is changed, restart the Nova services:<br /> # for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done<br /> <br /> == Neutron Server Node ==<br /> === Installation === <br /> <br /> 1. Neutron server uses MySQL database. Make sure you have running MySQL database for neutron. <br /> <br /> If Neutron server is already running, you should stop it. <br /> <br /> #/etc/init.d/neutron-server stop<br /> <br /> If you want to use a database that was created you have to drop it and create another one. For example: <br /> <br /> mysql -u root -ppassword &lt;&lt;EOF<br /> drop database if exists neutron;<br /> create database neutron;<br /> EOF<br /> <br /> If you want to create a new one: <br /> <br /> mysql -u root -ppassword &lt;&lt;EOF<br /> drop database if exists neutron;<br /> create database neutron;<br /> GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'localhost' IDENTIFIED BY 'password';<br /> GRANT ALL PRIVILEGES ON neutron.* TO 'neutron' IDENTIFIED BY 'password';<br /> FLUSH PRIVILEGES;<br /> EOF<br /> <br /> 2. Install the required Deb for the Mellanox Neutron plugin: <br /> <br /> apt-get install -y neutron-plugin-mlnx-agent<br /> <br /> 3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment. <br /> <br /> 4. Change /etc/default/neutron-server :<br /> <br /> NEUTRON_PLUGIN_CONFIG=&quot;/etc/neutron/plugins/mlnx/mlnx_conf.ini&quot;<br /> <br /> === Configuration ===<br /> <br /> 1. Make the Mellanox plugin the current Neutron plugin by editing neutron.conf and changing core_plugin. <br /> <br /> core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin<br /> <br /> 2. Plugin configuration: Edit the configuration file: /etc/neutron/plugins/mlnx/mlnx_conf.ini <br /> <br /> [MLNX]<br /> tenant_network_type - must be set on of supported tenant network types. <br /> Possible values: 'vlan' for Ethernet or 'ib' for Infiniband.'vlan' is the default.<br /> network_vlan_ranges - must be configured to specify the names of the physical <br /> <br /> networks managed by the mellanox plugin, <br /> along with the ranges of VLAN IDs available on each physical network for allocation to virtual networks. <br /> Possible range is vlan range is 1-4093. <br /> The default is &quot;default:1:100&quot; <br /> &lt;fabric name &gt;:&lt;vlan range start&gt;:&lt;vlan end range&gt;<br /> <br /> [agent]<br /> rpc_support_old_agents = True<br /> Note: rpc_support_old_agents should be set to 'True' (non default).<br /> <br /> === Start Services ===<br /> <br /> Start (or restart) the Neutron server <br /> #/etc/init.d/neutron-server start<br /> <br /> == Compute Nodes ==<br /> <br /> === Installation ===<br /> <br /> 1. If you didn't download Mellanox OpenStack debian files download it : <br /> <br /> http://www.mellanox.com/downloads/solutions/openstack/<br /> <br /> 2. Install the eswitchd debian files: <br /> dpkg -i eswitchd_0.8-1_amd64.deb<br /> dpkg -i python-mlnxvif_0.6-1_all.deb<br /> apt-get install -f # fix dependencies<br /> apt-get install -y neutron-plugin-mlnx-agent<br /> <br /> 3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)<br /> <br /> === Configuration === <br /> <br /> 1. Configure /etc/eswitchd/eswitchd.conf if needed <br /> <br /> Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here]) <br /> <br /> 2. Modify /etc/nova/nova.conf <br /> <br /> compute_driver=nova.virt.libvirt.driver.LibvirtDriver<br /> libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver<br /> security_group_api=nova<br /> connection_type=libvirt<br /> <br /> In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following: <br /> <br /> libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver<br /> <br /> 3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment. <br /> [AGENT]<br /> rpc_support_old_agents = True<br /> [ESWITCH]<br /> physical_interface_mapping - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: &lt;fabric name&gt;:&lt; PF name&gt; (Only releavant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is &quot;default:autoeth&quot;. <br /> vnic_type - type of VM network interface: 'mlnx_direct' or 'hostdev' according to libvirt terminology.<br /> hostdev: this is the traditional method of assigning any generic PCI device to a guest (SR-IOV).<br /> mlnx_direct: this is a method to provide macvtap device on top of the PCI device (default).<br /> bridge: - When using Linux Bridge Plugin on top of eIPoIB device<br /> daemon_endpoint - 'tcp://127.0.0.1:60001'<br /> <br /> For a plugin configuration file example (Havana), please refer to Mellanox config *ini file (click [https://github.com/openstack/neutron/blob/stable/havana/etc/neutron/plugins/mlnx/mlnx_conf.ini here])<br /> <br /> 4. (Optional) Modify /etc/init/neutron-plugin-mlnx-agent.conf as described in LP bug #[https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1257732 1257732]<br /> <br /> 5. Check that ebrctl is listed somewhere in /etc/nova/rootwrap.d/*, otherwise add this to /etc/nova/rootwrap.d/compute.filters file in the [Filters] section.<br /> ebrctl: CommandFilter, ebrctl, root<br /> <br /> === Start Services ===<br /> <br /> 1. Restart Nova. <br /> <br /> #/etc/init.d/nova-compute restart<br /> <br /> 2. Start eswitch Daemon <br /> <br /> #/etc/init.d/eswitchd start<br /> <br /> 3. Start the Neutron agent <br /> <br /> #/etc/init.d/neutron-plugin-mlnx-agent start<br /> <br /> Note: eswitch Daemon should be running before the Neutron Agent is started.<br /> <br /> == Network Node ==<br /> <br /> 1. Install the Neutron Linux bridge plugin, the Neutron DHCP agent and the Neutron L3 agent:<br /> <br /> # apt-get install neutron-plugin-linuxbridge-agent neutron-dhcp-agent neutron-l3-agent<br /> <br /> 2. Copy /etc/neutron/neutron.conf from the Compute node.<br /> <br /> 3. Change the following configuration of the ini file (/etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini). The &quot;default&quot; in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini . <br /> [linux_bridge] <br /> physical_interface_mappings = default:eth2 <br /> <br /> 4. Configure the DHCP agent according to the following guidelines <br /> <br /> Update the following file: /etc/neutron/dhcp_agent.ini with:<br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver <br /> <br /> For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_dhcp_agent OpenStack documentation]<br /> <br /> 5. Configure the L3 agent according to the following guidelines <br /> <br /> Update the following file: /etc/neutron/l3_agent.ini with:<br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver <br /> <br /> For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_l3_agent OpenStack documentation]<br /> <br /> 6. Configure the Metadata agent according to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#nova_config_metadata OpenStack documentation]<br /> <br /> 7. Start the Neutron Linux Bridge, DHCP and L3 agents <br /> # restart neutron-plugin-linuxbridge-agent<br /> # restart neutron-dhcp-agent<br /> # restart neutron-l3-agent<br /> # restart neutron-metadata-agent<br /> <br /> == Troubleshooting ==<br /> <br /> * If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand&diff=69071 2014-11-26T16:44:16Z

<p>May: /* OpenSM Node */</p> <hr /> <div>= Overview = <br /> <br /> This wiki page describes how to tune Mellanox OpenStack on Ubuntu 12.04 servers<br /> <br /> = Prerequisites =<br /> <br /> * A running OpenStack environment installed with the ML2 plugin on top of OVS.<br /> * All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119) <br /> * Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers<br /> * SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community [http://community.mellanox.com/docs/DOC-1317 click here].<br /> * The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes<br /> * VLANs configured on the ports in the switch.<br /> <br /> = InfiniBand Network = <br /> <br /> == OpenSM Node ==<br /> <br /> To configure OpenSM node: <br /> <br /> 1. Edit the file /etc/opensm/partitions.conf by adding the following line:<br /> management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;<br /> Create the file if it does not exist. <br /> <br /> For each VLAN value in the VLAN range used for VLAN network segmentation, add the following line in the file /etc/opensm/partitions.conf (for vlan-pkey association):<br /> vlan&lt;decimal-value&gt;=0x&lt;hexadecimal-value&gt;, ipoib, sl=0, defmember=full: ALL;<br /> In this manual, the VLAN range is 2-10 (mlnx_agent.conf /ml2_conf.ini). Therefore, add the following lines:<br /> vlan1=0x1, ipoib, sl=0, defmember=full : ALL;<br /> vlan2=0x2, ipoib, sl=0, defmember=full : ALL;<br /> vlan3=0x3, ipoib, sl=0, defmember=full : ALL;<br /> vlan4=0x4, ipoib, sl=0, defmember=full : ALL;<br /> vlan5=0x5, ipoib, sl=0, defmember=full : ALL;<br /> vlan6=0x6, ipoib, sl=0, defmember=full : ALL;<br /> vlan7=0x7, ipoib, sl=0, defmember=full : ALL;<br /> vlan8=0x8, ipoib, sl=0, defmember=full : ALL;<br /> vlan9=0x9, ipoib, sl=0, defmember=full : ALL;<br /> vlan10=0xa, ipoib, sl=0, defmember=full : ALL;<br /> 2. Modify the following line in the file /etc/opensm/opensm.conf from FALSE to TRUE:<br /> allow_both_pkeys TRUE<br /> 3. Restart OpenSM:<br /> service opensmd restart<br /> 4. Verify that all ports on all nodes have their ib interface (i.e. ib0) active, run ibdev2netdev on all hosts, and make sure that the port is up:<br /> mlx4_0 port 1 ==&gt; ib0 (Up)<br /> If the port is down, try bringing it up by running “ifconfig ib0 up”, and make sure your switch ports are of type “InfiniBand”, and that all nodes can reach the OpenSM node by running “sminfo”.<br /> <br /> == Controller Node == <br /> <br /> In /etc/nova/nova.conf ensure you have the following line: <br /> security_group_api=nova<br /> <br /> If you change, restart the nova services<br /> # for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done<br /> <br /> == Neutron Server Node ==<br /> === Installation === <br /> <br /> 1. Neutron server uses MySQL database. Make sure you have running MySQL database for neutron. <br /> <br /> If Neutron server is already running, you should stop it. <br /> <br /> #/etc/init.d/neutron-server stop<br /> <br /> If you want to use a database that was created you have to drop it and create another one. For example: <br /> <br /> mysql -u root -ppassword &lt;&lt;EOF<br /> drop database if exists neutron;<br /> create database neutron;<br /> EOF<br /> <br /> If you want to create a new one: <br /> <br /> mysql -u root -ppassword &lt;&lt;EOF<br /> drop database if exists neutron;<br /> create database neutron;<br /> GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'localhost' IDENTIFIED BY 'password';<br /> GRANT ALL PRIVILEGES ON neutron.* TO 'neutron' IDENTIFIED BY 'password';<br /> FLUSH PRIVILEGES;<br /> EOF<br /> <br /> 2. Install the required Deb for the Mellanox Neutron plugin: <br /> <br /> apt-get install -y neutron-plugin-mlnx-agent<br /> <br /> 3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment. <br /> <br /> 4. Change /etc/default/neutron-server :<br /> <br /> NEUTRON_PLUGIN_CONFIG=&quot;/etc/neutron/plugins/mlnx/mlnx_conf.ini&quot;<br /> <br /> === Configuration ===<br /> <br /> 1. Make the Mellanox plugin the current Neutron plugin by editing neutron.conf and changing core_plugin. <br /> <br /> core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin<br /> <br /> 2. Plugin configuration: Edit the configuration file: /etc/neutron/plugins/mlnx/mlnx_conf.ini <br /> <br /> [MLNX]<br /> tenant_network_type - must be set on of supported tenant network types. <br /> Possible values: 'vlan' for Ethernet or 'ib' for Infiniband.'vlan' is the default.<br /> network_vlan_ranges - must be configured to specify the names of the physical <br /> <br /> networks managed by the mellanox plugin, <br /> along with the ranges of VLAN IDs available on each physical network for allocation to virtual networks. <br /> Possible range is vlan range is 1-4093. <br /> The default is &quot;default:1:100&quot; <br /> &lt;fabric name &gt;:&lt;vlan range start&gt;:&lt;vlan end range&gt;<br /> <br /> [agent]<br /> rpc_support_old_agents = True<br /> Note: rpc_support_old_agents should be set to 'True' (non default).<br /> <br /> === Start Services ===<br /> <br /> Start (or restart) the Neutron server <br /> #/etc/init.d/neutron-server start<br /> <br /> == Compute Nodes ==<br /> <br /> === Installation ===<br /> <br /> 1. If you didn't download Mellanox OpenStack debian files download it : <br /> <br /> http://www.mellanox.com/downloads/solutions/openstack/<br /> <br /> 2. Install the eswitchd debian files: <br /> dpkg -i eswitchd_0.8-1_amd64.deb<br /> dpkg -i python-mlnxvif_0.6-1_all.deb<br /> apt-get install -f # fix dependencies<br /> apt-get install -y neutron-plugin-mlnx-agent<br /> <br /> 3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)<br /> <br /> === Configuration === <br /> <br /> 1. Configure /etc/eswitchd/eswitchd.conf if needed <br /> <br /> Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here]) <br /> <br /> 2. Modify /etc/nova/nova.conf <br /> <br /> compute_driver=nova.virt.libvirt.driver.LibvirtDriver<br /> libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver<br /> security_group_api=nova<br /> connection_type=libvirt<br /> <br /> In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following: <br /> <br /> libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver<br /> <br /> 3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment. <br /> [AGENT]<br /> rpc_support_old_agents = True<br /> [ESWITCH]<br /> physical_interface_mapping - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: &lt;fabric name&gt;:&lt; PF name&gt; (Only releavant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is &quot;default:autoeth&quot;. <br /> vnic_type - type of VM network interface: 'mlnx_direct' or 'hostdev' according to libvirt terminology.<br /> hostdev: this is the traditional method of assigning any generic PCI device to a guest (SR-IOV).<br /> mlnx_direct: this is a method to provide macvtap device on top of the PCI device (default).<br /> bridge: - When using Linux Bridge Plugin on top of eIPoIB device<br /> daemon_endpoint - 'tcp://127.0.0.1:60001'<br /> <br /> For a plugin configuration file example (Havana), please refer to Mellanox config *ini file (click [https://github.com/openstack/neutron/blob/stable/havana/etc/neutron/plugins/mlnx/mlnx_conf.ini here])<br /> <br /> 4. (Optional) Modify /etc/init/neutron-plugin-mlnx-agent.conf as described in LP bug #[https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1257732 1257732]<br /> <br /> 5. Check that ebrctl is listed somewhere in /etc/nova/rootwrap.d/*, otherwise add this to /etc/nova/rootwrap.d/compute.filters file in the [Filters] section.<br /> ebrctl: CommandFilter, ebrctl, root<br /> <br /> === Start Services ===<br /> <br /> 1. Restart Nova. <br /> <br /> #/etc/init.d/nova-compute restart<br /> <br /> 2. Start eswitch Daemon <br /> <br /> #/etc/init.d/eswitchd start<br /> <br /> 3. Start the Neutron agent <br /> <br /> #/etc/init.d/neutron-plugin-mlnx-agent start<br /> <br /> Note: eswitch Daemon should be running before the Neutron Agent is started.<br /> <br /> == Network Node ==<br /> <br /> 1. Install the Neutron Linux bridge plugin, the Neutron DHCP agent and the Neutron L3 agent:<br /> <br /> # apt-get install neutron-plugin-linuxbridge-agent neutron-dhcp-agent neutron-l3-agent<br /> <br /> 2. Copy /etc/neutron/neutron.conf from the Compute node.<br /> <br /> 3. Change the following configuration of the ini file (/etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini). The &quot;default&quot; in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini . <br /> [linux_bridge] <br /> physical_interface_mappings = default:eth2 <br /> <br /> 4. Configure the DHCP agent according to the following guidelines <br /> <br /> Update the following file: /etc/neutron/dhcp_agent.ini with:<br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver <br /> <br /> For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_dhcp_agent OpenStack documentation]<br /> <br /> 5. Configure the L3 agent according to the following guidelines <br /> <br /> Update the following file: /etc/neutron/l3_agent.ini with:<br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver <br /> <br /> For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_l3_agent OpenStack documentation]<br /> <br /> 6. Configure the Metadata agent according to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#nova_config_metadata OpenStack documentation]<br /> <br /> 7. Start the Neutron Linux Bridge, DHCP and L3 agents <br /> # restart neutron-plugin-linuxbridge-agent<br /> # restart neutron-dhcp-agent<br /> # restart neutron-l3-agent<br /> # restart neutron-metadata-agent<br /> <br /> == Troubleshooting ==<br /> <br /> * If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand&diff=69070 2014-11-26T16:39:15Z

<p>May: </p> <hr /> <div>= Overview = <br /> <br /> This wiki page describes how to tune Mellanox OpenStack on Ubuntu 12.04 servers<br /> <br /> = Prerequisites =<br /> <br /> * A running OpenStack environment installed with the ML2 plugin on top of OVS.<br /> * All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119) <br /> * Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers<br /> * SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community [http://community.mellanox.com/docs/DOC-1317 click here].<br /> * The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes<br /> * VLANs configured on the ports in the switch.<br /> <br /> = InfiniBand Network = <br /> <br /> == OpenSM Node ==<br /> <br /> To configure OpenSM node: <br /> <br /> 1. Edit the file /etc/opensm/partitions.conf by adding the following line:<br /> management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;<br /> Create the file if it does not exist. <br /> For each VLAN value in the VLAN range used for VLAN network segmentation, add the following line in the file /etc/opensm/partitions.conf (for vlan-pkey association):<br /> vlan&lt;decimal-value&gt;=0x&lt;hexadecimal-value&gt;, ipoib, sl=0, defmember=full: ALL;<br /> In this manual, the VLAN range is 2-10 (mlnx_agent.conf /ml2_conf.ini). Therefore, add the following lines:<br /> vlan1=0x1, ipoib, sl=0, defmember=full : ALL;<br /> vlan2=0x2, ipoib, sl=0, defmember=full : ALL;<br /> vlan3=0x3, ipoib, sl=0, defmember=full : ALL;<br /> vlan4=0x4, ipoib, sl=0, defmember=full : ALL;<br /> vlan5=0x5, ipoib, sl=0, defmember=full : ALL;<br /> vlan6=0x6, ipoib, sl=0, defmember=full : ALL;<br /> vlan7=0x7, ipoib, sl=0, defmember=full : ALL;<br /> vlan8=0x8, ipoib, sl=0, defmember=full : ALL;<br /> vlan9=0x9, ipoib, sl=0, defmember=full : ALL;<br /> vlan10=0xa, ipoib, sl=0, defmember=full : ALL;<br /> 2. Modify the following line in the file /etc/opensm/opensm.conf from FALSE to TRUE:<br /> allow_both_pkeys TRUE<br /> 3. Restart OpenSM:<br /> service opensmd restart<br /> 4. Verify that all ports on all nodes have their ib interface (i.e. ib0) active, run ibdev2netdev on all hosts, and make sure that the port is up:<br /> mlx4_0 port 1 ==&gt; ib0 (Up)<br /> If the port is down, try bringing it up by running “ifconfig ib0 up”, and make sure your switch ports are of type “InfiniBand”, and that all nodes can reach the OpenSM node by running “sminfo”.<br /> <br /> == Controller Node == <br /> <br /> In /etc/nova/nova.conf ensure you have the following line: <br /> security_group_api=nova<br /> <br /> If you change, restart the nova services<br /> # for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done<br /> <br /> == Neutron Server Node ==<br /> === Installation === <br /> <br /> 1. Neutron server uses MySQL database. Make sure you have running MySQL database for neutron. <br /> <br /> If Neutron server is already running, you should stop it. <br /> <br /> #/etc/init.d/neutron-server stop<br /> <br /> If you want to use a database that was created you have to drop it and create another one. For example: <br /> <br /> mysql -u root -ppassword &lt;&lt;EOF<br /> drop database if exists neutron;<br /> create database neutron;<br /> EOF<br /> <br /> If you want to create a new one: <br /> <br /> mysql -u root -ppassword &lt;&lt;EOF<br /> drop database if exists neutron;<br /> create database neutron;<br /> GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'localhost' IDENTIFIED BY 'password';<br /> GRANT ALL PRIVILEGES ON neutron.* TO 'neutron' IDENTIFIED BY 'password';<br /> FLUSH PRIVILEGES;<br /> EOF<br /> <br /> 2. Install the required Deb for the Mellanox Neutron plugin: <br /> <br /> apt-get install -y neutron-plugin-mlnx-agent<br /> <br /> 3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment. <br /> <br /> 4. Change /etc/default/neutron-server :<br /> <br /> NEUTRON_PLUGIN_CONFIG=&quot;/etc/neutron/plugins/mlnx/mlnx_conf.ini&quot;<br /> <br /> === Configuration ===<br /> <br /> 1. Make the Mellanox plugin the current Neutron plugin by editing neutron.conf and changing core_plugin. <br /> <br /> core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin<br /> <br /> 2. Plugin configuration: Edit the configuration file: /etc/neutron/plugins/mlnx/mlnx_conf.ini <br /> <br /> [MLNX]<br /> tenant_network_type - must be set on of supported tenant network types. <br /> Possible values: 'vlan' for Ethernet or 'ib' for Infiniband.'vlan' is the default.<br /> network_vlan_ranges - must be configured to specify the names of the physical <br /> <br /> networks managed by the mellanox plugin, <br /> along with the ranges of VLAN IDs available on each physical network for allocation to virtual networks. <br /> Possible range is vlan range is 1-4093. <br /> The default is &quot;default:1:100&quot; <br /> &lt;fabric name &gt;:&lt;vlan range start&gt;:&lt;vlan end range&gt;<br /> <br /> [agent]<br /> rpc_support_old_agents = True<br /> Note: rpc_support_old_agents should be set to 'True' (non default).<br /> <br /> === Start Services ===<br /> <br /> Start (or restart) the Neutron server <br /> #/etc/init.d/neutron-server start<br /> <br /> == Compute Nodes ==<br /> <br /> === Installation ===<br /> <br /> 1. If you didn't download Mellanox OpenStack debian files download it : <br /> <br /> http://www.mellanox.com/downloads/solutions/openstack/<br /> <br /> 2. Install the eswitchd debian files: <br /> dpkg -i eswitchd_0.8-1_amd64.deb<br /> dpkg -i python-mlnxvif_0.6-1_all.deb<br /> apt-get install -f # fix dependencies<br /> apt-get install -y neutron-plugin-mlnx-agent<br /> <br /> 3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)<br /> <br /> === Configuration === <br /> <br /> 1. Configure /etc/eswitchd/eswitchd.conf if needed <br /> <br /> Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here]) <br /> <br /> 2. Modify /etc/nova/nova.conf <br /> <br /> compute_driver=nova.virt.libvirt.driver.LibvirtDriver<br /> libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver<br /> security_group_api=nova<br /> connection_type=libvirt<br /> <br /> In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following: <br /> <br /> libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver<br /> <br /> 3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment. <br /> [AGENT]<br /> rpc_support_old_agents = True<br /> [ESWITCH]<br /> physical_interface_mapping - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: &lt;fabric name&gt;:&lt; PF name&gt; (Only releavant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is &quot;default:autoeth&quot;. <br /> vnic_type - type of VM network interface: 'mlnx_direct' or 'hostdev' according to libvirt terminology.<br /> hostdev: this is the traditional method of assigning any generic PCI device to a guest (SR-IOV).<br /> mlnx_direct: this is a method to provide macvtap device on top of the PCI device (default).<br /> bridge: - When using Linux Bridge Plugin on top of eIPoIB device<br /> daemon_endpoint - 'tcp://127.0.0.1:60001'<br /> <br /> For a plugin configuration file example (Havana), please refer to Mellanox config *ini file (click [https://github.com/openstack/neutron/blob/stable/havana/etc/neutron/plugins/mlnx/mlnx_conf.ini here])<br /> <br /> 4. (Optional) Modify /etc/init/neutron-plugin-mlnx-agent.conf as described in LP bug #[https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1257732 1257732]<br /> <br /> 5. Check that ebrctl is listed somewhere in /etc/nova/rootwrap.d/*, otherwise add this to /etc/nova/rootwrap.d/compute.filters file in the [Filters] section.<br /> ebrctl: CommandFilter, ebrctl, root<br /> <br /> === Start Services ===<br /> <br /> 1. Restart Nova. <br /> <br /> #/etc/init.d/nova-compute restart<br /> <br /> 2. Start eswitch Daemon <br /> <br /> #/etc/init.d/eswitchd start<br /> <br /> 3. Start the Neutron agent <br /> <br /> #/etc/init.d/neutron-plugin-mlnx-agent start<br /> <br /> Note: eswitch Daemon should be running before the Neutron Agent is started.<br /> <br /> == Network Node ==<br /> <br /> 1. Install the Neutron Linux bridge plugin, the Neutron DHCP agent and the Neutron L3 agent:<br /> <br /> # apt-get install neutron-plugin-linuxbridge-agent neutron-dhcp-agent neutron-l3-agent<br /> <br /> 2. Copy /etc/neutron/neutron.conf from the Compute node.<br /> <br /> 3. Change the following configuration of the ini file (/etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini). The &quot;default&quot; in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini . <br /> [linux_bridge] <br /> physical_interface_mappings = default:eth2 <br /> <br /> 4. Configure the DHCP agent according to the following guidelines <br /> <br /> Update the following file: /etc/neutron/dhcp_agent.ini with:<br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver <br /> <br /> For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_dhcp_agent OpenStack documentation]<br /> <br /> 5. Configure the L3 agent according to the following guidelines <br /> <br /> Update the following file: /etc/neutron/l3_agent.ini with:<br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver <br /> <br /> For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_l3_agent OpenStack documentation]<br /> <br /> 6. Configure the Metadata agent according to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#nova_config_metadata OpenStack documentation]<br /> <br /> 7. Start the Neutron Linux Bridge, DHCP and L3 agents <br /> # restart neutron-plugin-linuxbridge-agent<br /> # restart neutron-dhcp-agent<br /> # restart neutron-l3-agent<br /> # restart neutron-metadata-agent<br /> <br /> == Troubleshooting ==<br /> <br /> * If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand&diff=69069 2014-11-26T16:33:16Z

<p>May: /* Ethernet Network */</p> <hr /> <div>= Overview = <br /> <br /> This wiki page describes how to tune Mellanox OpenStack on Ubuntu 12.04 servers<br /> <br /> = Prerequisites =<br /> <br /> * A running OpenStack environment installed with the ML2 plugin on top of OVS.<br /> * All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119) <br /> * Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers<br /> * SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community [http://community.mellanox.com/docs/DOC-1317 click here].<br /> * The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes<br /> * VLANs configured on the ports in the switch.<br /> <br /> = InfiniBand Network = <br /> <br /> == Controller Node == <br /> <br /> In /etc/nova/nova.conf ensure you have the following line: <br /> security_group_api=nova<br /> <br /> If you change, restart the nova services<br /> # for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done<br /> <br /> == Neutron Server Node ==<br /> === Installation === <br /> <br /> 1. Neutron server uses MySQL database. Make sure you have running MySQL database for neutron. <br /> <br /> If Neutron server is already running, you should stop it. <br /> <br /> #/etc/init.d/neutron-server stop<br /> <br /> If you want to use a database that was created you have to drop it and create another one. For example: <br /> <br /> mysql -u root -ppassword &lt;&lt;EOF<br /> drop database if exists neutron;<br /> create database neutron;<br /> EOF<br /> <br /> If you want to create a new one: <br /> <br /> mysql -u root -ppassword &lt;&lt;EOF<br /> drop database if exists neutron;<br /> create database neutron;<br /> GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'localhost' IDENTIFIED BY 'password';<br /> GRANT ALL PRIVILEGES ON neutron.* TO 'neutron' IDENTIFIED BY 'password';<br /> FLUSH PRIVILEGES;<br /> EOF<br /> <br /> 2. Install the required Deb for the Mellanox Neutron plugin: <br /> <br /> apt-get install -y neutron-plugin-mlnx-agent<br /> <br /> 3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment. <br /> <br /> 4. Change /etc/default/neutron-server :<br /> <br /> NEUTRON_PLUGIN_CONFIG=&quot;/etc/neutron/plugins/mlnx/mlnx_conf.ini&quot;<br /> <br /> === Configuration ===<br /> <br /> 1. Make the Mellanox plugin the current Neutron plugin by editing neutron.conf and changing core_plugin. <br /> <br /> core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin<br /> <br /> 2. Plugin configuration: Edit the configuration file: /etc/neutron/plugins/mlnx/mlnx_conf.ini <br /> <br /> [MLNX]<br /> tenant_network_type - must be set on of supported tenant network types. <br /> Possible values: 'vlan' for Ethernet or 'ib' for Infiniband.'vlan' is the default.<br /> network_vlan_ranges - must be configured to specify the names of the physical <br /> <br /> networks managed by the mellanox plugin, <br /> along with the ranges of VLAN IDs available on each physical network for allocation to virtual networks. <br /> Possible range is vlan range is 1-4093. <br /> The default is &quot;default:1:100&quot; <br /> &lt;fabric name &gt;:&lt;vlan range start&gt;:&lt;vlan end range&gt;<br /> <br /> [agent]<br /> rpc_support_old_agents = True<br /> Note: rpc_support_old_agents should be set to 'True' (non default).<br /> <br /> === Start Services ===<br /> <br /> Start (or restart) the Neutron server <br /> #/etc/init.d/neutron-server start<br /> <br /> == Compute Nodes ==<br /> <br /> === Installation ===<br /> <br /> 1. If you didn't download Mellanox OpenStack debian files download it : <br /> <br /> http://www.mellanox.com/downloads/solutions/openstack/<br /> <br /> 2. Install the eswitchd debian files: <br /> dpkg -i eswitchd_0.8-1_amd64.deb<br /> dpkg -i python-mlnxvif_0.6-1_all.deb<br /> apt-get install -f # fix dependencies<br /> apt-get install -y neutron-plugin-mlnx-agent<br /> <br /> 3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)<br /> <br /> === Configuration === <br /> <br /> 1. Configure /etc/eswitchd/eswitchd.conf if needed <br /> <br /> Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here]) <br /> <br /> 2. Modify /etc/nova/nova.conf <br /> <br /> compute_driver=nova.virt.libvirt.driver.LibvirtDriver<br /> libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver<br /> security_group_api=nova<br /> connection_type=libvirt<br /> <br /> In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following: <br /> <br /> libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver<br /> <br /> 3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment. <br /> [AGENT]<br /> rpc_support_old_agents = True<br /> [ESWITCH]<br /> physical_interface_mapping - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: &lt;fabric name&gt;:&lt; PF name&gt; (Only releavant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is &quot;default:autoeth&quot;. <br /> vnic_type - type of VM network interface: 'mlnx_direct' or 'hostdev' according to libvirt terminology.<br /> hostdev: this is the traditional method of assigning any generic PCI device to a guest (SR-IOV).<br /> mlnx_direct: this is a method to provide macvtap device on top of the PCI device (default).<br /> bridge: - When using Linux Bridge Plugin on top of eIPoIB device<br /> daemon_endpoint - 'tcp://127.0.0.1:60001'<br /> <br /> For a plugin configuration file example (Havana), please refer to Mellanox config *ini file (click [https://github.com/openstack/neutron/blob/stable/havana/etc/neutron/plugins/mlnx/mlnx_conf.ini here])<br /> <br /> 4. (Optional) Modify /etc/init/neutron-plugin-mlnx-agent.conf as described in LP bug #[https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1257732 1257732]<br /> <br /> 5. Check that ebrctl is listed somewhere in /etc/nova/rootwrap.d/*, otherwise add this to /etc/nova/rootwrap.d/compute.filters file in the [Filters] section.<br /> ebrctl: CommandFilter, ebrctl, root<br /> <br /> === Start Services ===<br /> <br /> 1. Restart Nova. <br /> <br /> #/etc/init.d/nova-compute restart<br /> <br /> 2. Start eswitch Daemon <br /> <br /> #/etc/init.d/eswitchd start<br /> <br /> 3. Start the Neutron agent <br /> <br /> #/etc/init.d/neutron-plugin-mlnx-agent start<br /> <br /> Note: eswitch Daemon should be running before the Neutron Agent is started.<br /> <br /> == Network Node ==<br /> <br /> 1. Install the Neutron Linux bridge plugin, the Neutron DHCP agent and the Neutron L3 agent:<br /> <br /> # apt-get install neutron-plugin-linuxbridge-agent neutron-dhcp-agent neutron-l3-agent<br /> <br /> 2. Copy /etc/neutron/neutron.conf from the Compute node.<br /> <br /> 3. Change the following configuration of the ini file (/etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini). The &quot;default&quot; in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini . <br /> [linux_bridge] <br /> physical_interface_mappings = default:eth2 <br /> <br /> 4. Configure the DHCP agent according to the following guidelines <br /> <br /> Update the following file: /etc/neutron/dhcp_agent.ini with:<br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver <br /> <br /> For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_dhcp_agent OpenStack documentation]<br /> <br /> 5. Configure the L3 agent according to the following guidelines <br /> <br /> Update the following file: /etc/neutron/l3_agent.ini with:<br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver <br /> <br /> For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_l3_agent OpenStack documentation]<br /> <br /> 6. Configure the Metadata agent according to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#nova_config_metadata OpenStack documentation]<br /> <br /> 7. Start the Neutron Linux Bridge, DHCP and L3 agents <br /> # restart neutron-plugin-linuxbridge-agent<br /> # restart neutron-dhcp-agent<br /> # restart neutron-l3-agent<br /> # restart neutron-metadata-agent<br /> <br /> == Troubleshooting ==<br /> <br /> * If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand&diff=69068 2014-11-26T16:32:16Z

<p>May: /* Prerequisites */</p> <hr /> <div>= Overview = <br /> <br /> This wiki page describes how to tune Mellanox OpenStack on Ubuntu 12.04 servers<br /> <br /> = Prerequisites =<br /> <br /> * A running OpenStack environment installed with the ML2 plugin on top of OVS.<br /> * All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119) <br /> * Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers<br /> * SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community [http://community.mellanox.com/docs/DOC-1317 click here].<br /> * The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes<br /> * VLANs configured on the ports in the switch.<br /> <br /> = Ethernet Network = <br /> <br /> == Controller Node == <br /> <br /> In /etc/nova/nova.conf ensure you have the following line: <br /> security_group_api=nova<br /> <br /> If you change, restart the nova services<br /> # for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done<br /> <br /> == Neutron Server Node ==<br /> === Installation === <br /> <br /> 1. Neutron server uses MySQL database. Make sure you have running MySQL database for neutron. <br /> <br /> If Neutron server is already running, you should stop it. <br /> <br /> #/etc/init.d/neutron-server stop<br /> <br /> If you want to use a database that was created you have to drop it and create another one. For example: <br /> <br /> mysql -u root -ppassword &lt;&lt;EOF<br /> drop database if exists neutron;<br /> create database neutron;<br /> EOF<br /> <br /> If you want to create a new one: <br /> <br /> mysql -u root -ppassword &lt;&lt;EOF<br /> drop database if exists neutron;<br /> create database neutron;<br /> GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'localhost' IDENTIFIED BY 'password';<br /> GRANT ALL PRIVILEGES ON neutron.* TO 'neutron' IDENTIFIED BY 'password';<br /> FLUSH PRIVILEGES;<br /> EOF<br /> <br /> 2. Install the required Deb for the Mellanox Neutron plugin: <br /> <br /> apt-get install -y neutron-plugin-mlnx-agent<br /> <br /> 3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment. <br /> <br /> 4. Change /etc/default/neutron-server :<br /> <br /> NEUTRON_PLUGIN_CONFIG=&quot;/etc/neutron/plugins/mlnx/mlnx_conf.ini&quot;<br /> <br /> === Configuration ===<br /> <br /> 1. Make the Mellanox plugin the current Neutron plugin by editing neutron.conf and changing core_plugin. <br /> <br /> core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin<br /> <br /> 2. Plugin configuration: Edit the configuration file: /etc/neutron/plugins/mlnx/mlnx_conf.ini <br /> <br /> [MLNX]<br /> tenant_network_type - must be set on of supported tenant network types. <br /> Possible values: 'vlan' for Ethernet or 'ib' for Infiniband.'vlan' is the default.<br /> network_vlan_ranges - must be configured to specify the names of the physical <br /> <br /> networks managed by the mellanox plugin, <br /> along with the ranges of VLAN IDs available on each physical network for allocation to virtual networks. <br /> Possible range is vlan range is 1-4093. <br /> The default is &quot;default:1:100&quot; <br /> &lt;fabric name &gt;:&lt;vlan range start&gt;:&lt;vlan end range&gt;<br /> <br /> [agent]<br /> rpc_support_old_agents = True<br /> Note: rpc_support_old_agents should be set to 'True' (non default).<br /> <br /> === Start Services ===<br /> <br /> Start (or restart) the Neutron server <br /> #/etc/init.d/neutron-server start<br /> <br /> == Compute Nodes ==<br /> <br /> === Installation ===<br /> <br /> 1. If you didn't download Mellanox OpenStack debian files download it : <br /> <br /> http://www.mellanox.com/downloads/solutions/openstack/<br /> <br /> 2. Install the eswitchd debian files: <br /> dpkg -i eswitchd_0.8-1_amd64.deb<br /> dpkg -i python-mlnxvif_0.6-1_all.deb<br /> apt-get install -f # fix dependencies<br /> apt-get install -y neutron-plugin-mlnx-agent<br /> <br /> 3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)<br /> <br /> === Configuration === <br /> <br /> 1. Configure /etc/eswitchd/eswitchd.conf if needed <br /> <br /> Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here]) <br /> <br /> 2. Modify /etc/nova/nova.conf <br /> <br /> compute_driver=nova.virt.libvirt.driver.LibvirtDriver<br /> libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver<br /> security_group_api=nova<br /> connection_type=libvirt<br /> <br /> In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following: <br /> <br /> libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver<br /> <br /> 3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment. <br /> [AGENT]<br /> rpc_support_old_agents = True<br /> [ESWITCH]<br /> physical_interface_mapping - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: &lt;fabric name&gt;:&lt; PF name&gt; (Only releavant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is &quot;default:autoeth&quot;. <br /> vnic_type - type of VM network interface: 'mlnx_direct' or 'hostdev' according to libvirt terminology.<br /> hostdev: this is the traditional method of assigning any generic PCI device to a guest (SR-IOV).<br /> mlnx_direct: this is a method to provide macvtap device on top of the PCI device (default).<br /> bridge: - When using Linux Bridge Plugin on top of eIPoIB device<br /> daemon_endpoint - 'tcp://127.0.0.1:60001'<br /> <br /> For a plugin configuration file example (Havana), please refer to Mellanox config *ini file (click [https://github.com/openstack/neutron/blob/stable/havana/etc/neutron/plugins/mlnx/mlnx_conf.ini here])<br /> <br /> 4. (Optional) Modify /etc/init/neutron-plugin-mlnx-agent.conf as described in LP bug #[https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1257732 1257732]<br /> <br /> 5. Check that ebrctl is listed somewhere in /etc/nova/rootwrap.d/*, otherwise add this to /etc/nova/rootwrap.d/compute.filters file in the [Filters] section.<br /> ebrctl: CommandFilter, ebrctl, root<br /> <br /> === Start Services ===<br /> <br /> 1. Restart Nova. <br /> <br /> #/etc/init.d/nova-compute restart<br /> <br /> 2. Start eswitch Daemon <br /> <br /> #/etc/init.d/eswitchd start<br /> <br /> 3. Start the Neutron agent <br /> <br /> #/etc/init.d/neutron-plugin-mlnx-agent start<br /> <br /> Note: eswitch Daemon should be running before the Neutron Agent is started.<br /> <br /> == Network Node ==<br /> <br /> 1. Install the Neutron Linux bridge plugin, the Neutron DHCP agent and the Neutron L3 agent:<br /> <br /> # apt-get install neutron-plugin-linuxbridge-agent neutron-dhcp-agent neutron-l3-agent<br /> <br /> 2. Copy /etc/neutron/neutron.conf from the Compute node.<br /> <br /> 3. Change the following configuration of the ini file (/etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini). The &quot;default&quot; in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini . <br /> [linux_bridge] <br /> physical_interface_mappings = default:eth2 <br /> <br /> 4. Configure the DHCP agent according to the following guidelines <br /> <br /> Update the following file: /etc/neutron/dhcp_agent.ini with:<br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver <br /> <br /> For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_dhcp_agent OpenStack documentation]<br /> <br /> 5. Configure the L3 agent according to the following guidelines <br /> <br /> Update the following file: /etc/neutron/l3_agent.ini with:<br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver <br /> <br /> For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_l3_agent OpenStack documentation]<br /> <br /> 6. Configure the Metadata agent according to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#nova_config_metadata OpenStack documentation]<br /> <br /> 7. Start the Neutron Linux Bridge, DHCP and L3 agents <br /> # restart neutron-plugin-linuxbridge-agent<br /> # restart neutron-dhcp-agent<br /> # restart neutron-l3-agent<br /> # restart neutron-metadata-agent<br /> <br /> == Troubleshooting ==<br /> <br /> * If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Icehouse-Ubuntu-InfiniBand&diff=69067 2014-11-26T16:31:22Z

<p>May: </p> <hr /> <div>= Overview = <br /> <br /> This wiki page describes how to tune Mellanox OpenStack on Ubuntu 12.04 servers<br /> <br /> = Prerequisites =<br /> <br /> You need at least 3 nodes:<br /> * Controller node running the neutron-server service.<br /> * Compute node running the eswitchd, nova-compute &amp; neutron-plugin-mlnx-agent services.<br /> * Network node running the neutron-plugin-linuxbridge-agent, neutron-dhcp-agent and neutron-l3-agent services.<br /> <br /> = Ethernet Network = <br /> <br /> == Controller Node == <br /> <br /> In /etc/nova/nova.conf ensure you have the following line: <br /> security_group_api=nova<br /> <br /> If you change, restart the nova services<br /> # for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done<br /> <br /> == Neutron Server Node ==<br /> === Installation === <br /> <br /> 1. Neutron server uses MySQL database. Make sure you have running MySQL database for neutron. <br /> <br /> If Neutron server is already running, you should stop it. <br /> <br /> #/etc/init.d/neutron-server stop<br /> <br /> If you want to use a database that was created you have to drop it and create another one. For example: <br /> <br /> mysql -u root -ppassword &lt;&lt;EOF<br /> drop database if exists neutron;<br /> create database neutron;<br /> EOF<br /> <br /> If you want to create a new one: <br /> <br /> mysql -u root -ppassword &lt;&lt;EOF<br /> drop database if exists neutron;<br /> create database neutron;<br /> GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'localhost' IDENTIFIED BY 'password';<br /> GRANT ALL PRIVILEGES ON neutron.* TO 'neutron' IDENTIFIED BY 'password';<br /> FLUSH PRIVILEGES;<br /> EOF<br /> <br /> 2. Install the required Deb for the Mellanox Neutron plugin: <br /> <br /> apt-get install -y neutron-plugin-mlnx-agent<br /> <br /> 3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment. <br /> <br /> 4. Change /etc/default/neutron-server :<br /> <br /> NEUTRON_PLUGIN_CONFIG=&quot;/etc/neutron/plugins/mlnx/mlnx_conf.ini&quot;<br /> <br /> === Configuration ===<br /> <br /> 1. Make the Mellanox plugin the current Neutron plugin by editing neutron.conf and changing core_plugin. <br /> <br /> core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin<br /> <br /> 2. Plugin configuration: Edit the configuration file: /etc/neutron/plugins/mlnx/mlnx_conf.ini <br /> <br /> [MLNX]<br /> tenant_network_type - must be set on of supported tenant network types. <br /> Possible values: 'vlan' for Ethernet or 'ib' for Infiniband.'vlan' is the default.<br /> network_vlan_ranges - must be configured to specify the names of the physical <br /> <br /> networks managed by the mellanox plugin, <br /> along with the ranges of VLAN IDs available on each physical network for allocation to virtual networks. <br /> Possible range is vlan range is 1-4093. <br /> The default is &quot;default:1:100&quot; <br /> &lt;fabric name &gt;:&lt;vlan range start&gt;:&lt;vlan end range&gt;<br /> <br /> [agent]<br /> rpc_support_old_agents = True<br /> Note: rpc_support_old_agents should be set to 'True' (non default).<br /> <br /> === Start Services ===<br /> <br /> Start (or restart) the Neutron server <br /> #/etc/init.d/neutron-server start<br /> <br /> == Compute Nodes ==<br /> <br /> === Installation ===<br /> <br /> 1. If you didn't download Mellanox OpenStack debian files download it : <br /> <br /> http://www.mellanox.com/downloads/solutions/openstack/<br /> <br /> 2. Install the eswitchd debian files: <br /> dpkg -i eswitchd_0.8-1_amd64.deb<br /> dpkg -i python-mlnxvif_0.6-1_all.deb<br /> apt-get install -f # fix dependencies<br /> apt-get install -y neutron-plugin-mlnx-agent<br /> <br /> 3. In case you would like to use Ethernet in para-virtualized mode the VIF driver is already included in Nova package. Otherwise, Install Mellanox VIF driver (Make sure nova is installed on your server)<br /> <br /> === Configuration === <br /> <br /> 1. Configure /etc/eswitchd/eswitchd.conf if needed <br /> <br /> Please Refer to Mellanox Community for the eSwitchd installation notes ( click [http://community.mellanox.com/docs/DOC-1126 here]) <br /> <br /> 2. Modify /etc/nova/nova.conf <br /> <br /> compute_driver=nova.virt.libvirt.driver.LibvirtDriver<br /> libvirt_vif_driver=mlnxvif.vif.MlxEthVIFDriver<br /> security_group_api=nova<br /> connection_type=libvirt<br /> <br /> In case you didn't install the Mellanox VIF driver, and you plan to use Ethernet only in paravirtualized ,mode: change the following: <br /> <br /> libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtGenericVIFDriver<br /> <br /> 3. Modify the /etc/neutron/plugins/mlnx/mlnx_conf.ini file to reflect your environment. <br /> [AGENT]<br /> rpc_support_old_agents = True<br /> [ESWITCH]<br /> physical_interface_mapping - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network. The format of this paramter is: &lt;fabric name&gt;:&lt; PF name&gt; (Only releavant on Compute node). PF Name can either be the PF (Physical Function) Name or 'autoeth' for automatic Ethernet configuration,'autoib' for automatic Infiniband configuration.The default is &quot;default:autoeth&quot;. <br /> vnic_type - type of VM network interface: 'mlnx_direct' or 'hostdev' according to libvirt terminology.<br /> hostdev: this is the traditional method of assigning any generic PCI device to a guest (SR-IOV).<br /> mlnx_direct: this is a method to provide macvtap device on top of the PCI device (default).<br /> bridge: - When using Linux Bridge Plugin on top of eIPoIB device<br /> daemon_endpoint - 'tcp://127.0.0.1:60001'<br /> <br /> For a plugin configuration file example (Havana), please refer to Mellanox config *ini file (click [https://github.com/openstack/neutron/blob/stable/havana/etc/neutron/plugins/mlnx/mlnx_conf.ini here])<br /> <br /> 4. (Optional) Modify /etc/init/neutron-plugin-mlnx-agent.conf as described in LP bug #[https://bugs.launchpad.net/ubuntu/+source/neutron/+bug/1257732 1257732]<br /> <br /> 5. Check that ebrctl is listed somewhere in /etc/nova/rootwrap.d/*, otherwise add this to /etc/nova/rootwrap.d/compute.filters file in the [Filters] section.<br /> ebrctl: CommandFilter, ebrctl, root<br /> <br /> === Start Services ===<br /> <br /> 1. Restart Nova. <br /> <br /> #/etc/init.d/nova-compute restart<br /> <br /> 2. Start eswitch Daemon <br /> <br /> #/etc/init.d/eswitchd start<br /> <br /> 3. Start the Neutron agent <br /> <br /> #/etc/init.d/neutron-plugin-mlnx-agent start<br /> <br /> Note: eswitch Daemon should be running before the Neutron Agent is started.<br /> <br /> == Network Node ==<br /> <br /> 1. Install the Neutron Linux bridge plugin, the Neutron DHCP agent and the Neutron L3 agent:<br /> <br /> # apt-get install neutron-plugin-linuxbridge-agent neutron-dhcp-agent neutron-l3-agent<br /> <br /> 2. Copy /etc/neutron/neutron.conf from the Compute node.<br /> <br /> 3. Change the following configuration of the ini file (/etc/neutron/plugins/linuxbridge/linuxbridge_conf.ini). The &quot;default&quot; in the example is the name of the physical network as configured in /etc/neutron/plugins/mlnx/mlnx_conf.ini . <br /> [linux_bridge] <br /> physical_interface_mappings = default:eth2 <br /> <br /> 4. Configure the DHCP agent according to the following guidelines <br /> <br /> Update the following file: /etc/neutron/dhcp_agent.ini with:<br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver <br /> <br /> For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_dhcp_agent OpenStack documentation]<br /> <br /> 5. Configure the L3 agent according to the following guidelines <br /> <br /> Update the following file: /etc/neutron/l3_agent.ini with:<br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver <br /> <br /> For additional information refer to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#section_adv_cfg_l3_agent OpenStack documentation]<br /> <br /> 6. Configure the Metadata agent according to the [http://docs.openstack.org/admin-guide-cloud/content/ch_networking.html#nova_config_metadata OpenStack documentation]<br /> <br /> 7. Start the Neutron Linux Bridge, DHCP and L3 agents <br /> # restart neutron-plugin-linuxbridge-agent<br /> # restart neutron-dhcp-agent<br /> # restart neutron-l3-agent<br /> # restart neutron-metadata-agent<br /> <br /> == Troubleshooting ==<br /> <br /> * If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.</div>

May https://wiki.openstack.org/w/index.php?title=Mellanox-Neutron-Icehouse-Ubuntu-Ethernet&diff=69066 2014-11-26T16:30:09Z

<p>May: /* Network Node */</p> <hr /> <div>= Overview = <br /> <br /> This wiki page describes how to tune Mellanox OpenStack on Ubuntu 12.04 servers<br /> <br /> = Prerequisites =<br /> <br /> * A running OpenStack environment installed with the ML2 plugin on top of OVS.<br /> * All nodes equipped with Mellanox ConnectX®-3 Network Adapter (http://www.mellanox.com/page/products_dyn?product_family=119) <br /> * Mellanox OFED 2.2 or greater installed on all nodes. Please refer to Mellanox website for the latest OFED: http://www.mellanox.com/page/products_dyn?product_family=26&amp;mtag=linux_sw_drivers<br /> * SR-IOV enabled on all compute nodes. For more information, please refer to Mellanox Community [http://community.mellanox.com/docs/DOC-1317 click here].<br /> * The software package iproute2 - (http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2 ) installed on all Compute nodes<br /> * VLANs configured on the ports in the switch.<br /> <br /> = Ethernet Network = <br /> <br /> == Controller Node == <br /> <br /> In /etc/nova/nova.conf, ensure you have the following line: <br /> security_group_api=nova<br /> <br /> If the value is changed, restart the Nova services:<br /> # for i in $(initctl list | grep '^nova' | awk '{print $1}'); do service $i restart; done<br /> <br /> === Neutron Server Node ===<br /> NOTE: The Neutron Server can sometimes be part of the Controller node.<br /> <br /> To configure the Neutron-Server node:<br /> <br /> 1. Edit /etc/neutron/neutron.conf, and make sure Neutron is using ML2 plugin:<br /> [DEFAULT]<br /> core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin<br /> 2. Modify the ML2 configuration file (/etc/neutron/plugins/ml2/ml2_conf.ini) as follows:<br /> [ml2]<br /> type_drivers = vlan,flat<br /> tenant_network_types = vlan<br /> mechanism_drivers = openvswitch,mlnx <br /> [ml2_type_vlan]<br /> network_vlan_ranges=&lt;physical_network&gt;:&lt;min_vlan&gt;:&lt;max_vlan&gt; //(i.e. default:2:10) <br /> [securitygroup]<br /> enable_security_group = True <br /> [eswitch]<br /> vnic_type = hostdev<br /> apply_profile_patch = True<br /> 3. Install the required Deb for the Mellanox Neutron plugin (from Ubuntu’s repository):<br /> apt-get install neutron-plugin-mlnx-agent<br /> 4. Edit /etc/neutron/plugins/mlnx/mlnx_conf.ini to match your environment:<br /> [mlnx]<br /> network_vlan_ranges = &lt;physical_network&gt;:&lt;min_vlan:&lt;max_vlan&gt; //(i.e. default:2:10)<br /> tenant_network_type = vlan <br /> [eswitch]<br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;eth interface&gt; //(i.e. default:eth0)<br /> vnic_type = hostdev<br /> [agent]<br /> support_old_agents = True<br /> <br /> 5. Restart neutron-server: <br /> /etc/init.d/neutron-server start<br /> <br /> == Compute Nodes ==<br /> <br /> To configure the Compute node: <br /> <br /> 1. Download mlnxvif and eSwitch Daemon (eSwitchd) from the following link:<br /> http://www.mellanox.com/downloads/solutions/openstack/icehouse/debian/<br /> 2. Install the Mellanox plugin services mlnxvif and eSwitchd:<br /> dpkg –i python-mlnxvif_2014.1.1-1_all.deb<br /> dpkg –i python-eswitchd_0.10-1_all.deb<br /> 3. Edit /etc/eswitchd/eswitchd.conf as follows:<br /> [DAEMON]<br /> fabrics=&lt;physical_network&gt;:&lt;eth-interface&gt; //(i.e. default:eth0)<br /> 4. Edit /etc/nova/nova.conf as follows:<br /> [DEFAULT]<br /> compute_driver = libvirt.LibvirtDriver<br /> libvirt_vif_driver = mlnxvif.vif.MlxEthVIFDriver<br /> security_group_api = nova<br /> 5. Check if “ebrctl” appears in one of Nova’s rootwrap files:<br /> grep 'ebrctl' /etc/nova/rootwrap.d/*<br /> If “ebrctl” does not appear in one of Nova’s rootwrap files, edit the file /etc/nova/rootwrap.d/compute.filters as follows:<br /> [Filters]<br /> ebrctl: CommandFilter, ebrctl, root<br /> 6. Install the Neutron Mellanox agent (from Ubuntu repository):<br /> apt-get install neutron-plugin-mlnx-agent<br /> 7. Copy /etc/neutron/plugins/mlnx/mlnx_conf.ini from the Neutron Server node to the same path on each Compute node.<br /> 8. Restart the following services:<br /> /etc/init.d/nova-compute restart<br /> /etc/init.d/eswitchd start<br /> /etc/init.d/neutron-plugin-mlnx-agent start<br /> <br /> == Network Node ==<br /> <br /> To configure the Network node:<br /> <br /> 1. Configure eIPoIB (for Linux bridge plugin), and edit the file /etc/infiniband/openib.conf as follows:<br /> E_IPOIB_LOAD=yes<br /> 2. Edit /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini as follows:<br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;eth-interface&gt; <br /> //(i.e. default:eth2)<br /> 3. Restart Open vSwitch agent:<br /> service neutron-openvswitch-agent restart<br /> 4. Copy /etc/neutron/neutron.conf from the Compute node to the same path on the Network node.<br /> <br /> 5. Edit /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini as follows:<br /> physical_interface_mappings = &lt;physical_network&gt;:&lt;eth interface&gt; <br /> //(i.e. default:eth2)<br /> 6. Edit /etc/neutron/dhcp_agent.ini and /etc/neutron/l3_agent.ini as follows:<br /> [DEFAULT] <br /> interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver<br /> 7. Restart the following services:<br /> service neutron-plugin-linuxbridge-agent restart <br /> service neutron-dhcp-agent restart<br /> service neutron-l3-agent restart<br /> <br /> == Troubleshooting ==<br /> <br /> * If you're seeing request timeout errors in the plugin's logs, make sure that the eswitchd service is started and try to increase the request_timeout value in the plugin configuration file.</div>

May