Jump to: navigation, search

Difference between revisions of "Mellanox-Cinder"

(References)
m
 
(14 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 
= Overview =  
 
= Overview =  
iSER (iSCSI over RDMA) Mellanox OpenStack support to Cinder.<br />
+
Mellanox added iSCSI Extensions for RDMA (iSER) that does not need the TCP layer. Therefore, it provides lower latency and higher performance.<br />
This can allow 5x faster bandwidth compared to using iSCSI TCP. <br /><br />
+
It permits data to be transferred directly into and out of SCSI computer memory buffers, which connects computers to storage devices, without intermediate data copies.<br />
For example, over RAM device LUN I got ~1.3GBps Vs. ~5.5GBps (TCP Vs. iSER), and much lower CPU overhead.
+
iSER is a server network protocol that extends the Internet Small Computer System Interface (iSCSI) protocol to use Remote Direct Memory Access (RDMA).<br /><br />
 
+
Please contact Mellanox for further information and benchmark results.
= Havana Release =
 
  
 
== Installation ==
 
== Installation ==
The Mellanox Cinder pacakge is embeded within the OpenStack Cinder, no patches or plugins are needed.  
+
Provisioned through standard Cinder installation.
Just follow the formal OpenStack Cinder installation
 
  
 
== Configuration ==
 
== Configuration ==
In order to enable iSER, need to adjust these values at '''/etc/cinder/cinder.conf''':
+
In order to enable iSER, adjust these values at '''/etc/cinder/cinder.conf''':
 
 
iser_ip_address = <ipoib/roce_address>
 
volume_driver = cinder.volume.drivers.lvm.LVMISERDriver
 
 
 
'''iser_ip_address''' is required to do a "discovery" over the IB/RoCE interface from the initiator side.
 
 
 
'''volume_driver''' points cinder to use the ISERDriver, instead the LVMISCSIDriver.
 
  
 +
iscsi_ip_address = <ipoib/roce_address>
 +
iscsi_protocol = iser
 +
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
  
On nova-compute side change the following in '''/etc/nova/nova.conf''':
+
'''iscsi_ip_address''' is required to do a "discovery" over the IB/RoCE interface from the initiator side.
  
libvirt_volume_drivers = iser=nova.virt.libvirt.volume.LibvirtISERVolumeDriver
+
'''volume_driver''' points cinder to use the LVMVolumeDriver.
  
Restart Cinder and Nova compute services
+
Restart Cinder service
  
 
  #/etc/init.d/openstack-cinder-volume restart
 
  #/etc/init.d/openstack-cinder-volume restart
#/etc/init.d/openstack-nova-compute restart
 
 
For troubleshooting issues refer to section [https://wiki.openstack.org/wiki/Mellanox-Cinder#Known_Issues Known Issues]
 
 
= Grizzly Release =
 
== Installation ==
 
=== Getting the code ===
 
wget https://github.com/mellanox-openstack/cinder-iser/archive/stable/grizzly.zip
 
cd cinder-iser-stable-grizzly
 
 
=== Applying the patches ===
 
You have two choices:
 
# To apply this support, replace the files under "cinder/cinder/" And "nova/nova/" Respectively.
 
# Apply the patches under "cinder/" And "nova/", don't forget to copy "cinder/cinder/volume/iser.py" if you choose this way.
 
 
== Configuration ==
 
In order to enable iSER, need to adjust these values at '''/etc/cinder/cinder.conf''':
 
  
iser_ip_address = <ipoib/roce_address>
+
For troubleshooting issues, refer to section [https://wiki.openstack.org/wiki/Mellanox-Cinder#Known_Issues Known Issues]
volume_driver = cinder.volume.drivers.lvm.LVMISERDriver
 
 
 
'''iser_ip_address''' is required to do a "discovery" over the IB/RoCE interface from the initiator side.
 
 
 
'''volume_driver''' points cinder to use the ISERDriver, instead the LVMISCSIDriver.
 
 
 
 
 
On nova-compute side change the following in '''/etc/nova/nova.conf''':
 
 
 
libvirt_volume_drivers = iser=nova.virt.libvirt.volume.LibvirtISERVolumeDriver
 
 
 
Restart Cinder and Nova compute services
 
 
 
#/etc/init.d/openstack-cinder-volume restart
 
#/etc/init.d/openstack-nova-compute restart
 
  
 
= Known Issues =
 
= Known Issues =
  
(1)  “scsi-target-utils” package  
+
(1)  “scsi-target-utils” package:
  
Note: for RH6.4 or below: “scsi-target-utils” package needs to be 1.0.38 and up, if you use RDO installation you should check the version.
+
'''Note''': For RH6.4 or below, “scsi-target-utils” package version needs to be 1.0.38 and above. If you use RDO installation, you should check the version.
The RPM can be found in http://www.mellanox.com/downloads/solutions/rpms/
+
The RPM can be found at http://www.mellanox.com/downloads/solutions/rpms/
 
   
 
   
 
  #wget http://www.mellanox.com/downloads/solutions/rpms/scsi-target-utils-1.0.39-v1.0.39.c1135a.x86_64.rpm
 
  #wget http://www.mellanox.com/downloads/solutions/rpms/scsi-target-utils-1.0.39-v1.0.39.c1135a.x86_64.rpm
 
  #rpm -Uvh scsi-target-utils-1.0.39-v1.0.39.c1135a.x86_64.rpm
 
  #rpm -Uvh scsi-target-utils-1.0.39-v1.0.39.c1135a.x86_64.rpm
  
In addition, RH6.5 has “scsi-target-utils” package with version 1.0.24-10, which is good enough, no further action is needed for RHEL6.5
+
In addition, RH6.5 includes “scsi-target-utils” package with version 1.0.24-10. This version number is sufficient and no further action is needed for RHEL6.5.
  
(2) Boot Instance from volume fails when iSER is enabled
+
(2) Using old versions of Openstack (Havana release or lower), boot Instance from volume fails when iSER is enabled:
  
 
Steps to reproduce the bug:
 
Steps to reproduce the bug:
Line 86: Line 48:
 
[https://bugs.launchpad.net/nova/+bug/1253578]
 
[https://bugs.launchpad.net/nova/+bug/1253578]
  
A patch can be found in here: [http://www.mellanox.com/downloads/solutions/openstack/havana/patches/volume.patch]
+
A patch can be found here: [http://www.mellanox.com/downloads/solutions/openstack/havana/patches/volume.patch]
  
To apply the patch run the following commands on '''all nova-compute nodes''':
+
To apply the patch, run the following commands on '''all nova-compute nodes''':
 
  # wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/volume.patch  
 
  # wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/volume.patch  
 
  # patch -p1 < volume.patch
 
  # patch -p1 < volume.patch
 
  # service nova-compute restart  
 
  # service nova-compute restart  
  
(3) Flow control
+
(3) Flow control:
  
In case the network is Ethernet, Make sure that flow control is enabled on the switches on the relevant ports across the network.
+
In case the network is Ethernet, make sure that flow control is enabled on the switches on the relevant ports across the network.
 +
 
 +
(4) Linux-IO (LIO) support limitation:
 +
 
 +
Open-source implementation of SCSI target in Openstack can be managed by either TGT or LIO.<br />
 +
In Openstack, iSER is supported over TGT only until the Kilo version, and iSER support over LIO is available from the Kilo release.<br />
 +
Please use iscsi_helper parameter to specify the ISCSI target type (for more details, please follow: [http://docs.openstack.org/kilo/config-reference/content/section_volume-misc.html Cinder editional options]).
  
 
= References =
 
= References =
1. [http://www.mellanox.com/openstack/ OpenStack solution page at Mellanox site]
+
1. [http://docs.openstack.org/kilo/config-reference/content/lvm-volume-driver.html LVM driver documentation]
 +
 
 +
2. [http://www.mellanox.com/openstack/ OpenStack solution page at Mellanox site]
 +
 
 +
3. [https://github.com/mellanox-openstack Legacy source repository]
 +
 
 +
4. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED web page]
  
2. [https://github.com/mellanox-openstack Source repository]
+
5. [http://docs.openstack.org/havana/install-guide/install/yum/content/cinder-controller.html Cinder-controller for Havana]
  
3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED web page]
+
6. [http://docs.openstack.org/havana/install-guide/install/yum/content/cinder-node.html Cinder-node for Havana]
  
4. [http://docs.openstack.org/havana/install-guide/install/yum/content/cinder-controller.html Cinder-controller]
 
  
5. [http://docs.openstack.org/havana/install-guide/install/yum/content/cinder-node.html Cinder-node]
 
  
 
Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack  Mellanox-OpenStack] wiki page.
 
Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack  Mellanox-OpenStack] wiki page.
  
For more details, please refer any inquiries to  [mailto:openstack@mellanox.com openstack@mellanox.com].
+
For more details, please direct any inquiries to  [mailto:openstack@mellanox.com openstack@mellanox.com].

Latest revision as of 07:45, 22 November 2016

Overview

Mellanox added iSCSI Extensions for RDMA (iSER) that does not need the TCP layer. Therefore, it provides lower latency and higher performance.
It permits data to be transferred directly into and out of SCSI computer memory buffers, which connects computers to storage devices, without intermediate data copies.
iSER is a server network protocol that extends the Internet Small Computer System Interface (iSCSI) protocol to use Remote Direct Memory Access (RDMA).

Please contact Mellanox for further information and benchmark results.

Installation

Provisioned through standard Cinder installation.

Configuration

In order to enable iSER, adjust these values at /etc/cinder/cinder.conf:

iscsi_ip_address = <ipoib/roce_address>
iscsi_protocol = iser
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver

iscsi_ip_address is required to do a "discovery" over the IB/RoCE interface from the initiator side.

volume_driver points cinder to use the LVMVolumeDriver.

Restart Cinder service

#/etc/init.d/openstack-cinder-volume restart

For troubleshooting issues, refer to section Known Issues

Known Issues

(1) “scsi-target-utils” package:

Note: For RH6.4 or below, “scsi-target-utils” package version needs to be 1.0.38 and above. If you use RDO installation, you should check the version. The RPM can be found at http://www.mellanox.com/downloads/solutions/rpms/

#wget http://www.mellanox.com/downloads/solutions/rpms/scsi-target-utils-1.0.39-v1.0.39.c1135a.x86_64.rpm
#rpm -Uvh scsi-target-utils-1.0.39-v1.0.39.c1135a.x86_64.rpm

In addition, RH6.5 includes “scsi-target-utils” package with version 1.0.24-10. This version number is sufficient and no further action is needed for RHEL6.5.

(2) Using old versions of Openstack (Havana release or lower), boot Instance from volume fails when iSER is enabled:

Steps to reproduce the bug:

- Create new volume with image source.

- Launch a new instance boot from the above volume using iSER transport.

The bug is defined here: [1]

A patch can be found here: [2]

To apply the patch, run the following commands on all nova-compute nodes:

# wget http://www.mellanox.com/downloads/solutions/openstack/havana/patches/volume.patch 
# patch -p1 < volume.patch
# service nova-compute restart 

(3) Flow control:

In case the network is Ethernet, make sure that flow control is enabled on the switches on the relevant ports across the network.

(4) Linux-IO (LIO) support limitation:

Open-source implementation of SCSI target in Openstack can be managed by either TGT or LIO.
In Openstack, iSER is supported over TGT only until the Kilo version, and iSER support over LIO is available from the Kilo release.
Please use iscsi_helper parameter to specify the ISCSI target type (for more details, please follow: Cinder editional options).

References

1. LVM driver documentation

2. OpenStack solution page at Mellanox site

3. Legacy source repository

4. Mellanox OFED web page

5. Cinder-controller for Havana

6. Cinder-node for Havana


Return to Mellanox-OpenStack wiki page.

For more details, please direct any inquiries to openstack@mellanox.com.