|
|
(138 intermediate revisions by 2 users not shown) |
Line 1: |
Line 1: |
| | | |
− |
| + | This page is no longer available, refer to https://wiki.openstack.org/wiki/Mellanox-Neutron |
− | = Overview =
| |
− | == Mellanox Quantum Plugin ==
| |
− | The Openstack Mellanox Quantum plugin supports Mellanox embedded switch functionality as part of the VPI (Ethernet/InfiniBand) HCA.
| |
− | Mellanox Quantum Plugin allows hardware vNICs (based on SR-IOV virtual functions) per each Virtual Machine vNIC to have its unique
| |
− | connectivity, security, and QoS attributes. Hardware vNICs can be mapped to the guest VMs through para-virtualization (using a Tap device), or directly as a Virtual PCI device to the guest, allowing higher performance and advanced features such as RDMA (remote direct memory access).
| |
− | | |
− | Hardware-based switching, provides better performance, functionality, and security/isolation for virtual cloud environments.
| |
− | Future versions of the plug-in will include OpenFlow API to control and monitor the embedded switch and vNICs functionality
| |
− | | |
− | This plugin is implemented according to Plugin-Agent pattern.
| |
− | | |
− | | |
− | +-----------------+ +--------------+
| |
− | | Controller node | | Compute node |
| |
− | +-----------------------------------+ +-----------------------------------+
| |
− | | +-----------+ +----------+ | | +----------+ +----------+ |
| |
− | | | | | | | | | | zmq | | |
| |
− | | | Openstack | v2.0 | Mellanox | | RPC | | Mellanox |REQ/REP| Mellanox | |
| |
− | | | Quantum +------+ Quantum +-----------+ Quantum +-------+ Embedded | |
| |
− | | | | | Plugin | | | | Agent | | Switch | |
| |
− | | | | | | | | | | | (NIC) | |
| |
− | | +-----------+ +----------+ | | +----------+ +----------+ |
| |
− | +-----------------------------------+ +-----------------------------------+
| |
− |
| |
− | * Openstack Mellanox Quantum Plugin implements the Quantum v2.0 API.
| |
− | * Mellanox Quantum Plugin processes the Quantum API calls and manages network segmentation ID allocation.
| |
− | * The plugin uses databases to store configuration and allocation mapping.
| |
− | * The plugin maintains compatibility to Linux Bridge Plugin, supports DHCP and L3 Agents by running L2 Linux Bridge Agent on Network Node.
| |
− | * Mellanox Openstack Quantum Agent (L2 Agent) runs on each compute node.
| |
− | * Agent should apply VIF connectivity based on mapping between a VIF (VM vNIC) and Embedded Switch port.
| |
− | | |
− | == Mellanox Nova VIF Driver ==
| |
− | The Mellanox Nova VIF driver should be used when running Mellanox Quantum Plugin. This driver supports the VIF plugin by binding vNIC (para-virtualized or SR-IOV with optional RDMA guest access) to the embedded switch port.
| |
− | | |
− | == Prerequisites ==
| |
− | '''The following are the Mellanox Quantum Plugin prerequisites:'''
| |
− | | |
− | 1. The software package python-zmq ([https://github.com/zeromq/pyzmq github]) must be installed. EPEL repository can be used as well.
| |
− | | |
− | 2. python-setuptools (use "yum install python-setuptools")
| |
− | | |
− | 3. python-pip (use "yum install python-pip")
| |
− | | |
− | '''Prerequisites only for Compute Servers :'''
| |
− | | |
− | 1. RH 6.3 or above.
| |
− | | |
− | 2. Compute nodes should be equiped with Mellanox ConnectX®-3 Network Adapter ([http://www.mellanox.com/page/infiniband_cards_overview link])
| |
− | | |
− | 3. Mellanox OFED 2.0.3 is installed. Contact [mailto:openstack@mellanox.com?subject=MLNX_OFED2.0 openstack@mellanox.com] to retreive this version.
| |
− | Refer to Mellanox website for the latest OFED documentation ([http://www.mellanox.com/page/products_dyn?product_family=26 link])
| |
− | | |
− | 4. Enable SR-IOV on ConnectX-3 card. Refer to [http://community.mellanox.com/community/develop/cloud-developers/blog/2013/04/07/mellanox-ofed-driver-installation-with-sr-iov Mellanox Community]
| |
− | | |
− | 5. The software package iproute2 - ([http://www.linuxgrill.com/anonymous/iproute2/ Code] [http://www.policyrouting.org/iproute2.doc.html Documentation]) must be installed. Reuired only to be installed on compute nodes
| |
− | | |
− | 6. The software package ethtool ([http://www.kernel.org/pub/software/network/ethtool/ Code]) must be installed (version 3.8 or higher). Reuired only to be installed on compute nodes.
| |
− | | |
− | 7. oslo.config (use "pip-python install oslo.config"). Reuired only to be installed on compute nodes.
| |
− | | |
− | == Code Structure ==
| |
− | | |
− | Mellanox Quantum Plugin and the Nova VIF driver are located at [http://community.mellanox.com/docs/DOC-1187 Mellanox OpenStack RPMs]
| |
− | | |
− | 1. Quantum Plugin package structure:
| |
− | quantum/etc/quantum/plugins/mlnx -plugin configuration
| |
− | mlnx_conf.ini - sample plugin configuration
| |
− | | |
− | quantum/quantum/plugins/mlnx - plugin code
| |
− | /agent - Agent code
| |
− | /common - common code
| |
− | /db - plugin persistency model and wrapping methods
| |
− | mlnx_plugin.py - Mellanox Openstack Plugin
| |
− | rpc_callbacks.py - RPC handler for received messages
| |
− | agent_notify_api.py - Agent RPC notify methods
| |
− | | |
− | Mellanox Quantum Plugin is located under /quantum/quantum/plugins/.
| |
− | | |
− | 2. Nova VIF driver package structure is:
| |
− | nova/nova/mlnx - nova vif driver code
| |
− | | |
− | Mellanox Nova VIF driver is located under /nova/virt/libvirt/.
| |
− | | |
− | = Mellanox Quantum Plugin Installation (for Grizzly) =
| |
− | == On the Quantum Server Node ==
| |
− | 1. Copy Mellanox OpenStack plugin to the installed quantum plugin directory (usually /usr/lib/python2.7/dist-packages/quantum/plugins).
| |
− | #cp -a mellanox-quantum-plugin/quantum/quantum/plugins/mlnx /usr/lib/python2.7/dist-packages/quantum/plugins
| |
− | | |
− | 2. Modify the /etc/quantum/quantum.conf file.
| |
− | core_plugin = quantum.plugins.mlnx.mlnx_plugin.MellanoxEswitchPlugin
| |
− | | |
− | 3. Copy the Mellanox plugin configuration.
| |
− | #mkdir -p /etc/quantum/plugins/mlnx
| |
− | #cp mellanox-quantum-plugin/quantum/etc/quantum/plugins/mlnx/mlnx_conf.ini /etc/quantum/plugins/mlnx
| |
− | | |
− | 4. Modify the /etc/quantum/plugins/mlnx/mlnx_conf.ini file to reflect your environment.
| |
− | | |
− | 5. If you run the Quantum server using init script - change the quantum server configuration to point to Mellanox Quantum Plugin
| |
− | | |
− | For example, In Ubuntu change /etc/default/quantum-server
| |
− | QUANTUM_PLUGIN_CONFIG="/etc/quantum/plugins/mlnx/mlnx_conf.ini"
| |
− | 6. Run the server.
| |
− | #quantum-server --config-file /etc/quantum/quantum.conf --config-file /etc/quantum/plugins/mlnx/mlnx_conf.ini
| |
− | or
| |
− | #/etc/init.d/quantum-server start
| |
− | | |
− | == On Compute Nodes ==
| |
− | | |
− | === Nova-compute ===
| |
− | | |
− | 1. Copy the Nova Mellanox VIF driver.
| |
− | #cp -a mellanox-quantum-plugin/nova/nova/virt/libvirt/mlnx /usr/lib/python2.6/site-packages/nova/virt/libvirt
| |
− | | |
− | 2. Modify nova.conf.
| |
− | compute_driver=nova.virt.libvirt.driver.LibvirtDriver
| |
− | libvirt_vif_driver=nova.virt.libvirt.mlnx.vif.MlxEthVIFDriver
| |
− | vnic_type=direct - can be either 'direct' or 'hostdev'
| |
− | fabric=default - specifies physical network for vNICs (currently support one fabric per node)
| |
− | | |
− | 3. Restart Nova.
| |
− | | |
− | === The eswitchd Daemon ===
| |
− | The eswitchd source files are located in the mellanox-eswitchd repository [https://github.com/mellanox-openstack/mellanox-eswitchd eSwitchd source repository]
| |
− | 1. Clone and install the eSwitchd
| |
− | #git clone https://github.com/mellanox-openstack/mellanox-eswitchd
| |
− | #cd mellanox-eswitchd
| |
− | #python setup.py install
| |
− | | |
− | 2. Copy the configuration and init files of the eSwitchd.
| |
− | #mkdir /etc/eswitchd
| |
− | #cp etc/eswitchd/eswitchd.conf /etc/eswitchd
| |
− | #cp etc/init.d/eswitchd /etc/init.d
| |
− | | |
− | 3. Modify the configuration file according to your environment
| |
− | # cat /etc/eswitchd/eswitchd.conf
| |
− | [DEFAULT]
| |
− | default_log_levels="eswitchd=DEBUG"
| |
− | verbose=False
| |
− | debug=False
| |
− | log_file=/var/log/eswitchd.log
| |
− | [DAEMON]
| |
− | '''fabrics='mlx1:eth4'''' --> Important line '<fabric_name>:<PF Name>'.
| |
− | --> The fabric name should match the name in quantum agent and/or openFlow agent.
| |
− | default_timeout=4000
| |
− | | |
− | 3. Run the daemon:
| |
− | # /etc/init.d/eswitchd start
| |
− | | |
− | === Quantum Agent ===
| |
− | 1. Copy Mellanox OpenStack agent.
| |
− | #cp -a mellanox-quantum-plugin/quantum/quantum/plugins/mlnx /usr/lib/python2.6/site-packages/quantum/plugins
| |
− | | |
− | 2. Copy the quantum.conf and mlnx_conf.ini file to the compute node.
| |
− | #mkdir -p /etc/quantum/plugins/mlnx
| |
− | #cp mellanox-quantum-plugin/quantum/etc/quantum/plugins/mlnx/mlnx_conf.ini /etc/quantum/plugins/mlnx
| |
− | | |
− | 3. Modify the Quantum Agent configuration at /etc/quantum/plugins/mlnx/mlnx_conf.ini.
| |
− | | |
− | 4. Run the agent.
| |
− | #python /usr/lib/python2.6/site-packages/quantum/plugins/mlnx/agent/eswitch_quantum_agent.py --config-file /etc/quantum/quantum.conf --config-file /etc/quantum/plugins/mlnx/mlnx_conf.ini
| |
− | | |
− | = Mellanox Quantum Plugin Installation (for Havana) =
| |
− | == On the Quantum Server Node ==
| |
− | | |
− | 1. Select Mellanox eSwitch plugin as core plugin for Quantum server.
| |
− |
| |
− | 2. Modify the /etc/quantum/plugins/mlnx/mlnx_conf.ini file to reflect your environment.
| |
− | | |
− | 3. If you run the Quantum server using init script - change the quantum server configuration to point to Mellanox Quantum Plugin
| |
− | | |
− | For example, In Ubuntu change /etc/default/quantum-server
| |
− | QUANTUM_PLUGIN_CONFIG="/etc/quantum/plugins/mlnx/mlnx_conf.ini"
| |
− | | |
− | 4. Run the server.
| |
− | #quantum-server --config-file /etc/quantum/quantum.conf --config-file /etc/quantum/plugins/mlnx/mlnx_conf.ini
| |
− | or
| |
− | #/etc/init.d/quantum-server start
| |
− | | |
− | == On Compute Nodes ==
| |
− | | |
− | === Nova-compute ===
| |
− | | |
− | 1. Copy the Nova Mellanox VIF driver.
| |
− | #cp -a mellanox-quantum-plugin/nova/nova/virt/libvirt/mlnx /usr/lib/python2.6/site-packages/nova/virt/libvirt
| |
− | | |
− | 2. Modify nova.conf.
| |
− | compute_driver=nova.virt.libvirt.driver.LibvirtDriver
| |
− | libvirt_vif_driver=nova.virt.libvirt.mlnx.vif.MlxEthVIFDriver
| |
− | vnic_type=direct - can be either 'direct' or 'hostdev'
| |
− | fabric=default - specifies physical network for vNICs (currently support one fabric per node)
| |
− | | |
− | 3. Restart Nova.
| |
− | | |
− | === The eswitchd Daemon ===
| |
− | The eswitchd source files are located in the mellanox-eswitchd repository [https://github.com/mellanox-openstack/mellanox-eswitchd eSwitchd source repository]
| |
− | 1. Clone and install the eSwitchd
| |
− | #git clone https://github.com/mellanox-openstack/mellanox-eswitchd
| |
− | #cd mellanox-eswitchd
| |
− | #python setup.py install
| |
− | | |
− | 2. Copy the configuration and init files of the eSwitchd.
| |
− | #mkdir /etc/eswitchd
| |
− | #cp etc/eswitchd/eswitchd.conf /etc/eswitchd
| |
− | #cp etc/init.d/eswitchd /etc/init.d
| |
− | | |
− | 3. Modify the configuration file according to your environment
| |
− | # cat /etc/eswitchd/eswitchd.conf
| |
− | [DEFAULT]
| |
− | default_log_levels="eswitchd=DEBUG"
| |
− | verbose=False
| |
− | debug=False
| |
− | log_file=/var/log/eswitchd.log
| |
− | [DAEMON]
| |
− | '''fabrics='mlx1:eth4'''' --> Important line '<fabric_name>:<PF Name>'.
| |
− | --> The fabric name should match the name in quantum agent and/or openFlow agent.
| |
− | default_timeout=4000
| |
− | | |
− | 3. Run the daemon:
| |
− | # /etc/init.d/eswitchd start
| |
− | | |
− | === Quantum Agent ===
| |
− | | |
− | 1. Modify the Quantum Agent configuration at /etc/quantum/plugins/mlnx/mlnx_conf.ini.
| |
− | | |
− | 2. Run the agent.
| |
− | #python /usr/lib/python2.6/site-packages/quantum/plugins/mlnx/agent/eswitch_quantum_agent.py --config-file /etc/quantum/quantum.conf --config-file /etc/quantum/plugins/mlnx/mlnx_conf.ini
| |
− | | |
− | = Mellanox Quantum Plugin Configuration =
| |
− | == Quantum Configuration ==
| |
− | 1. Make the Mellanox plugin the current quantum plugin by editing quantum.conf and changing core_plugin.
| |
− | core_plugin = quantum.plugins.mlnx.mlnx_plugin.MellanoxEswitchPlugin
| |
− | | |
− | 2. Database configuration: Install MySQL on the central server. Create a database named "quantum".
| |
− | | |
− | 3. Plugin configuration:
| |
− | Edit the configuration file: etc/quantum/plugins/mlnx/mlnx_conf.ini
| |
− | | |
− | On central server node
| |
− | | |
− | [DATABASE]
| |
− | sql_connection - must match the mysql configuration
| |
− | | |
− | [MLNX]
| |
− | tenant_network_type - must be set on of supported tenant network types.
| |
− | Possible values are 'vlan' for Ethernet or 'ib' for Infiniband.'vlan' is the default.
| |
− | network_vlan_ranges - must be configured to specify the names of the physical networks
| |
− | managed by the mellanox plugin, along with the ranges of VLAN IDs
| |
− | available on each physical network for allocation to virtual networks.
| |
− | Possible range is vlan range is 1-4093. The default is default:2:100 <fabric name >:<vlan range start>:<vlan end range>
| |
− |
| |
− | On compute node(s)
| |
− | | |
− | [AGENT]
| |
− | polling_interval - interfval to poll for existing vNICs
| |
− | rpc - must be set to True
| |
− | | |
− | [ESWITCH]
| |
− | physical_interface_mapping - the network_interface_mappings maps each physical network name to the physical interface (on top of Mellanox Adapter) connecting the node to that physical network.
| |
− | vnic_type - type of VM network interface: 'direct' or 'hostdev' according to libvirt terminology.
| |
− | hostdev: this is the traditional method of assigning any generic PCI device to a guest (SR-IOV).
| |
− | direct: this is a method to provide macvtap device on top of the PCI device (default).
| |
− | daemon_endpoint - eswitch daemon end point connection (URL) (default value='tcp://127.0.0.1:500')
| |
− | request_timeout - the number of milliseconds the agent will wait for response on request to daemon. (default=3000 msec)
| |
− |
| |
− | For a plugin configuration file example, please refer to [https://github.com/mellanox-openstack/mellanox-quantum-plugin/blob/master/quantum/etc/quantum/plugins/mlnx/mlnx_conf.ini Mellanox config *ini file].
| |
− | | |
− | == Nova Configuration (Compute Node(s)) ==
| |
− | ------------------------------------
| |
− | Edit the nova.conf file.
| |
− | 1. Configure the vif driver, and libvirt/vif type
| |
− | compute_driver=nova.virt.libvirt.driver.LibvirtDriver
| |
− | connection_type=libvirt
| |
− | libvirt_vif_driver=nova.virt.libvirt.mlnx.vif.MlxEthVIFDriver
| |
− | 2. Configure vnic_type ('direct' or 'hostdev'). This will be the default vNIC type when creating a new vNIC. (for example, when using the dashboard it to create new vNIC, this will be the default type.
| |
− | vnic_type= direct
| |
− | 3. Define the embedded switch-managed physical network (currently single fabric on node).
| |
− | fabric=default - specifies physical network for vNICs
| |
− | 4. Enable DHCP server to allow VMs to acquire IPs.
| |
− | quantum_use_dhcp=true
| |
− | | |
− | = Usage Examples =
| |
− | * In order to create SR-IOV interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating an SR-IOV Instance" chapter]
| |
− | * In order to create Para-Virtualized interface refer to [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack solution document "Creating a Para-Virtualized vNIC Instance" chapter]
| |
− | | |
− | = References =
| |
− | 1. [http://www.mellanox.com/openstack/ http://www.mellanox.com/openstack/]
| |
− | | |
− | 2. [https://github.com/mellanox-openstack Source repository]
| |
− | | |
− | 3. [http://www.mellanox.com/page/products_dyn?product_family=26 Mellanox OFED]
| |
− | | |
− | 4. [http://www.mellanox.com/sdn/stage/pdf/Mellanox-OpenStack-OpenFlow-Solution.pdf Mellanox OpenStack Solution Reference Architecture]
| |
− | | |
− | For more details, please refer your question to [mailto:openstack@mellanox.com openstack@mellanox.com]
| |
− | | |
− | Return to [https://wiki.openstack.org/wiki/Mellanox-OpenStack Mellanox-OpenStack] wiki page.
| |