Jump to: navigation, search

Difference between revisions of "Nova/proposal about usb passthrough"

(Created page with "== USB device and USB controller features == === Background === Currently, nova has supported function of pci-passthrough, but hasn't supported usb-passthrough. There were a...")
 
m (correct typo)
 
(39 intermediate revisions by one other user not shown)
Line 2: Line 2:
  
 
=== Background ===
 
=== Background ===
−
Currently, nova has supported function of pci-passthrough, but hasn't supported usb-passthrough.
+
Currently, nova has supported feature of pci-passthrough, but hasn't supported usb-passthrough.
−
There were a bp about it in https://blueprints.launchpad.net/nova/+spec/host-usb-passthrough but hasn't progress by now.
+
There were a bp about it in https://blueprints.launchpad.net/nova/+spec/host-usb-passthrough but don't have progress by now.
  
−
As I know, some telecom and enterprise customers have desired for usb-passthrough especially in private cloud.
+
As I know, some telecom and enterprise customers have requirement of usb-passthrough especially in private cloud.
−
So I think maybe it's a good choice to provide this function in openstack.
+
So I think maybe it's a good choice to provide this feature in openstack.
  
 
=== Use case ===
 
=== Use case ===
 
Our customers have the following requirement:
 
Our customers have the following requirement:
−
1. There are some erp softwares deployed in virtual machines that need usb-key for authentication. Customers will first insert usb-key to host, then use the cloud management software to attach usb-key to vm,
 
−
    the last guest os will discovery usb-key and we can use it.
 
−
2. Current popular usb device are supporting usb 2.0 or usb 3.0 standard, so our cloud management software should guarantee to use appropriate usb controller
 
  
−
=== Technical verification ===
+
1. There are some ERP softwares deployed in virtual machines that need usb-key for authentication. Customers will first insert usb-key to host, then use the cloud management software to attach usb-key to vm,  the last guest os will discovery usb-key and we can use it.
−
1. Test case
 
−
    Creating vm with different type and number usb controller and usb device.
 
  
−
2. Xml definition
+
2. Popular usb device are supporting usb 2.0 or usb 3.0 standard, so our cloud management software should guarantee the transfer speed of usb device.
−
  (1)Sample of usb controller:
 
−
    <controller type='usb' index='2' model='piix3-uhci'/>
 
−
    <controller type='usb' index='1' model='ehci'/>
 
  
−
  (2)Sample of pass-through usb devices
+
=== Test case ===
−
    <hostdev mode='subsystem' type='usb'>
+
I have tested some cases of creating vm with different type and amount of usb controllers and usb devices.<br/>
−
    < source>
+
I got the following result:<br/>
−
      <vendor id='0x136b'/>
+
1. Usb controllers have a maximum port restriction. Uhci controller supports a maxmum of 2 ports while ehci controller supports 6 ports.<br/>
−
      <product id='0x0003'/>
+
2. If we create vm without usb controller, qemu will create a piix3-usb-uhci controller by default but not create the other type's.<br/>
−
      <address bus='2' device='2'/>
+
3. If we create vm with usb device but no usb controller, the usb device will default be attached to the default uhci controller, but speed may mismatch.<br/>
−
    </source>
+
4. There are two ways to support more usb devices. 1. Adding cascaded hubs and attaching usb devices to them. 2.Creating more controllers and connect usb devices to different ones.<br/>
−
    <address type='usb' bus='1' port='1'/>
+
5. Some articles said that we can use product+vendor or bus+device properties to indentify a usb device, but we found that if we reboot host or plug/unplug usb device to/from host the device number may change, so use bus+port to identify a usb device maybe a good choice.
−
    </hostdev>
 
  
−
  (3)Sample of emulated usb disks
+
=== Consideration about usb controller ===
−
      <disk type='file' device='disk'>
+
I have three alternatives to implement this feature:
−
      <driver name='qemu' type='qcow2'/>
 
−
      < source file='/home/vms/usb/sdb.qcow2'/>
 
−
      <target dev='sdb' bus='usb'/>
 
−
      <address type='usb' bus='1' port='1'/>
 
−
    </disk>
 
  
−
3. The test results
+
Option 1:
−
  (1)Usb controllers have a maximum port restriction. Uhci controller supports a maxmum of 2 ports while ehci controller supports 6.
 
−
  (2)If creating vm without usb controller, qemu will default create a piix3-usb-uhci controller but not creating the other type's.
 
−
  (3)If creating vm with usb device but no usb controller, the usb device will default be attached to the default uhci controller, but speed may mismatch.
 
−
  (4)There are two ways to support more usb devices. 1. Adding cascaded hubs and attaching usb devices to them. 2.Creating more controllers and connect usb devices to different ones.
 
−
  (5)When creating vm with a pair of usb hub and ehci controller, the vm crashed. Qemu thrown a error(why?)
 
  
−
=== Proposed solution ===
+
Nova supports function and API  for creating usb controller with specified type and number.
−
1. Use cases
+
When admin/user want to attach usb devices to virtual machine, they should specify the type of usb controllers, then nova will select matched usb controllers/ports which are created by admin/user for these device.If usb controllers/ports are not enough, admin/user should create new ones.  
−
    Case 1:
 
−
    (1)Admin/User request to create a flavor with usb controllers(key arguments: type, count).
 
−
    (2)System creates a flavor containing usb controller information.
 
−
    (3)Admin/User request to create a vm with flavor created above.  
 
−
    (4)System first validates usb arguments specified in flavor, if valid then create a vm with usb controllers.  
 
  
−
    Case 2:
+
Advantage:
−
    (1)Admin/User request to create a flavor with usb devices(key arguments: unique identifiers of devices).
+
(1)Admin/user can choice what they want.
−
    (2)System creates a flavor containing usb device information.
+
(2)System is flexible for future extention.
−
    (3)Admin/User request to create a vm with flavor created above.
 
−
    (4)System first validates if usb devices are exist, available, and in the same host, if valid then create a vm with usb devices in the host. The usb devices will be attached to the default uhci controller created by qemu .  
 
  
−
    Case 3:  
+
Shortcoming:
−
    (1)Admin/User request to create a flavor with usb controllers(key argument: type, count) and usb devices(key arguments: identifiers and usb controller type).
+
(1)Admin/user's operations are complex.
−
    (2)System creates a flavor containing usb device information and usb controller information.
+
(2)The implementation of this function may be complicated. System should make decision about which usb device to which usb controller and save the relationship.
−
    (3)Admin/User request to create a vm with flavor created above.
 
−
    (4)Admin/User first validate if usb devices are exist, available,in the same host, and matching with controllers(type, have available port and so on), if valid then create a vm with usb devices and usb controllers in the host.
 
  
−
2. System requirements and things should be considered for usb controller(An optional Implementation):
+
Option 2:  
−
    (1)System supports creating a flavor with usb controller.
 
−
        Adding a property for usb controller to extra_specs field of flavor. The scheme may like {"usb_controllers":[{"type":"ehci", "count": 1}]}
 
−
        '''Note:''' Because libvirt/qemu will create a default pii3-uhi controller, so system will only create usb controllers except pii3-uhci. 
 
  
−
    (2)System supports creating vm with usb controller.
+
Nova dosen't expose operation of usb controller to admin/user.
−
        Driver(ibvirt) layper need to support constructing usb controller xml configuration.
+
When admin/user want to attach usb devices to virtual machine, they can specify the type of usb controllers optionaly, then nova will automatically create appropriate usb controllers/ports or choose exist ones for these devices.
  
−
    (3)System supports managing relationship between usb controllers and usb devices and storing them in instance_system_metadata table, so that
+
Advantage:
−
      <1>every usb device will have unique port, <2>vm can create usb controller and attach usb devices correctly the next starting.  
+
(1)Admin/user's operations are simple.
−
        The usb controller schema may like:
 
−
        {
 
−
          "usb_controller_requests":
 
−
          [
 
−
            {
 
−
                "type": "ehci",
 
−
                "count": 2,
 
−
                "usb_controllers":
 
−
                [
 
−
                  {
 
−
                    "index": 1,
 
−
                    "ports":
 
−
                    [
 
−
                      {
 
−
                        "port": 1,
 
−
                        "usb_device_id": 5
 
−
                      }
 
−
                    ]
 
−
                  }
 
−
                ]
 
−
            }
 
−
          ]
 
−
        }
 
  
−
3. System requirements and things should be considered for usb device(An optional Implementation):
+
Shortcoming:
−
    (1)System supports discovering usb devices and saving them to DB.
+
(1)The implementation of this function may be complicated. System should make decision about when to create usb controller, which usb device to which usb controller and save the relationship.
−
        The main process can refer to pci-passthrough implements:
 
−
        <1>Using white-list function and lsusb like commands to retrieve available usb devices of a host.
 
−
        <2>Adding a table usb_devices in DB to store usb device information.
 
−
        <3>Adding auto-update function to synchronize information between host(compute node) and DB.  
 
  
−
    (2)System supports creating a flavor with usb device.
+
Option 3:  
−
        Adding a property for usb device to extra_specs field of flavor. The scheme may like {"usb_devices":[{"usb_device_id": 5}]}
 
−
        '''Note:'''
 
−
        It is necessary to exposure usb device id(maybe some other information also are necessary) information to admin/user because the most common scenarios are attaching specified usb device to vm.
 
−
        The mechanism of automatically selecting compute node to provide pci devices is not appropriate in here.
 
  
−
    (3)System supports creating vm with usb devices.
+
Nova creates some default usb controller(like just create a ehci usb controller) when creating virtual machine, then the virtual machine will have a default uhci controller and a ehci controller.  
−
        <1>Scheduler layer needs to scheduler request to the appropriate compute node, so a UsbDeviceFilter.py filter may be needed.
+
When admin/user want to attach usb devices to virtual machine, they can specify the type of usb controller optionaly(null or ehci), then nova will attach usb device to uhci or ehci controller.
−
        <2>Driver(ibvirt) layper supports constructing usb device xml configuration.
 
−
        <3>Compute layer supports allocating usb controller for usb device.
 
−
              If not specifying usb controllers in request, compute layer not specify usb controller for usb device, then libvirt/qemu will attach usb device to the default pii3-uhci controller.
 
−
              If specifying usb controllers in request, system will first create usb controller which processes have been described above. It then choose available controller(type and port) for usb device.
 
−
        <4>DB layper supports storing usb devices. The instance_system_metadata table is a good choice.
 
−
              The usb device schema may like
 
−
              {
 
−
                "usb_devices_requests":
 
−
                [
 
−
                  {
 
−
                    "usb_device_id": 5
 
−
                  }
 
−
                ]
 
−
              }
 
  
−
    (4)System supports detaching usb devices when suspending a vm with usb devices, and re-attaching usb devices when resuming the vm.
+
Advantage:
 +
(1)Admin/user's operations are simple.
 +
(2)The implementation of this function is simple.
 +
(3)In real business scenes a vm commonly dosen't need very much usb devices. Uhci usb controller can support many usb devices by using cascaded hubs, ehci usb controller can support maxmum of 6 usb device, may be they are enough for use.  
  
−
    (5)System supports detaching usb devices before making snapshot of a vm and re-attaching usb devices after making snapshot. System also support the protection of usb device when using snapshot to resume a vm.
+
Shortcoming:
 +
(1)Only support some types of usb controller.
 +
(2)The number of supported usb devices are restricted.
  
−
    (6)System supports reporting error when migrating vm with usb device.
+
=== Consideration about usb-passthrough and hot-plug ===
−
        If a vm has usb devices attached, it is not allowed to migrate.
+
The core ideas are referring to the implementation of pci-passthrough. But there is an important difference that for pci device nova can automatically select a compute node to provide pci devices to vm but for usb device user should specify which use device to which vm.
 +
 
 +
===== Things should be considered for usb device: =====
 +
1. Nova should support auto-discovering usb devices and saving them to DB.<br/>
 +
2. Nova should support creating a flavor with usb device.<br/>
 +
3. Nova should support creating vm with flavor containing usb devices.<br/>
 +
4. Nova should support automatically attaching or detaching usb devices in some use cases like starting virtual machine, suspend/resume virtual machine, make snapshot of virtual machine, and so on.<br/>
 +
5. Nova should hot-plug/cold-plug usb devices to an exist virtual machine.<br/>
 +
 
 +
===== Related API: =====
 +
Nova should support the following interfaces.<br/>
 +
1. We should add usb device API so that nova can support querying usb device information. <br/>
 +
List usb devices on the nodes request:<br>
 +
GET v2/​{tenant_id}/​os-usb-device
 +
The response JSON may like below :
 +
{"usb_devices": [{"id": 1, "compute_node_id": 1, "vendor_id": "8086", "product_id": "1520", "address":"0000:01:00.0", "status":"available"}]}
 +
Request:
 +
GET v2/​{tenant_id}/​os-usb-device/detail
 +
The response JSON may like below :
 +
{"usb_devices": [{"id": 1, "compute_node_id": 1, "vendor_id": "8086", "product_id": "1520", "address":"0000:01:00.0", "dev_id": "usb_0000_04_10_0",  "dev_type":"type-USB", "status":"available", "instance_uuid":"", "extra_info":"", "created_at": null, "deleted": false,  "deleted_at": null,}]}
 +
Show usb device infomation Request:
 +
GET v2/​{tenant_id}/​os-usb-device/{usb_device_id}
 +
The response JSON may like below :
 +
{"usb_device": {"id": 1, "compute_node_id": 1, "vendor_id": "8086", "product_id": "1520", "address":"0000:01:00.0", "dev_id": "usb_0000_04_10_0",  "dev_type":"type-USB", "status":"available", "instance_uuid":"", "extra_info":"", "created_at": null, "deleted": false,  "deleted_at": null,}}
 +
 
 +
2. We should extend hypervisor API to add usb device stats information so that nova can support querying usb device stats of a hypervisor. <br/>
 +
List and show usb device stats on hypervisors request:
 +
GET v2/​{tenant_id}​/os-hypervisors/​{hypervisor_hostname}
 +
The response JSON contains the variable "usb_device_stats":
 +
{"hypervisor": {"usb_stats": [{"id": 1, "vendor_id": "8086", "product_id": "1520", "address":"0000:01:00.0", "status":"available", "instance_uuid":""}]}}
 +
 
 +
3.We should extend server API to add usb assignment information so that nova can support querying usb device stats of a server.<br/>
 +
List and show usb devices of a virtual machine:
 +
GET v2/​{tenant_id}​/servers/​{server_id}
 +
The response JSON will contain the variable "os-usb-device:usb_devices":
 +
{ "server": {"os-usb-device:usb_devices": [{"id": 1}], "tenant_id": "openstack", "user_id": "admin"}}
 +
 
 +
4.We should extend server API to add usb device hot-plug and cold-plug function.
 +
Attach usb device:
 +
Post  v2/​{tenant_id}​/servers/​{server_id}/action
 +
  {
 +
      "attach_usb_devices": {"usb_devices_requests":[{"id": 1}]}
 +
  }
 +
Detach usb device:
 +
Post  v2/​{tenant_id}​/servers/​{server_id}/action
 +
  {
 +
      "detach_usb_devices": {"usb_devices_requests":[{"id": 1}]}
 +
  }

Latest revision as of 12:24, 14 August 2015

USB device and USB controller features

Background

Currently, nova has supported feature of pci-passthrough, but hasn't supported usb-passthrough. There were a bp about it in https://blueprints.launchpad.net/nova/+spec/host-usb-passthrough but don't have progress by now.

As I know, some telecom and enterprise customers have requirement of usb-passthrough especially in private cloud. So I think maybe it's a good choice to provide this feature in openstack.

Use case

Our customers have the following requirement:

1. There are some ERP softwares deployed in virtual machines that need usb-key for authentication. Customers will first insert usb-key to host, then use the cloud management software to attach usb-key to vm, the last guest os will discovery usb-key and we can use it.

2. Popular usb device are supporting usb 2.0 or usb 3.0 standard, so our cloud management software should guarantee the transfer speed of usb device.

Test case

I have tested some cases of creating vm with different type and amount of usb controllers and usb devices.
I got the following result:
1. Usb controllers have a maximum port restriction. Uhci controller supports a maxmum of 2 ports while ehci controller supports 6 ports.
2. If we create vm without usb controller, qemu will create a piix3-usb-uhci controller by default but not create the other type's.
3. If we create vm with usb device but no usb controller, the usb device will default be attached to the default uhci controller, but speed may mismatch.
4. There are two ways to support more usb devices. 1. Adding cascaded hubs and attaching usb devices to them. 2.Creating more controllers and connect usb devices to different ones.
5. Some articles said that we can use product+vendor or bus+device properties to indentify a usb device, but we found that if we reboot host or plug/unplug usb device to/from host the device number may change, so use bus+port to identify a usb device maybe a good choice.

Consideration about usb controller

I have three alternatives to implement this feature:

Option 1:

Nova supports function and API for creating usb controller with specified type and number. When admin/user want to attach usb devices to virtual machine, they should specify the type of usb controllers, then nova will select matched usb controllers/ports which are created by admin/user for these device.If usb controllers/ports are not enough, admin/user should create new ones.

Advantage: (1)Admin/user can choice what they want. (2)System is flexible for future extention.

Shortcoming: (1)Admin/user's operations are complex. (2)The implementation of this function may be complicated. System should make decision about which usb device to which usb controller and save the relationship.

Option 2:

Nova dosen't expose operation of usb controller to admin/user. When admin/user want to attach usb devices to virtual machine, they can specify the type of usb controllers optionaly, then nova will automatically create appropriate usb controllers/ports or choose exist ones for these devices.

Advantage: (1)Admin/user's operations are simple.

Shortcoming: (1)The implementation of this function may be complicated. System should make decision about when to create usb controller, which usb device to which usb controller and save the relationship.

Option 3:

Nova creates some default usb controller(like just create a ehci usb controller) when creating virtual machine, then the virtual machine will have a default uhci controller and a ehci controller. When admin/user want to attach usb devices to virtual machine, they can specify the type of usb controller optionaly(null or ehci), then nova will attach usb device to uhci or ehci controller.

Advantage: (1)Admin/user's operations are simple. (2)The implementation of this function is simple. (3)In real business scenes a vm commonly dosen't need very much usb devices. Uhci usb controller can support many usb devices by using cascaded hubs, ehci usb controller can support maxmum of 6 usb device, may be they are enough for use.

Shortcoming: (1)Only support some types of usb controller. (2)The number of supported usb devices are restricted.

Consideration about usb-passthrough and hot-plug

The core ideas are referring to the implementation of pci-passthrough. But there is an important difference that for pci device nova can automatically select a compute node to provide pci devices to vm but for usb device user should specify which use device to which vm.

Things should be considered for usb device:

1. Nova should support auto-discovering usb devices and saving them to DB.
2. Nova should support creating a flavor with usb device.
3. Nova should support creating vm with flavor containing usb devices.
4. Nova should support automatically attaching or detaching usb devices in some use cases like starting virtual machine, suspend/resume virtual machine, make snapshot of virtual machine, and so on.
5. Nova should hot-plug/cold-plug usb devices to an exist virtual machine.

Related API:

Nova should support the following interfaces.
1. We should add usb device API so that nova can support querying usb device information.
List usb devices on the nodes request:

GET v2/​{tenant_id}/​os-usb-device

The response JSON may like below : {"usb_devices": [{"id": 1, "compute_node_id": 1, "vendor_id": "8086", "product_id": "1520", "address":"0000:01:00.0", "status":"available"}]} Request: 

GET v2/​{tenant_id}/​os-usb-device/detail

The response JSON may like below : 

{"usb_devices": [{"id": 1, "compute_node_id": 1, "vendor_id": "8086", "product_id": "1520", "address":"0000:01:00.0", "dev_id": "usb_0000_04_10_0",  "dev_type":"type-USB", "status":"available", "instance_uuid":"", "extra_info":"", "created_at": null, "deleted": false,  "deleted_at": null,}]}

Show usb device infomation Request: 

GET v2/​{tenant_id}/​os-usb-device/{usb_device_id}

The response JSON may like below : 

{"usb_device": {"id": 1, "compute_node_id": 1, "vendor_id": "8086", "product_id": "1520", "address":"0000:01:00.0", "dev_id": "usb_0000_04_10_0",  "dev_type":"type-USB", "status":"available", "instance_uuid":"", "extra_info":"", "created_at": null, "deleted": false,  "deleted_at": null,}}

2. We should extend hypervisor API to add usb device stats information so that nova can support querying usb device stats of a hypervisor.
List and show usb device stats on hypervisors request: 

GET v2/​{tenant_id}​/os-hypervisors/​{hypervisor_hostname}

The response JSON contains the variable "usb_device_stats": 

{"hypervisor": {"usb_stats": [{"id": 1, "vendor_id": "8086", "product_id": "1520", "address":"0000:01:00.0", "status":"available", "instance_uuid":""}]}}

3.We should extend server API to add usb assignment information so that nova can support querying usb device stats of a server.
List and show usb devices of a virtual machine: 

GET v2/​{tenant_id}​/servers/​{server_id}

The response JSON will contain the variable "os-usb-device:usb_devices": 

{ "server": {"os-usb-device:usb_devices": [{"id": 1}], "tenant_id": "openstack", "user_id": "admin"}}

4.We should extend server API to add usb device hot-plug and cold-plug function. Attach usb device: 

Post  v2/​{tenant_id}​/servers/​{server_id}/action
  {
      "attach_usb_devices": {"usb_devices_requests":[{"id": 1}]}
  }

Detach usb device: 

Post  v2/​{tenant_id}​/servers/​{server_id}/action
  {
      "detach_usb_devices": {"usb_devices_requests":[{"id": 1}]}
  }
Retrieved from "https://wiki.openstack.org/w/index.php?title=Nova/proposal_about_usb_passthrough&oldid=88071"