Cyborg/TestReport/InspurNVMeSSD

Inspur NVMe SSD Cyborg Driver Test Report
This test report shows the current Inspur NVMe SSD driver is well tested in local env where Inspur NVMe SSD disk located. Instead of a 3rd party CI, this report can be the proof.

Enable Inspur NVMe SSD driver in cyborg agent section
[root@node11 cyborg]$ cat /etc/kolla/cyborg/cyborg.conf | grep inspur enabled_drivers = inspur_nvme_ssd_driver

Report logs
Aug 20 17:06:48 ubuntu cyborg-conductor[9441]: 2021-01-20 17:06:48.909 9441 INFO cyborg.conductor.manager [req-e9c7b9ec-eff4- 4480-8178-134037b60524 - - - - -] Start differing devices. Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2021-01-20 17:06:49.501 9441 INFO cyborg.common.placement_client [req-e9c7b9ec- eff4-4480-8178-134037b60524 - - - - -] Creating resource provider Inspur Electronic Information Industry Co., Ltd. Device_0000:60:00.0 Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2021-01-20 17:06:49.553 9441 INFO cyborg.common.placement_client [req-e9c7b9ec- eff4-4480-8178-134037b60524 - - - - -] [req-c5566486-40b5-4a20-b4f4-b0b96b7812c9] Created resource provider record via placement API for resource provider with UUID bcedfee6-c1f0-349c-acfa-7d39033b7b8c and name Inspur Electronic Information Industry Co., Ltd. Device_0000:60:00.0. Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2021-01-20 17:06:49.715 9441 INFO cyborg.common.placement_client [req-e9c7b9ec- eff4-4480-8178-134037b60524 - - - - -] Created trait CUSTOM_SSD_INSPUR Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2021-01-20 17:06:49.788 9441 INFO cyborg.common.placement_client [req-e9c7b9ec- eff4-4480-8178- 134037b60524 - - - - -] Created trait CUSTOM_SSD_PRODUCT_ID_1003 Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2021-01-20 17:06:49.950 9441 INFO cyborg.conductor.manager [req-e9c7b9ec-eff4- 4480-8178-134037b60524 - - - - -] Start differing deploybles. Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2021-01-20 17:06:49.955 9441 INFO cyborg.conductor.manager [req-e9c7b9ec-eff4- 4480-8178-134037b60524 - - - - -] Start differing attributes. Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2021-01-20 17:06:49.972 9441 INFO cyborg.conductor.manager [req-e9c7b9ec-eff4- 4480-8178-134037b60524 - - - - -] Start differing attach_handles. Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2021-01-20 17:06:49.973 9441 INFO cyborg.conductor.manager [req-e9c7b9ec-eff4- 4480-8178-134037b60524 - - - - -] new info list ['{"bus": "60", "device": "00", "domain": "0000", "function": "0"}'] Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2021-01-20 17:06:49.973 9441 INFO cyborg.conductor.manager [req-e9c7b9ec-eff4- 4480-8178-134037b60524 - - - - -] old info list ['{"bus": "60", "device": "00", "domain": "0000", "function": "0"}'] ...

check cyborg record
root@ubuntu:~# openstack accelerator device list +--+--++--+-+   | uuid                                 | type | vendor | hostname | std_board_info                                                  | +--+--++--+-+   | 462a37e7-3654-44ab-b5bf-c462cd10f8a4 | SSD| 1bd4   | ubuntu   | {"product_id": "1003", "controller": "Processing accelerators"} | +--+--++--+-

root@ubuntu:~# openstack accelerator deployable list +--+--+---+   | uuid                                 | name                                                                 | device_id | +--+--+---+   | 3355050d-6a21-4129-bab8-63280598684e | ubuntu_0000:60:00.0 |         2 | +--+--+---+

check placement record
[root@ubuntu cyborg]$ openstack resource provider list +--+++   | uuid                                 | name                       | generation | +--+++   | bcedfee6-c1f0-349c-acfa-7d39033b7b8c  | ubuntu_0000:60:00.0                 |        2 | | 1b3419ee-f45c-4b4e-ba91-7715b9070da8 | ubuntu |          60 |

[root@ubuntu cyborg]$ openstack resource provider inventory list bcedfee6-c1f0-349c-acfa-7d39033b7b8c ++--+--+--+--+---+---+   | resource_class | allocation_ratio | min_unit | max_unit | reserved | step_size | total | ++--+--+--+--+---+---+   | CUSTOM_SSD         |              1.0 |        1  |        1  |        0  |         1   |     1  | ++--+--+--+--+---+---+

create device profile and flavor
root@ubuntu:/opt/stack/cyborg# openstack accelerator device profile create inspur-nvme-ssd '[{"resources:CUSTOM_SSD":1, "trait:CUSTOM_SSD_PRODUCT_ID_1003": "required", "trait:CUSTOM_SSD_INSPUR": "required"}]'

+++   | Field       | Value                                                                                                                                  | +---+-+   | created_at  | 2021-01-21 10:47:06+00:00                                                                                           | | updated_at | None                                                                                                                              | | uuid           | 63cac6f6-1122-46d1-825e-b288886ebdf5                                                                     | | name         | inspur-nvme-ssd                                                                                                                     | | groups       | [{'resources:CUSTOM_SSD': '1', 'trait:CUSTOM_SSD_PRODUCT_ID_1003': 'required', 'trait:CUSTOM_SSD_INSPUR': 'required'}]                                                              | +---+- ---+

root@ubuntu:/opt/stack/cyborg# openstack accelerator device profile list +--+-++   | uuid                                       | name        | groups                                                                                                                               | +--+-++   | 63cac6f6-1122-46d1-825e-b288886ebdf5 | inspur-nvme-ssd | [{'resources:CUSTOM_SSD': '1', 'trait:CUSTOM_SSD_PRODUCT_ID_1003': 'required', 'trait:CUSTOM_SSD_INSPUR': 'required'}] |

root@ubuntu:/opt/stack/cyborg# nova flavor-key 3 set accel:device_profile='inspur-nvme-ssd' root@ubuntu:/opt/stack/cyborg# nova flavor-show 3 ++---+   | Property                   | Value                                                             | ++---+   | OS-FLV-DISABLED:disabled   | False                                                             | | OS-FLV-EXT-DATA:ephemeral | 0                                                                 | | description               | -                                                                 | | disk                      | 40                                                                | | extra_specs               | {"accel:device_profile": "inspur-nvme-ssd", "hw_rng:allowed": "True"} | | id                        | 3                                                                 | | name                      | m1.medium                                                         | | os-flavor-access:is_public | True                                                             | | ram                       | 4096                                                              | | rxtx_factor               | 1.0                                                               | | swap                      | 0                                                                 | | vcpus                     | 2                                                                 | ++---+

boot a VM with Inspur NVMe SSD attached
In this step, I gave some output of cyborg-api log to show how Nova interact with Cyborg.

Nova get DP details
Aug 21 17:33:48 ubuntu devstack@cyborg-api.service[9353]: 2021-01-21 17:33:48.594 9365 INFO cyborg.api.controllers.v2.device_profiles [req-e89db88c-80f1-420f-b4b3-a38998f41f98 613b4ba7e86a4058b458e4e9496a0507 fef0949e2ce6415ab5cd001ba254e563 - default default] [device_profiles] get_all returned: {'device_profiles': [{'name': 'inspur-nvme-ssd', 'description': None, 'uuid': '63cac6f6-1122-46d1-825e- b288886ebdf5', 'groups': [{'resources:CUSTOM_SSD': '1', 'trait:CUSTOM_SSD_PRODUCT_ID_1003': 'required', 'trait:CUSTOM_SSD_INSPUR': 'required'}], 'created_at': '2021-01-21 09:13:29+00:00', 'updated_at': 'None', 'links': [{'href': 'http://10.180.249.199/accelerator/v2/device _profiles/63cac6f6-1122-46d1-825e-b288886ebdf5', 'rel': 'self'}]}]}

ARQ creation
Aug 18 02:03:10 vca-server devstack@cyborg-api.service[2243671]: 2021-01-18 02:03:10.052 2243674 INFO cyborg.api.controllers.v2.arqs [req-73c06ce1-b9df-4d94-8760-e4ab5f595328 1f86481df3024d8bb74e213e23dcc2e4 7cd6c62783934a148605e271b4f6f922 - default default] [arq] post req = ({'device_profile_name': 'inspur-nvme-ssd'})

ARQ binding
Aug 21 17:33:49 ubuntu devstack@cyborg-api.service[9353]: 2021-01-21 17:33:49.980 9360 INFO cyborg.api.controllers.v2.arqs [req-f92f6792-4730-4cb7-9932-9ca4f393500a 613b4ba7e86a4058b458e4e9496a0507 fef0949e2ce6415ab5cd001ba254e563 - default default] [arqs] patch. list=({'549f40f9-ff45-4b55-8d7d-89f0f6822203': [   {'path': '/hostname', 'op': 'add', 'value': 'ubuntu'}, {'path': '/device_rp_uuid', 'op': 'add',    'value': 'bcedfee6-c1f0-349c-acfa-7d39033b7b8c'}, {'path': '/instance_uuid', 'op': 'add',    'value': '45716b59-7eaf-4663-bd7e- f83416f7ab48'}]})

Get bound ARQ
Aug 21 17:33:51 ubuntu devstack@cyborg-api.service[9353]: 2021-01-21 17:33:51.025 9366 INFO cyborg.api.controllers.v2.arqs [req-d094f906-3969-49d5-88e2-8459a1b72158 613b4ba7e86a4058b458e4e9496a0507 fef0949e2ce6415ab5cd001ba254e563 - default default] [arqs:get_all] Returned:

Check the VM
log in to the VM, run`lspci | grep Inspur` It will show a pci device which means the Inspur NVMe SSD has been passthrough to the VM sucessfully.

[root@host-10-180-249-159 ~]# lspci|grep Inspur 00:05.0 Processing accelerators: Inspur Electronic Information Industry Co., Ltd. Device 1003(rev 04)

delete a VM with Inspur NVMe SSD detached
In this step, I gave some output of cyborg and nova log to show how Nova interact with Cyborg.

Nova api delete vm request
Aug 22 09:54:48 ubuntu devstack@n-api.service[1808]: [pid: 1916|app: 0|req: 25/296] 10.180.249.199 {70 vars in 1494 bytes} [Sat Aug 22 09:54:47 2020] DELETE /compute/v2.1/servers/45716b59-7eaf-4663-bd7e-f83416f7ab48 => generated 0 bytes in 404 msecs (HTTP/1.1 204) 9 headers in 363 bytes (2 switches on core 0)

Nova compute start delete vm
Aug 22 09:54:48 ubuntu nova-compute[20353]: #033[00;32mDEBUG nova.compute.manager [#033[01;36m None req-a307b6cf-101a-4e9c-923f-a6fd4d8f6e6f #033[00;36madmin admin#033[00;32m] #033[01;35m [instance: 45716b59-7eaf-4663-bd7e-f83416f7ab48] #033[00;32mStart destroying the instance on the hypervisor. #033[00m #033[00;33m#033[00m#033[00m

Nova compute start delete arq
Aug 22 09:54:50 ubuntu nova-compute[20353]: #033[00;32mDEBUG nova.compute.utils [#033[01;36mNone req-a307b6cf-101a-4e9c-923f-a6fd4d8f6e6f #033[00;36madmin admin#033[00;32m] #033[01;35m#033[00;32mCalling Cyborg to delete ARQs for instance 45716b59-7eaf-4663-bd7e-f83416f7ab48#033[00m #033[00;33m

Cyborg api start delete arq
Aug 22 09:54:50 ubuntu devstack@cyborg-api.service[9353]: 2021-01-22 09:54:50.466 9364 INFO cyborg.api.controllers.v2.arqs [req-3b7aca17-b96c-4d08-9c99-7df8093fc927 613b4ba7e86a4058b458e4e9496a0507 fef0949e2ce6415ab5cd001ba254e563 - default default] [arqs] delete. instance=(45716b59-7eaf-4663-bd7e-f83416f7ab48)

Cyborg api delete arq
Aug 22 09:54:50 ubuntu devstack@cyborg-api.service[9353]: 2021-01-22 09:54:50.487 9364 INFO cyborg.objects.ext_arq [req-3b7aca17-b96c-4d08-9c99-7df8093fc927 613b4ba7e86a4058b458e4e9496a0507 fef0949e2ce6415ab5cd001ba254e563 - default default] Deleting obj_extarq uuid 549f40f9-ff45-4b55-8d7d-89f0f6822203 for instance 45716b59-7eaf-4663-bd7e-f83416f7ab48#033[00m

Arq delete successfully
Aug 22 09:54:50 2020] DELETE /accelerator/v2/accelerator_requests?instance= 45716b59-7eaf-4663-bd7e-f83416f7ab48 => generated 0 bytes in 80 msecs (HTTP/1.1 204) 7 headers in 260 bytes (1 switches on core 0)