Difference between revisions of "Cyborg/TestReport/InspurNVMeSSD"

Latest revision as of 09:20, 15 February 2021

Inspur NVMe SSD Cyborg Driver Test Report

This test report shows the current Inspur NVMe SSD driver is well tested in local env where Inspur NVMe SSD disk located. Instead of a 3rd party CI, this report can be the proof.

report part

Enable Inspur NVMe SSD driver in cyborg agent section

   [root@node11 cyborg]$ cat /etc/kolla/cyborg/cyborg.conf  | grep inspur
   enabled_drivers = inspur_nvme_ssd_driver

Report logs

   Aug 20 17:06:48 ubuntu cyborg-conductor[9441]: 2021-01-20 17:06:48.909 9441 INFO cyborg.conductor.manager [req-e9c7b9ec-eff4-
   4480-8178-134037b60524 - - - - -] Start differing devices.
   Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2021-01-20 17:06:49.501 9441 INFO cyborg.common.placement_client [req-e9c7b9ec-
   eff4-4480-8178-134037b60524 - - - - -] Creating resource provider
   Inspur Electronic Information Industry Co., Ltd. Device_0000:60:00.0
   Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2021-01-20 17:06:49.553 9441 INFO cyborg.common.placement_client [req-e9c7b9ec-
   eff4-4480-8178-134037b60524 - - - - -] [req-c5566486-40b5-4a20-b4f4-b0b96b7812c9] Created resource provider record via placement
   API for resource provider with UUID bcedfee6-c1f0-349c-acfa-7d39033b7b8c and name
   Inspur Electronic Information Industry Co., Ltd. Device_0000:60:00.0.
   Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2021-01-20 17:06:49.715 9441 INFO cyborg.common.placement_client [req-e9c7b9ec-
   eff4-4480-8178-134037b60524 - - - - -] Created trait CUSTOM_SSD_INSPUR
   Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2021-01-20 17:06:49.788 9441 INFO cyborg.common.placement_client [req-e9c7b9ec-
   eff4-4480-8178- 134037b60524 - - - - -] Created trait CUSTOM_SSD_PRODUCT_ID_1003
   Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2021-01-20 17:06:49.950 9441 INFO cyborg.conductor.manager [req-e9c7b9ec-eff4-
   4480-8178-134037b60524 -  - - - -] Start differing deploybles.
   Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2021-01-20 17:06:49.955 9441 INFO cyborg.conductor.manager [req-e9c7b9ec-eff4-
   4480-8178-134037b60524 - - - - -] Start differing attributes.
   Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2021-01-20 17:06:49.972 9441 INFO cyborg.conductor.manager [req-e9c7b9ec-eff4-
   4480-8178-134037b60524 - - - - -] Start differing attach_handles.
   Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2021-01-20 17:06:49.973 9441 INFO cyborg.conductor.manager [req-e9c7b9ec-eff4-
   4480-8178-134037b60524 - - - - -] new info list ['{"bus": "60", "device": "00", "domain": "0000", "function": "0"}']
   Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2021-01-20 17:06:49.973 9441 INFO cyborg.conductor.manager [req-e9c7b9ec-eff4-
   4480-8178-134037b60524 - - - - -] old info list ['{"bus": "60", "device": "00", "domain": "0000", "function": "0"}']    
   ...

check cyborg record

   root@ubuntu:~# openstack accelerator device list
   +--------------------------------------+------+--------+----------+-----------------------------------------------------------------+
   | uuid                                 | type | vendor | hostname | std_board_info                                                  |
   +--------------------------------------+------+--------+----------+-----------------------------------------------------------------+
   | 462a37e7-3654-44ab-b5bf-c462cd10f8a4 | SSD| 1bd4   | ubuntu   | {"product_id": "1003", "controller": "Processing accelerators"} |
   +--------------------------------------+------+--------+----------+-----------------------------------------------------------------

   root@ubuntu:~# openstack accelerator deployable list
   +--------------------------------------+----------------------------------------------------------------------+-----------+
   | uuid                                 | name                                                                 | device_id |
   +--------------------------------------+----------------------------------------------------------------------+-----------+
   | 3355050d-6a21-4129-bab8-63280598684e | ubuntu_0000:60:00.0 |         2 |
   +--------------------------------------+----------------------------------------------------------------------+-----------+

check placement record

   [root@ubuntu cyborg]$ openstack resource provider list
   +--------------------------------------+----------------------------+------------+
   | uuid                                 | name                       | generation |
   +--------------------------------------+----------------------------+------------+
   | bcedfee6-c1f0-349c-acfa-7d39033b7b8c  | ubuntu_0000:60:00.0                 |        2 |
   | 1b3419ee-f45c-4b4e-ba91-7715b9070da8 |  ubuntu |          60 |

   [root@ubuntu cyborg]$ openstack resource provider inventory list bcedfee6-c1f0-349c-acfa-7d39033b7b8c
   +----------------+------------------+----------+----------+----------+-----------+-------+
   | resource_class | allocation_ratio | min_unit | max_unit | reserved | step_size | total |
   +----------------+------------------+----------+----------+----------+-----------+-------+
   | CUSTOM_SSD         |              1.0 |        1  |        1  |        0  |         1   |     1  |
   +----------------+------------------+----------+----------+----------+-----------+-------+

Boot a VM with Inspur NVMe SSD attached

create device profile and flavor

   root@ubuntu:/opt/stack/cyborg# openstack accelerator device profile create inspur-nvme-ssd '[{"resources:CUSTOM_SSD":1, "trait:CUSTOM_SSD_PRODUCT_ID_1003": "required",  "trait:CUSTOM_SSD_INSPUR": "required"}]'

   +------------+--------------------------------------------------------------------------------------------------------------------+
   | Field       | Value                                                                                                                                  |
   +---------------+-----------------------------------------------------------------------------------------------------------------+
   | created_at  | 2021-01-21 10:47:06+00:00                                                                                           |
   | updated_at | None                                                                                                                               |
   | uuid            | 63cac6f6-1122-46d1-825e-b288886ebdf5                                                                     |
   | name          | inspur-nvme-ssd                                                                                                                     |
   | groups        | [{'resources:CUSTOM_SSD': '1', 'trait:CUSTOM_SSD_PRODUCT_ID_1003': 'required',
                            'trait:CUSTOM_SSD_INSPUR': 'required'}]                                                               |
   +---------------+------------------------------------------------------------------------------------------------------------- ---+

   root@ubuntu:/opt/stack/cyborg# openstack accelerator device profile list
   +--------------------------------------+-------------+--------------------------------------------------------------------------------------------------------------------+
   | uuid                                       | name        | groups                                                                                                                               |
   +--------------------------------------+-------------+--------------------------------------------------------------------------------------------------------------------+
   | 63cac6f6-1122-46d1-825e-b288886ebdf5 | inspur-nvme-ssd | [{'resources:CUSTOM_SSD': '1', 'trait:CUSTOM_SSD_PRODUCT_ID_1003': 'required',
                                                                                                 'trait:CUSTOM_SSD_INSPUR': 'required'}] |

   root@ubuntu:/opt/stack/cyborg# nova flavor-key 3 set accel:device_profile='inspur-nvme-ssd' 
   root@ubuntu:/opt/stack/cyborg# nova flavor-show 3
   +----------------------------+-------------------------------------------------------------------+
   | Property                   | Value                                                             |
   +----------------------------+-------------------------------------------------------------------+
   | OS-FLV-DISABLED:disabled   | False                                                             |
   | OS-FLV-EXT-DATA:ephemeral  | 0                                                                 |
   | description                | -                                                                 |
   | disk                       | 40                                                                |
   | extra_specs                | {"accel:device_profile": "inspur-nvme-ssd", "hw_rng:allowed": "True"} |
   | id                         | 3                                                                 |
   | name                       | m1.medium                                                         |
   | os-flavor-access:is_public | True                                                              |
   | ram                        | 4096                                                              |
   | rxtx_factor                | 1.0                                                               |
   | swap                       | 0                                                                 |
   | vcpus                      | 2                                                                 |
   +----------------------------+-------------------------------------------------------------------+

boot a VM with Inspur NVMe SSD attached

In this step, I gave some output of cyborg-api log to show how Nova interact with Cyborg.

Nova get DP details

   Aug 21 17:33:48 ubuntu devstack@cyborg-api.service[9353]: 2021-01-21 17:33:48.594 9365 INFO
   cyborg.api.controllers.v2.device_profiles
   [req-e89db88c-80f1-420f-b4b3-a38998f41f98 613b4ba7e86a4058b458e4e9496a0507
   fef0949e2ce6415ab5cd001ba254e563 - default default] [device_profiles] get_all returned:
   {'device_profiles': [{'name': 'inspur-nvme-ssd', 'description': None, 'uuid': '63cac6f6-1122-46d1-825e-
     b288886ebdf5', 'groups': [{'resources:CUSTOM_SSD': '1', 'trait:CUSTOM_SSD_PRODUCT_ID_1003': 'required',
   'trait:CUSTOM_SSD_INSPUR': 'required'}], 'created_at': '2021-01-21 09:13:29+00:00',
   'updated_at': 'None', 'links': [{'href': 'http://10.180.249.199/accelerator/v2/device
   _profiles/63cac6f6-1122-46d1-825e-b288886ebdf5', 'rel': 'self'}]}]}

ARQ creation

   Aug 18 02:03:10 vca-server devstack@cyborg-api.service[2243671]: 2021-01-18 02:03:10.052 2243674 INFO
   cyborg.api.controllers.v2.arqs [req-73c06ce1-b9df-4d94-8760-e4ab5f595328 1f86481df3024d8bb74e213e23dcc2e4
   7cd6c62783934a148605e271b4f6f922 - default default] [arq] post req = ({'device_profile_name': 'inspur-nvme-ssd'})

ARQ binding

   Aug 21 17:33:49 ubuntu devstack@cyborg-api.service[9353]: 2021-01-21 17:33:49.980 9360 INFO
   cyborg.api.controllers.v2.arqs [req-f92f6792-4730-4cb7-9932-9ca4f393500a 613b4ba7e86a4058b458e4e9496a0507
   fef0949e2ce6415ab5cd001ba254e563 - default default] [arqs] patch. list=({'549f40f9-ff45-4b55-8d7d-89f0f6822203': [
   {'path': '/hostname', 'op': 'add', 'value': 'ubuntu'}, {'path': '/device_rp_uuid', 'op': 'add',
   'value': 'bcedfee6-c1f0-349c-acfa-7d39033b7b8c'}, {'path': '/instance_uuid', 'op': 'add',
   'value': '45716b59-7eaf-4663-bd7e- f83416f7ab48'}]})

Get bound ARQ

   Aug 21 17:33:51 ubuntu devstack@cyborg-api.service[9353]: 2021-01-21 17:33:51.025 9366 INFO cyborg.api.controllers.v2.arqs
   [req-d094f906-3969-49d5-88e2-8459a1b72158 613b4ba7e86a4058b458e4e9496a0507 fef0949e2ce6415ab5cd001ba254e563
   - default default] [arqs:get_all] Returned:<cyborg.api.controllers.v2.arqs.ARQCollection object at 0x7fc2deb187b8>

Check the VM

log in to the VM , run`lspci | grep Inspur`

It will show a pci device which means the Inspur NVMe SSD has been passthrough to the VM sucessfully.

   [root@host-10-180-249-159 ~]# lspci|grep Inspur
   00:05.0 Processing accelerators: Inspur Electronic Information Industry Co., Ltd. Device 1003(rev 04)

delete a VM with Inspur NVMe SSD detached

In this step, I gave some output of cyborg and nova log to show how Nova interact with Cyborg.

Nova api delete vm request

   Aug 22 09:54:48 ubuntu devstack@n-api.service[1808]: [pid: 1916|app: 0|req: 25/296] 10.180.249.199 ()
   {70 vars in 1494 bytes} [Sat Aug 22 09:54:47 2020]
   DELETE /compute/v2.1/servers/45716b59-7eaf-4663-bd7e-f83416f7ab48 => generated 0 bytes
   in 404 msecs (HTTP/1.1 204) 9 headers in 363 bytes (2 switches on core 0)

Nova compute start delete vm

   Aug 22 09:54:48 ubuntu nova-compute[20353]: #033[00;32mDEBUG nova.compute.manager [#033[01;36m None
   req-a307b6cf-101a-4e9c-923f-a6fd4d8f6e6f #033[00;36madmin admin#033[00;32m] #033[01;35m
   [instance: 45716b59-7eaf-4663-bd7e-f83416f7ab48] #033[00;32mStart destroying the instance on the hypervisor.
   #033[00m #033[00;33mTemplate:(pid=20353) shutdown instance /opt/stack/nova/nova/compute/manager.py:2837#033[00m#033[00m

Nova compute start delete arq

   Aug 22 09:54:50 ubuntu nova-compute[20353]: #033[00;32mDEBUG nova.compute.utils [#033[01;36mNone
   req-a307b6cf-101a-4e9c-923f-a6fd4d8f6e6f 
   #033[00;36madmin admin#033[00;32m] #033[01;35m#033[00;32mCalling Cyborg to delete ARQs for instance
   45716b59-7eaf-4663-bd7e-f83416f7ab48#033[00m #033[00;33m{{(pid=20353) delete_arqs_if_needed
   /opt/stack/nova/nova/compute/utils.py:1554}}

Cyborg api start delete arq

   Aug 22 09:54:50 ubuntu devstack@cyborg-api.service[9353]: 2021-01-22 09:54:50.466 9364 INFO
   cyborg.api.controllers.v2.arqs [req-3b7aca17-b96c-4d08-9c99-7df8093fc927
   613b4ba7e86a4058b458e4e9496a0507 fef0949e2ce6415ab5cd001ba254e563 -
   default default] [arqs] delete. instance=(45716b59-7eaf-4663-bd7e-f83416f7ab48)

Cyborg api delete arq

   Aug 22 09:54:50 ubuntu devstack@cyborg-api.service[9353]: 2021-01-22 09:54:50.487 9364 INFO
   cyborg.objects.ext_arq [req-3b7aca17-b96c-4d08-9c99-7df8093fc927
   613b4ba7e86a4058b458e4e9496a0507 fef0949e2ce6415ab5cd001ba254e563 - default default]
   Deleting obj_extarq uuid 549f40f9-ff45-4b55-8d7d-89f0f6822203
   for instance 45716b59-7eaf-4663-bd7e-f83416f7ab48#033[00m

Arq delete successfully

   Aug 22 09:54:50 2020] DELETE /accelerator/v2/accelerator_requests?instance=
   45716b59-7eaf-4663-bd7e-f83416f7ab48 => generated 0 bytes
   in 80 msecs (HTTP/1.1 204) 7 headers in 260 bytes (1 switches on core 0)