Cyborg/TestReport/InspurFPGA

Inspur FPGA Cyborg Driver Test Report
This test report shows the current Inspur FPGA driver is well tested in local env where Inspur FPGA card located. Instead of a 3rd party CI, this report can be the proof.

Enable Inspur FPGA driver in cyborg agent section
[root@node11 cyborg]$ cat /etc/kolla/cyborg/cyborg.conf | grep inspur enabled_drivers = inspur_fpga_driver

Report logs
Aug 20 17:06:48 ubuntu cyborg-conductor[9441]: 2020-08-20 17:06:48.909 9441 INFO cyborg.conductor.manager [req-e9c7b9ec-eff4- 4480-8178-134037b60524 - - - - -] Start differing devices. Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2020-08-20 17:06:49.501 9441 INFO cyborg.common.placement_client [req-e9c7b9ec- eff4-4480-8178-134037b60524 - - - - -] Creating resource provider Inspur Electronic Information Industry Co., Ltd. Device_0000:60:00.0 Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2020-08-20 17:06:49.553 9441 INFO cyborg.common.placement_client [req-e9c7b9ec- eff4-4480-8178-134037b60524 - - - - -] [req-c5566486-40b5-4a20-b4f4-b0b96b7812c9] Created resource provider record via placement API for resource provider with UUID bcedfee6-c1f0-349c-acfa-7d39033b7b8c and name Inspur Electronic Information Industry Co., Ltd. Device_0000:60:00.0. Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2020-08-20 17:06:49.715 9441 INFO cyborg.common.placement_client [req-e9c7b9ec- eff4-4480-8178-134037b60524 - - - - -] Created trait CUSTOM_FPGA_INSPUR Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2020-08-20 17:06:49.788 9441 INFO cyborg.common.placement_client [req-e9c7b9ec- eff4-4480-8178- 134037b60524 - - - - -] Created trait CUSTOM_FPGA_PRODUCT_ID_A115 Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2020-08-20 17:06:49.950 9441 INFO cyborg.conductor.manager [req-e9c7b9ec-eff4- 4480-8178-134037b60524 - - - - -] Start differing deploybles. Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2020-08-20 17:06:49.955 9441 INFO cyborg.conductor.manager [req-e9c7b9ec-eff4- 4480-8178-134037b60524 - - - - -] Start differing attributes. Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2020-08-20 17:06:49.972 9441 INFO cyborg.conductor.manager [req-e9c7b9ec-eff4- 4480-8178-134037b60524 - - - - -] Start differing attach_handles. Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2020-08-20 17:06:49.973 9441 INFO cyborg.conductor.manager [req-e9c7b9ec-eff4- 4480-8178-134037b60524 - - - - -] new info list ['{"bus": "60", "device": "00", "domain": "0000", "function": "0"}'] Aug 20 17:06:49 ubuntu cyborg-conductor[9441]: 2020-08-20 17:06:49.973 9441 INFO cyborg.conductor.manager [req-e9c7b9ec-eff4- 4480-8178-134037b60524 - - - - -] old info list ['{"bus": "60", "device": "00", "domain": "0000", "function": "0"}'] ...

check cyborg record
root@ubuntu:~# openstack accelerator device list +--+--++--+-+   | uuid                                 | type | vendor | hostname | std_board_info                                                  | +--+--++--+-+   | 462a37e7-3654-44ab-b5bf-c462cd10f8a4 | FPGA | 1bd4   | ubuntu   | {"product_id": "a115", "controller": "Processing accelerators"} | +--+--++--+-

root@ubuntu:~# openstack accelerator deployable list +--+--+---+   | uuid                                 | name                                                                 | device_id | +--+--+---+   | 3355050d-6a21-4129-bab8-63280598684e | Inspur Electronic Information Industry Co., Ltd. Device_0000:60:00.0 |         2 | +--+--+---+

check placement record
[root@ubuntu cyborg]$ openstack resource provider list +--+++   | uuid                                 | name                       | generation | +--+++   | bcedfee6-c1f0-349c-acfa-7d39033b7b8c  | Inspur Electronic Information Industry Co., Ltd. Device_0000:60:00.0                 |        2 | | 1b3419ee-f45c-4b4e-ba91-7715b9070da8 | ubuntu |          60 |

[root@ubuntu cyborg]$ openstack resource provider inventory list bcedfee6-c1f0-349c-acfa-7d39033b7b8c ++--+--+--+--+---+---+   | resource_class | allocation_ratio | min_unit | max_unit | reserved | step_size | total | ++--+--+--+--+---+---+   | FPGA         |              1.0 |        1  |        1  |        0  |         1   |     1  | ++--+--+--+--+---+---+

create device profile and flavor
root@ubuntu:/opt/stack/cyborg# openstack accelerator device profile create inspur-fpga '[{"resources:FPGA":1, "trait:CUSTOM_FPGA_PRODUCT_ID_A115": "required", "trait:CUSTOM_FPGA_INSPUR": "required"}]'

+++   | Field       | Value                                                                                                                                  | +---+-+   | created_at  | 2020-08-21 10:47:06+00:00                                                                                           | | updated_at | None                                                                                                                              | | uuid           | 63cac6f6-1122-46d1-825e-b288886ebdf5                                                                     | | name         | inspur-fpga                                                                                                                     | | groups       | [{'resources:FPGA': '1', 'trait:CUSTOM_FPGA_PRODUCT_ID_A115': 'required', 'trait:CUSTOM_FPGA_INSPUR': 'required'}]                                                              | +---+- ---+

root@ubuntu:/opt/stack/cyborg# openstack accelerator device profile list +--+-++   | uuid                                       | name        | groups                                                                                                                               | +--+-++   | 63cac6f6-1122-46d1-825e-b288886ebdf5 | inspur-fpga | [{'resources:FPGA': '1', 'trait:CUSTOM_FPGA_PRODUCT_ID_A115': 'required', 'trait:CUSTOM_FPGA_INSPUR': 'required'}] |

root@ubuntu:/opt/stack/cyborg# nova flavor-key 3 set accel:device_profile='inspur-fpga' root@ubuntu:/opt/stack/cyborg# nova flavor-show 3 ++---+   | Property                   | Value                                                             | ++---+   | OS-FLV-DISABLED:disabled   | False                                                             | | OS-FLV-EXT-DATA:ephemeral | 0                                                                 | | description               | -                                                                 | | disk                      | 40                                                                | | extra_specs               | {"accel:device_profile": "inspur-fpga", "hw_rng:allowed": "True"} | | id                        | 3                                                                 | | name                      | m1.medium                                                         | | os-flavor-access:is_public | True                                                             | | ram                       | 4096                                                              | | rxtx_factor               | 1.0                                                               | | swap                      | 0                                                                 | | vcpus                     | 2                                                                 | ++---+

boot a VM with Inspur FPGA attached
In this step, I gave some output of cyborg-api log to show how Nova interact with Cyborg.

Nova get DP details
Aug 21 17:33:48 ubuntu devstack@cyborg-api.service[9353]: 2020-08-21 17:33:48.594 9365 INFO cyborg.api.controllers.v2.device_profiles [req-e89db88c-80f1-420f-b4b3-a38998f41f98 613b4ba7e86a4058b458e4e9496a0507 fef0949e2ce6415ab5cd001ba254e563 - default default] [device_profiles] get_all returned: {'device_profiles': [{'name': 'inspur-fpga', 'description': None, 'uuid': '63cac6f6-1122-46d1-825e- b288886ebdf5', 'groups': [{'resources:FPGA': '1', 'trait:CUSTOM_FPGA_PRODUCT_ID_A115': 'required', 'trait:CUSTOM_FPGA_INSPUR': 'required'}], 'created_at': '2020-08-21 09:13:29+00:00', 'updated_at': 'None', 'links': [{'href': 'http://10.180.249.199/accelerator/v2/device _profiles/63cac6f6-1122-46d1-825e-b288886ebdf5', 'rel': 'self'}]}]}

ARQ creation
Aug 18 02:03:10 vca-server devstack@cyborg-api.service[2243671]: 2020-08-18 02:03:10.052 2243674 INFO cyborg.api.controllers.v2.arqs [req-73c06ce1-b9df-4d94-8760-e4ab5f595328 1f86481df3024d8bb74e213e23dcc2e4 7cd6c62783934a148605e271b4f6f922 - default default] [arq] post req = ({'device_profile_name': 'inspur-fpga'})

ARQ binding
Aug 21 17:33:49 ubuntu devstack@cyborg-api.service[9353]: 2020-08-21 17:33:49.980 9360 INFO cyborg.api.controllers.v2.arqs [req-f92f6792-4730-4cb7-9932-9ca4f393500a 613b4ba7e86a4058b458e4e9496a0507 fef0949e2ce6415ab5cd001ba254e563 - default default] [arqs] patch. list=({'549f40f9-ff45-4b55-8d7d-89f0f6822203': [   {'path': '/hostname', 'op': 'add', 'value': 'ubuntu'}, {'path': '/device_rp_uuid', 'op': 'add',    'value': 'bcedfee6-c1f0-349c-acfa-7d39033b7b8c'}, {'path': '/instance_uuid', 'op': 'add',    'value': '45716b59-7eaf-4663-bd7e- f83416f7ab48'}]})

Get bound ARQ
Aug 21 17:33:51 ubuntu devstack@cyborg-api.service[9353]: 2020-08-21 17:33:51.025 9366 INFO cyborg.api.controllers.v2.arqs [req-d094f906-3969-49d5-88e2-8459a1b72158 613b4ba7e86a4058b458e4e9496a0507 fef0949e2ce6415ab5cd001ba254e563 - default default] [arqs:get_all] Returned:

Check the VM
log in to the VM, run`lspci | grep Inspur` It will show a pci device which means the Inspur FPGA has been passthrough to the VM sucessfully.

[root@host-10-180-249-159 ~]# lspci|grep Inspur 00:05.0 Processing accelerators: Inspur Electronic Information Industry Co., Ltd. Device a115 (rev 04)

delete a VM with Inspur FPGA detached
In this step, I gave some output of cyborg and nova log to show how Nova interact with Cyborg.

Nova api delete vm request
Aug 22 09:54:48 ubuntu devstack@n-api.service[1808]: [pid: 1916|app: 0|req: 25/296] 10.180.249.199 {70 vars in 1494 bytes} [Sat Aug 22 09:54:47 2020] DELETE /compute/v2.1/servers/45716b59-7eaf-4663-bd7e-f83416f7ab48 => generated 0 bytes in 404 msecs (HTTP/1.1 204) 9 headers in 363 bytes (2 switches on core 0)

Nova compute start delete vm
Aug 22 09:54:48 ubuntu nova-compute[20353]: #033[00;32mDEBUG nova.compute.manager [#033[01;36m None req-a307b6cf-101a-4e9c-923f-a6fd4d8f6e6f #033[00;36madmin admin#033[00;32m] #033[01;35m [instance: 45716b59-7eaf-4663-bd7e-f83416f7ab48] #033[00;32mStart destroying the instance on the hypervisor. #033[00m #033[00;33m#033[00m#033[00m

Nova compute start delete arq
Aug 22 09:54:50 ubuntu nova-compute[20353]: #033[00;32mDEBUG nova.compute.utils [#033[01;36mNone req-a307b6cf-101a-4e9c-923f-a6fd4d8f6e6f #033[00;36madmin admin#033[00;32m] #033[01;35m#033[00;32mCalling Cyborg to delete ARQs for instance 45716b59-7eaf-4663-bd7e-f83416f7ab48#033[00m #033[00;33m

Cyborg api start delete arq
Aug 22 09:54:50 ubuntu devstack@cyborg-api.service[9353]: 2020-08-22 09:54:50.466 9364 INFO cyborg.api.controllers.v2.arqs [req-3b7aca17-b96c-4d08-9c99-7df8093fc927 613b4ba7e86a4058b458e4e9496a0507 fef0949e2ce6415ab5cd001ba254e563 - default default] [arqs] delete. instance=(45716b59-7eaf-4663-bd7e-f83416f7ab48)

Cyborg api delete arq
Aug 22 09:54:50 ubuntu devstack@cyborg-api.service[9353]: 2020-08-22 09:54:50.487 9364 INFO cyborg.objects.ext_arq [req-3b7aca17-b96c-4d08-9c99-7df8093fc927 613b4ba7e86a4058b458e4e9496a0507 fef0949e2ce6415ab5cd001ba254e563 - default default] Deleting obj_extarq uuid 549f40f9-ff45-4b55-8d7d-89f0f6822203 for instance 45716b59-7eaf-4663-bd7e-f83416f7ab48#033[00m

Arq delete successfully
Aug 22 09:54:50 2020] DELETE /accelerator/v2/accelerator_requests?instance= 45716b59-7eaf-4663-bd7e-f83416f7ab48 => generated 0 bytes in 80 msecs (HTTP/1.1 204) 7 headers in 260 bytes (1 switches on core 0)