Cyborg/TestReport/IntelQAT

Intel QAT Cyborg Driver Test Report
This test report shows the current qat driver is well tested in local env where QAT card located. Instead of a 3rd party CI, this report can be the proof.

Enable qat driver in cyborg agent section
$ cat /etc/cyborg/cyborg.conf | grep intel enabled_drivers = intel_qat_driver

Report logs
cyborg-conductor: we can see cyborg conductor call placement API to create releated resource provider Aug 18 01:34:28 vca-server cyborg-conductor[2243750]: 2020-08-18 01:34:28.929 2243750 INFO cyborg.conductor.manager [req-08b86085-4bc4-4cce-9fcd-9c60e99fc c33 - - - - -] Start differing devices. Aug 18 01:34:29 vca-server cyborg-conductor[2243750]: 2020-08-18 01:34:29.085 2243750 INFO cyborg.common.placement_client [req-08b86085-4bc4-4cce-9fcd-9c6 0e99fcc33 - - - - -] Resource class CUSTOM_QAT already exists Aug 18 01:34:29 vca-server cyborg-conductor[2243750]: 2020-08-18 01:34:29.093 2243750 INFO cyborg.common.placement_client [req-08b86085-4bc4-4cce-9fcd-9c6 0e99fcc33 - - - - -] Creating resource provider intel-qat-dev_0000:3d:02.7 Aug 18 01:34:29 vca-server cyborg-conductor[2243750]: 2020-08-18 01:34:29.102 2243750 INFO cyborg.common.placement_client [req-08b86085-4bc4-4cce-9fcd-9c6 0e99fcc33 - - - - -] [req-418258f7-a564-4d25-a81a-9e51062a27f3] Created resource provider record via placement API for resource provider with UUID 17cedae e-f4ad-3521-b815-b2f14c4fd621 and name intel-qat-dev_0000:3d:02.7. Aug 18 01:34:29 vca-server cyborg-conductor[2243750]: 2020-08-18 01:34:29.160 2243750 INFO cyborg.common.placement_client [req-08b86085-4bc4-4cce-9fcd-9c6 0e99fcc33 - - - - -] Resource class CUSTOM_QAT already exists Aug 18 01:34:29 vca-server cyborg-conductor[2243750]: 2020-08-18 01:34:29.168 2243750 INFO cyborg.common.placement_client [req-08b86085-4bc4-4cce-9fcd-9c6 0e99fcc33 - - - - -] Creating resource provider intel-qat-dev_0000:3d:02.0 Aug 18 01:34:29 vca-server cyborg-conductor[2243750]: 2020-08-18 01:34:29.180 2243750 INFO cyborg.common.placement_client [req-08b86085-4bc4-4cce-9fcd-9c6 0e99fcc33 - - - - -] [req-2d09804c-89a8-4299-bf78-0afa1760a85f] Created resource provider record via placement API for resource provider with UUID b937b6d 5-2ac0-3746-b3ba-f7792887a81f and name intel-qat-dev_0000:3d:02.0. Aug 18 01:34:29 vca-server cyborg-conductor[2243750]: 2020-08-18 01:34:29.237 2243750 INFO cyborg.common.placement_client [req-08b86085-4bc4-4cce-9fcd-9c6 0e99fcc33 - - - - -] Resource class CUSTOM_QAT already exists ...

check placement record
$ openstack resource provider list +--++---+   | uuid                                  | name                         | generation | +--++---+   | 8fc1b918-04e2-4ba1-be7d-69387d6803d8 | vca-server                 |         68 | | 17cedaee-f4ad-3521-b815-b2f14c4fd621 | intel-qat-dev_0000:3d:02.7 |         1 | | b937b6d5-2ac0-3746-b3ba-f7792887a81f | intel-qat-dev_0000:3d:02.0 |         1 | | 9e189749-2304-3aaf-b590-cab4209a5cbc | intel-qat-dev_0000:3d:02.5 |         1 | | 57841a47-b674-3fe4-a54d-12fbd6addeeb | intel-qat-dev_0000:3d:01.6 |         1 | | d127b5b3-39a0-3c89-9c5d-88bc4afa14bc | intel-qat-dev_0000:3d:02.3 |         1 | | 456bf96e-5a8d-36bb-b5ff-11c2a7a8ba3b | intel-qat-dev_0000:3d:01.4 |         1 | | 5bfd1cff-5c51-3cdc-a992-0cc0c92431b4 | intel-qat-dev_0000:3d:01.2 |         1 | | f19ed211-f71e-3d14-9149-4c06e47a965c | intel-qat-dev_0000:3d:01.0 |         1 | ....

There are 48 RPs created, because in this env, there are 3 QAT PFs, each of them is virtualized into 16 VFs.

$ openstack resource provider list | grep qat | wc -l 48

$ openstack resource provider inventory list d6cdd568-2d87-3668-a0aa-5d19787d4ce2 ++--+--+--+--+---+---+   | resource_class | allocation_ratio | min_unit | max_unit | reserved | step_size | total | ++--+--+--+--+---+---+   | CUSTOM_QAT     |              1.0 |        1 |        1 |        0 |         1 |     1 | ++--+--+--+--+---+---+

create device profile and flavor
$ echo $BODY | python3 -m json.tool [       {            "name": "QAT", "groups": [ {                   "resources:CUSTOM_QAT": "1" }           ]        }    ]

$ curl -s -H "$CTYPE" -H "$AUTH" POST -d "$BODY" $CYURL/device_profiles {"name": "QAT", "description": null, "uuid": "dd9d2759-eec3-4bfe-af95-24773d28e8f4", "groups": [{"resources:CUSTOM_QAT": "1"}], "created_at": "2020-08-18   05:50:48+00:00", "updated_at": "None", "links": [{"href": "http://localhost/accelerator/v2/device_profiles/dd9d2759-eec3-4bfe-af95-24773d28e8f4", "rel": $self"}]}

$ openstack flavor create fpga-flavor --property accel:device_profile='QAT' --disk 5 ++--+   | Field                      | Value                                | ++--+   | OS-FLV-DISABLED:disabled   | False                                | | OS-FLV-EXT-DATA:ephemeral | 0                                    | | disk                      | 5                                    | | id                        | f518569f-812a-4382-bfd3-9dc61ed848c9 | | name                      | fpga-flavor                          | | os-flavor-access:is_public | True                                | | properties                | accel:device_profile='QAT'           | | ram                       | 256                                  | | rxtx_factor               | 1.0                                  | | swap                      |                                      | | vcpus                     | 1                                    | ++--+

boot a VM with QAT attached
In this step, I gave some output of cyborg-api log to show how Nova interact with Cyborg.

Nova get DP details
Aug 18 02:03:09 vca-server devstack@cyborg-api.service[2243671]: 2020-08-18 02:03:09.631 2243678 INFO cyborg.api.controllers.v2.device_profiles [req-75741c76-593a-40bf- 8955-80eb0ec29c63 1f86481df3024d8bb74e213e23dcc2e4 7cd6c62783934a148605e271b4f6f922 - default default] [device_profiles] get_all returned: { 'device_profiles': [{'name': 'QAT', 'description': None, 'uuid': 'dd9d2759-eec3-4bfe-af95-24773d28e8f4', 'groups': [{'resources:CUSTOM_QAT': '1'}], 'created_at': '2020-08-18 05:50:48+00:00', 'updated_at': 'None', 'links': [{'href': 'http://10.238.153.121/accelerator/v2/device_profiles/dd9d2759-eec3-4bfe-af9 5-24773d28e8f4', 'rel': 'self'}]}]}

ARQ creation
Aug 18 02:03:10 vca-server devstack@cyborg-api.service[2243671]: 2020-08-18 02:03:10.052 2243674 INFO cyborg.api.controllers.v2.arqs [req-73c06ce1-b9df-4d94-8760- e4ab5f595328 1f86481df3024d8bb74e213e23dcc2e4 7cd6c62783934a148605e271b4f6f922 - default default] [arq] post req = ({'device_profile_name': 'QAT'})

ARQ binding
Aug 18 02:03:10 vca-server devstack@cyborg-api.service[2243671]: 2020-08-18 02:03:10.118 2243675 INFO cyborg.api.controllers.v2.arqs [req-fba1cad3-bccc-4c 4a-9a78-2cc145c4798b 1f86481df3024d8bb74e213e23dcc2e4 7cd6c62783934a148605e271b4f6f922 - default default] [arqs] patch. list=({'0fff41ad-487e-4a3d-8395-2c   4d35142c1f': [{'path': '/hostname', 'op': 'add', 'value': 'vca-server'}, {'path': '/device_rp_uuid', 'op': 'add', 'value': 'fb328624-7775-3d1e-bb36-478200    ea1a5a'}, {'path': '/instance_uuid', 'op': 'add', 'value': '92080153-a69a-4f1e-9ee7-34298673ad1e'}]})

Get bound ARQ
Aug 18 02:03:10 vca-server devstack@cyborg-api.service[2243671]: 2020-08-18 02:03:10.711 2243672 INFO cyborg.api.controllers.v2.arqs [req-bb8b6928-137e-4b 87-aac0-5fe2cecc22d7 1f86481df3024d8bb74e213e23dcc2e4 7cd6c62783934a148605e271b4f6f922 - default default] [arqs:get_all] Returned: 

Check the VM
Log in to the VM, run`lspci | grep 37c9`

It will show a pci device which means the QAT vf has been passthrough to the VM successfully.

$ lspci | grep 37c9 08:00.1 Intel Corporation Device 37c9 (rev 04)