Cyborg/TestReport/IntelNic

Intel x710 Nic Cyborg Driver Test Report
This test report shows the current intel nic driver is well tested in local env where Intel x710 Niccard located. Instead of a 3rd party CI, this report can be the proof.

Enable intel nic driver in cyborg agent section
$ vim /etc/cyborg/cyborg.conf [agent] enabled_drivers = fake_driver,intel_nic_driver
 * add intel_nic_driver in enabled_drivers
 * set "enabled_nic_types" to "x710_static", this means the pre-porgrammed NICs are supported.
 * set "physical_device_mappings" and "function_device_mappings" in "x710_static" section, indicating tha mapping relation between physnet/device, function/device.

[nic_devices] enabled_nic_types = x710_static

[x710_static] physical_device_mappings = physnet1:eth2|eth3 function_device_mappings = GTPv1:eth3|eth2 $ systemctl restart devstack@cyborg-agent
 * restart cyborg-agent

Report logs
cyborg-conductor: we can see cyborg conductor call placement API to create releated resource providerFeb 02 03:10:04 newbuy1217.maas cyborg-conductor[1836836]: 2021-02-02 03:10:04.476 1836836 INFO cyborg.common.placement_client [req-27d71744-8a79-4fb6-a2ca-2595b464051c - - - - -] Creating resource provider newbuy1217.maas_0000:86:02.3 Feb 02 03:10:04 newbuy1217.maas cyborg-conductor[1836836]: 2021-02-02 03:10:04.485 1836836 INFO cyborg.common.placement_client [req-27d71744-8a79-4fb6-a2ca- 2595b464051c - - - - -] [req-b6cbe2bb-201a-4c86-9789-02c0a4648be1] Created resource provider record via placement API for resource provider with UUID 5fe904cf-5d52-3dff-9a25- 635841deed1a and name newbuy1217.maas_0000:86:02.3. Feb 02 03:10:04 newbuy1217.maas cyborg-conductor[1836836]: 2021-02-02 03:10:04.520 1836836 INFO cyborg.common.placement_client [req-27d71744-8a79-4fb6-a2ca- 2595b464051c - - - - -] Trait CUSTOM_VF already existed Feb 02 03:10:04 newbuy1217.maas cyborg-conductor[1836836]: 2021-02-02 03:10:04.530 1836836 INFO cyborg.common.placement_client [req-27d71744-8a79-4fb6-a2ca- 2595b464051c - - - - -] Trait CUSTOM_GTPV1 already existed Feb 02 03:10:04 newbuy1217.maas cyborg-conductor[1836836]: 2021-02-02 03:10:04.540 1836836 INFO cyborg.common.placement_client [req-27d71744-8a79-4fb6-a2ca- 2595b464051c - - - - -] Trait CUSTOM_PHYSNET1 already existed Feb 02 03:10:04 newbuy1217.maas cyborg-conductor[1836836]: 2021-02-02 03:10:04.580 1836836 INFO cyborg.common.placement_client [req-27d71744-8a79-4fb6-a2ca- 2595b464051c - - - - -] Resource class CUSTOM_NIC already exists Feb 02 03:10:04 newbuy1217.maas cyborg-conductor[1836836]: 2021-02-02 03:10:04.587 1836836 INFO cyborg.common.placement_client [req-27d71744-8a79-4fb6-a2ca- 2595b464051c - - - - -] Creating resource provider newbuy1217.maas_0000:86:02.1

check placement record
$ openstack resource provider list +--+--++   | uuid                                 | name                         | generation | +--+--++   | 5977653e-12df-319f-a420-17716b7e5123 | newbuy1217.maas_0000:86:00.1 |          2 | | e71b41e9-da06-3b85-8131-b753c01c224b | newbuy1217.maas_0000:86:02.2 |         2 | | 84cd0653-049d-36c7-8ad3-d02f9bad4a8c | newbuy1217.maas_0000:86:02.0 |         2 | | 5fe904cf-5d52-3dff-9a25-635841deed1a | newbuy1217.maas_0000:86:02.3 |         2 | | e95f79d4-8ac1-3b0d-b1ce-002cb073fc9c | newbuy1217.maas_0000:86:02.1 |         2 | +--+--++

There are 5 RPs created because in this env, there are 2 NIC PFs, one of them is virtualized into 6 VFs, the other is not virtualized.

$ openstack resource provider inventory list 5fe904cf-5d52-3dff-9a25-635841deed1a ++--+--+--+--+---+---+   | resource_class | allocation_ratio | min_unit | max_unit | reserved | step_size | total | ++--+--+--+--+---+---+   | CUSTOM_NIC     |              1.0 |        1 |        1 |        0 |         1 |     1 | ++--+--+--+--+---+---+

create device profile and flavor
$ export GRP=[{\"resources\:CUSTOM_NIC\"\:\"1\"\,\"trait:CUSTOM_GTPV1\":\"required\"\,\"trait:CUSTOM_VF\":\"required\"}]

$ openstack accelerator device profile create sriovdp $GRP $ openstack accelerator device profile show d30efbf9-bdf5-45d1-9d34-74d720121971 ++--+   | Field      | Value                                                                                            | ++--+   | created_at | 2021-01-14T08:11:32+00:00                                                                        | | updated_at | None                                                                                            | | uuid      | d30efbf9-bdf5-45d1-9d34-74d720121971                                                             | | name      | sriovdp                                                                                       | | groups    | [{'resources:CUSTOM_NIC': '1', 'trait:CUSTOM_GTPV1': 'required', 'trait:CUSTOM_VF': 'required'}] | ++--+

$ openstack port create test --network public --device-profile new-sriovdp --vnic-type=accelerator-direct

$ neutron port-show 299e665f-502c-42f8-b744-058d8ead5a55 neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead. +---+--+   | Field                 | Value                                                                                | +---+--+   | admin_state_up        | True                                                                                 | | allowed_address_pairs |                                                                                     | | binding:host_id      |                                                                                      | | binding:profile      | {"physical_network": "physnet1"}                                                     | | binding:vif_details  | {}                                                                                   | | binding:vif_type     | unbound                                                                              | | binding:vnic_type    | accelerator-direct                                                                   | | created_at           | 2021-01-14T08:13:05Z                                                                 | | description          |                                                                                      | | device_id            |                                                                                      | | device_owner         |                                                                                      | | device_profile       | new-sriovdp                                                                          | | extra_dhcp_opts      |                                                                                      | | fixed_ips            | {"subnet_id": "26921a61-a89b-44d9-ba3a-6135cfe2ffec", "ip_address": "172.24.4.173"}  | |                      | {"subnet_id": "82914187-5c21-4a78-a787-a7befc10c8d1", "ip_address": "2001:db8::3aa"} | | id                   | 299e665f-502c-42f8-b744-058d8ead5a55                                                 | | mac_address          | fa:16:3e:3e:28:7b                                                                    | | name                 | test                                                                                 | | network_id           | db1a7132-6469-4b75-965e-dc103b1b04cb                                                 | | project_id           | 6aa3db65287949cfab355baff912a5f3                                                     | | revision_number      | 143                                                                                  | | security_groups      | 2fcaedb7-b148-4597-9bc4-6a5173034125                                                 | | status               | DOWN                                                                                 | | tags                 |                                                                                      | | tenant_id            | 6aa3db65287949cfab355baff912a5f3                                                     | | updated_at           | 2021-01-29T07:41:41Z                                                                 | +---+--+

boot a VM with x710 NIC VF attached
$ export ID=299e665f-502c-42f8-b744-058d8ead5a55 $ nova boot testbackend --flavor m1.tiny --image cirros-0.5.1-x86_64-disk  --nic port-id=$ID

Nova get DP details
Feb 03 03:24:30 newbuy1217.maas devstack@cyborg-api.service[1836730]: [pid: 1836744|app: 0|req: 8/181] 192.168.0.186 {60 va    rs in 1207 bytes} [Wed Feb  3 03:24:30 2021] GET /accelerator/v2/device_profiles => generated 738 bytes in 126 msecs (HTTP/1.1    200) 8 headers in 286 bytes (2 switches on core 0) Feb 03 03:29:34 newbuy1217.maas devstack@cyborg-api.service[1836730]: 2021-02-03 03:29:34.389 1836748 INFO cyborg.api.controll ers.v2.device_profiles [req-31b7d118-2df2-403e-8d5d-205be1de2145 50fe054b2deb4669abccf2f3ef980eb9 6aa3db65287949cfab355baff912 a5f3 - default default] [device_profiles] get_all returned:  Feb 03 03:29:34 newbuy1217.maas devstack@cyborg-api.service[1836730]: [pid: 1836748|app: 0|req: 8/182] 192.168.0.186 {60 va    rs in 1375 bytes} [Wed Feb  3 03:29:34 2021] GET /accelerator/v2/device_profiles?name=new-sriovdp => generated 414 bytes in 22 msecs (HTTP/1.1 200) 8 headers in 286 bytes (1 switches on core 0)

ARQ creation
Feb 03 03:29:34 newbuy1217.maas devstack@cyborg-api.service[1836730]: 2021-02-03 03:29:34.409 1836750 INFO cyborg.api.controll ers.v2.arqs [req-a7f6c123-f415-49d9-8aba-ba21582a41e5 50fe054b2deb4669abccf2f3ef980eb9 6aa3db65287949cfab355baff912a5f3 - defa ult default] [arq] post req = ({'device_profile_name': 'new-sriovdp'})

ARQ binding
Feb 03 03:29:34 newbuy1217.maas devstack@cyborg-api.service[1836730]: 2021-02-03 03:29:34.926 1836747 INFO cyborg.api.controll ers.v2.arqs [req-dc455c20-9b91-46b9-a3a5-076a8e5f3688 50fe054b2deb4669abccf2f3ef980eb9 6aa3db65287949cfab355baff912a5f3 - defa ult default] [arqs] patch. list=({'b9bf6023-f1c2-40ec-ab0d-04c3c4958e94': [{'path': '/hostname', 'op': 'add', 'value': 'newbuy   1217.maas'}, {'path': '/device_rp_uuid', 'op': 'add', 'value': 'e95f79d4-8ac1-3b0d-b1ce-002cb073fc9c'}, {'path': '/instance_uu    id', 'op': 'add', 'value': '1fd31114-40cb-4f0f-af49-794545ba8e38'}]})

Get bound ARQ
Feb 03 03:29:35 newbuy1217.maas devstack@cyborg-api.service[1836730]: [pid: 1836755|app: 0|req: 8/186] 192.168.0.186 {58 vars in 1248 bytes} [Wed Feb  3 03:29:35 2021] GET /accelerator/v2/accelerator_requests?instance=1fd31114-40cb-4f0f-af49-794545ba8e38 => generated 680 bytes in 18 msecs (HTTP/1.1 200) 8 headers in 286 bytes (2 switches    on core 0)

Check the VM
Log in to the VM, run`lspci`

It will show a pci device which means the x710 NIC vf has been passthrough to the VM successfully.

$ lspci 00:01.2 Class 0c03: 8086:7020 00:01.0 Class 0601: 8086:7000 00:04.0 Class 0200: 8086:154c 00:00.0 Class 0600: 8086:1237 00:01.3 Class 0680: 8086:7113 ...

We can see 8086:154c which is a x710 Nic' VF.

Nova would reject the server move or suspend
$ openstack server shelve testbackend Forbidden with instances that have accelerators. (HTTP 403) (Request-ID: req-9b988fb9-159f-43df-8f70-ef358f6b1672)

$ openstack server resize testbackend --flavor 3 Forbidden with instances that have accelerators. (HTTP 403) (Request-ID: req-9779d326-647d-4ad4-80b9-a5e1bdaaa436)

]$ openstack server suspend testbackend Forbidden with instances that have accelerators. (HTTP 403) (Request-ID: req-c827af77-c37b-4d66-833b-741781a7e03c)

$ openstack server evacuate testbackend Forbidden with instances that have accelerators. (HTTP 403) (Request-ID: req-21fdf02b-32a0-45b1-b424-8b54614943dc)

$ openstack server migrate --live-migration testbackend Forbidden with instances that have accelerators. (HTTP 403) (Request-ID: req-75591909-29f2-465a-9d16-a31e955e498a)