Jump to: navigation, search

Difference between revisions of "PCI configration Database and API"

(admin check PCI devices avaliable per host)
(Use cases)
Line 65: Line 65:
 
* I think we should list them here: v2/​{tenant_id}​/os-hosts/​{host_name}​/os-pci-devices (or similar)
 
* I think we should list them here: v2/​{tenant_id}​/os-hosts/​{host_name}​/os-pci-devices (or similar)
 
* See http://api.openstack.org/api-ref-compute.html and look for os-hosts
 
* See http://api.openstack.org/api-ref-compute.html and look for os-hosts
 +
 +
 +
====admin want to modify a white-list ====
 +
 +
if want to chang a white list, ie, want use just intel's accelerator, and remove a specify address from group:
 +
 +
# list avaliable global white list
 +
    nova pci-config list white-list
 +
 +
API will be:
 +
    get/v2/​{tenant_id}​/os-pci-config/white-list
 +
    return global white-list list
 +
 +
# check the specify node's white-list:
 +
    nova pci-config list white-list  node <node_id>
 +
 +
API will be:
 +
    get/v2/​{tenant_id}​/os-pci-config/white-list/node/<node-id>
 +
    return all white-list
 +
 +
# get detailed infomation about this whitelist:
 +
    nova pci-config show white-list  <white-list UUID>
 +
 +
API will be:
 +
    get/v2/​{tenant_id}​/os-pci-config/white-list/uuid/<UUID>
 +
    return detailed white list dict.
 +
 +
# delete one key from a pci white list
 +
    nova pci-config update  white-list <UUID>  unset  address= '0000:01:01:7'
 +
 +
API will be:
 +
    update/v2/​{tenant_id}​/os-pci-config/white-list/uuid/<UUID>/unset
 +
    with data { 'address':  '0000:01:01:7'}
 +
 +
# append or change one key's valule
 +
    nova pci-config update  white-list  <UUID>  set vonder_id= '8086'
 +
   
 +
API will be:
 +
    update/v2/​{tenant_id}​/os-pci-config/white-list/uuid/<UUID>
 +
    with data { 'address':  '0000:01:01:7'}
 +
 +
Review comments:
 +
* this is good information, but I don't like the flow or the API
 +
* please rework after seeing my comments above
 +
* I think this could be simpler and more consistent
 +
 +
====admin want to delete a whole white-list ====
 +
 
 +
delete white list by uuid:
 +
    nova pci-config delete white-list  <UUID>
 +
 +
API will be:
 +
    delete/v2/​{tenant_id}​/os-pci-config/white-list/uuid/<UUID>
 +
 +
Review comments:
 +
* as above, please revise
  
 
====admin  create global groups ====
 
====admin  create global groups ====
Line 130: Line 186:
 
Review:
 
Review:
 
* perfect, this is what I hoped for
 
* perfect, this is what I hoped for
 
====admin want to modify a white-list ====
 
 
if want to chang a white list, ie, want use just intel's accelerator, and remove a specify address from group:
 
 
# list avaliable global white list
 
    nova pci-config list white-list
 
 
API will be:
 
    get/v2/​{tenant_id}​/os-pci-config/white-list
 
    return global white-list list
 
 
# check the specify node's white-list:
 
    nova pci-config list white-list  node <node_id>
 
 
API will be:
 
    get/v2/​{tenant_id}​/os-pci-config/white-list/node/<node-id>
 
    return all white-list
 
 
# get detailed infomation about this whitelist:
 
    nova pci-config show white-list  <white-list UUID>
 
 
API will be:
 
    get/v2/​{tenant_id}​/os-pci-config/white-list/uuid/<UUID>
 
    return detailed white list dict.
 
 
# delete one key from a pci white list
 
    nova pci-config update  white-list <UUID>  unset  address= '0000:01:01:7'
 
 
API will be:
 
    update/v2/​{tenant_id}​/os-pci-config/white-list/uuid/<UUID>/unset
 
    with data { 'address':  '0000:01:01:7'}
 
 
# append or change one key's valule
 
    nova pci-config update  white-list  <UUID>  set vonder_id= '8086'
 
   
 
API will be:
 
    update/v2/​{tenant_id}​/os-pci-config/white-list/uuid/<UUID>
 
    with data { 'address':  '0000:01:01:7'}
 
 
Review comments:
 
* this is good information, but I don't like the flow or the API
 
* please rework after seeing my comments above
 
* I think this could be simpler and more consistent
 
 
====admin want to delete a whole white-list ====
 
 
 
delete white list by uuid:
 
    nova pci-config delete white-list  <UUID>
 
 
API will be:
 
    delete/v2/​{tenant_id}​/os-pci-config/white-list/uuid/<UUID>
 
 
Review comments:
 
* as above, please revise
 
  
 
====admin configures SRIOV====
 
====admin configures SRIOV====

Revision as of 06:23, 9 December 2013

API for PCI configration

To get a better between user and amdin, and remove redudent code from pci, alias will fade out, white-list will be used to map devices to an pci-flavor: the group to use for scheduler and configration. cuase KVM not support aggregator , and PCI support only KVM for now, we had to povide a global white list filter to define flavor, then partly relove the aggregating problem, this approach keep the possibility to take advange of aggregtor later.


User will see flavors like:

  • flavor that gives you a cheap GPU
  • flavor that gives you a big GPU
  • flavor that gives you two SSD disks (of varying types depending on where it lands) and a big GPU
  • flavor that gives you your public network via SRIOV (which could be one of several makes of network card, depending on the host picked)

And for the admin...

Admin sees:

  • per host devices (adding things to os-host/<host>/pci-device)
    • lists pci devices present
    • lists pci devices that are exposed to users, and which are in use or free
  • per pci-flavor
    • creates a pci-device description
    • specifies vendor-id, address, uuid, name, etc
    • this used to be a combination of alias and whitelist
    • this could be overlapping descriptions
  • flavor extra specs
    • this has entries that describe:
      • list of possible pci-flavors that could be picked
      • use the key: pci-passthrough:<label_just_to_make_uniqe value:<pci-flavor-uuid-1>,<pci-flavor-uuid-2>
    • for multiple devices, you just add multiple entries

take advantage of host aggregate:

  • host aggregates used to map hosts to pci-flavor ()
    • use host aggregates to expose specific pci-flavors as available on a particular host


Review comments (WIP)

Overall review comments:

  • need a better split between admin (setup) and user use cases
  • need to clearly define what the user will see

I think users will see... (move to text)

Use cases

admin check PCI devices avaliable per host

admin might want to know if there are some pci device avaliable to use, eventhough admin should know such infomation but if the inspect works, it's convenience. there is 2 methods to implemt inspect.

   nova  host-list pci-device 
   GET  v2/​{tenant_id}​/os-hosts/​{host_name}​os-host/<host>/os-pci-devices

return a summary infomation about pci devices on this host:

        [  {'vendor_id':'8086', 'product_id':'xxx',  'address': '0000:01:00.7',  'pci-type'VF', 'status': 'avaliable' , 'pci-flavor':'xxx'  }}

to find which pci-flavor this device belong to, status, we had to query the database.

currently pci device in the databse is after filter, if want inpect the device on node from db, we should let all device going in to database. db will too large then eventually slow down query, became a scale problem. so we'd use RPC call for this goal, use RPC call to get reulst from a node, show it to admin.

Review comments: (addressed, by yongli)

  • I think we need a REST API to list the PCI devices present on a host
  • That API could also show the groups
  • Also maybe it could say which are in use (as currently reported to the scheduler)
  • I think we should list them here: v2/​{tenant_id}​/os-hosts/​{host_name}​/os-pci-devices (or similar)
  • See http://api.openstack.org/api-ref-compute.html and look for os-hosts


admin want to modify a white-list

if want to chang a white list, ie, want use just intel's accelerator, and remove a specify address from group:

  1. list avaliable global white list
   nova pci-config list white-list 

API will be:

   get/v2/​{tenant_id}​/os-pci-config/white-list
   return global white-list list
  1. check the specify node's white-list:
   nova pci-config list white-list  node <node_id>

API will be:

   get/v2/​{tenant_id}​/os-pci-config/white-list/node/<node-id>
   return all white-list 
  1. get detailed infomation about this whitelist:
   nova pci-config show white-list  <white-list UUID>

API will be:

   get/v2/​{tenant_id}​/os-pci-config/white-list/uuid/<UUID>
   return detailed white list dict.
  1. delete one key from a pci white list
   nova pci-config update  white-list <UUID>   unset  address= '0000:01:01:7'

API will be:

    update/v2/​{tenant_id}​/os-pci-config/white-list/uuid/<UUID>/unset
    with data { 'address':  '0000:01:01:7'}
  1. append or change one key's valule
   nova pci-config update  white-list  <UUID>  set vonder_id= '8086'
   

API will be:

    update/v2/​{tenant_id}​/os-pci-config/white-list/uuid/<UUID>
    with data { 'address':  '0000:01:01:7'}

Review comments:

  • this is good information, but I don't like the flow or the API
  • please rework after seeing my comments above
  • I think this could be simpler and more consistent

admin want to delete a whole white-list

delete white list by uuid:

    nova pci-config delete white-list  <UUID>

API will be:

    delete/v2/​{tenant_id}​/os-pci-config/white-list/uuid/<UUID>

Review comments:

  • as above, please revise

admin create global groups

suppose data center have some identical hosts with same configration, so the address for same device will have same address, then amdin want group the address '0000:01:00:7' and '0000:01:01:7' to a group say 'QucikAssist'

than admin can create the white list in one cmd:

   nova pci-config create white-list  set description='xx'   address='0000:01:00:7','0000:01:01:7'  group=QuickAssist'

the API call then will be:

   create/v2/​{tenant_id}​/os-pci-config/white-list

with data:

   { 'group':'QuickAssist',  'address': ['0000:01:01:7', '0000:01:00:7'], 'description': "xxx"}

Review comments:

  • I would go for: create/v2/​{tenant_id}​/os-pci-device-flavor
  • please add a uuid for the group/flavor
  • why is there no vendor id here?
  • please consider the APIs for listing the current white-lists
  • we need all the CRUD operations really. Please see other APIs to copy the patterns we use there (http://api.openstack.org/api-ref-compute.html see host-aggregates)
  • I think you might want to add key-value metadata to these groups, like host-aggregates, but maybe lets ignore that for now, it can be added later

admin create per host groups

say data center have a special powerfull machine with a powerfull PCI 'QucikAssist' card located at '0000:01:00:7', admin want group it's VF to 'PowerDecode'.

create the white list in one cmd:

   nova pci-config create white-list  compute-node-id=<hostid> description='xx'  address= '0000:01:00:7'  group=PowerDecode'

the API call then will be :

   create/v2/​{tenant_id}​/os-pci-config/white-list/

with data:

   { 'group':'PowerDecode',  'address':'0000:01:00:7',  'description': "xxx"  , 'compute-node-id':'hostid'}

Review comments:

  • this feels wrong
  • please just use host aggregates with a special metadata key that links back to the above pci-groups (via uuid) to associate a host, or set of hosts, with the PCI devices

Check if white list work well

after about 30 seconds, white list take effect, so admin want to check is the white list work corretly, so use this: ( this API is covered by another bp(list it here for a full picture view): https://blueprints.launchpad.net/nova/+spec/pci-api-support )

nova pci-list node_id

admin configures alias that request device from a specific group

to allocate the device from a group, you need to create alias for it firstly, the flow is in same way as white list,except alias always global.

create the alias in one cmd:

   nova pci-config create alias  set name='GetMePowerfulldevice'  group=PowerDecode'

the API call then will be :

  create/v2/​{tenant_id}​/os-pci-config/alias

with data:

    { 'group':'PowerDecode', 'name': "GetMePowerfulldevice"}

Review comments:

  • this makes no sense to me
  • if we use host aggregates for the host association, we should just be able to use the group/flavor name here right?

admin configures flavours use above flavor

nova flavor-key m1.small set pci_passthrough:alias=GetMePowerfulldevice:2

Review comments:

  • again, this just becomes the pci-device-flavor

admin boot VM with this flavours

nova boot mytest --flavor m1.small --image=cirros-0.3.1-x86_64-uec

Review:

  • perfect, this is what I hoped for

admin configures SRIOV

Reviewer comments:

  • this info is missing
  • how does the admin associate PCI devices to neutron networks?
  • I think the answer is that the pci-flavor has some metadata that tells you what neutron network it maps to, but its messy

user requests SRIOV

  • 1) setup a group for NIC allocation, the group can based on PF or address.
   nova pci-config create white-list  set "description"='xx'   "address"='0000:01:00:7','0000:01:01:7'  "group"='Intel.NIC.vlan'
  • 2) config the pci manager to use the key 'group' to grouping the device
   add following configration to nova.conf
   pci_tracking_group_by = ['group']

if lack of this, the default grouping policy is use [vendor_id, product_id]


  • 3) config the alias
   nova pci-config create alias  set "name"='vlan-SRIOV'  "group"='intel.NIC.vlan'
  • 4) config the flavor: this is under discuss, neutron might want use --nic command line.
   nova flavor-key m1.small set "pci_passthrough:sriov:<nic1-tag>:alias"="vlan-SRIOV:2"

for sriov there should be a marker <nic1-tag> for nic, the <nic1-tag> will be store in the device allocated to this nic. after allocation finised on compute node, related network module can find the device via the <nic1-tag> to perferm interface configration.

  • 5) admin boot VM with this flavours
   nova boot mytest --flavor m1.small --image=cirros-0.3.1-x86_64-uec


Reviewer comments:

  • I really don't understand this SRIOV stuff
  • I think the VIF attach logic should do the deciding, and we probably need more config in the flavor extra specs to make the mapping

transite config file to API

  1. the config file for alias and whitelist defination is going to deprecated.
  2. if database is not NULL , configration is ommit and given deprecated warning.
  3. if database is NULL, config if read from the file, also given a deprecated notice, which will be remove start from next release.

with this solution, we move pci config from file to API.

DB for pci configration

  talbe: pci_config{
               id   : data base of this k,v pair
               UUID: which alias/whitelist k,v belong to
               compute_node_id:  easy to query per host configration, NULL for global configration
               object : easy to query by object whitelist/alias
               key 
               value
               value_type:  string, RE, list  # for further enhancement 
           }

alias is always global.

PCI releated Objects

  Objects: white list/ alias/ group keys(on demand grop)
  objects property:  sets of k,v
   

API interface

  • get summary info mation
   GET/v2/​{tenant_id}​/os-pci-config/{config-object-name}
   show the global configraion summary.
    i.e {config-object-name} is pci_alias:
    [{"pci_alias":'nic-group', 'UUID":"alias-UUID',  'description':"blabla"}  ]

  • get summary info mation on compute node
   GET/v2/​{tenant_id}​/os-pci-config/{config-object-name}/node/{compute-node-id'}
   show the per compute node  pci configraion summary:
   i.e {config-object-name} is pci_whitelist:
    [{"pci_whitelist":'NIC-2', 'UUID":"whitelist-UUID', 'description':"blabla" } ]
  
  • get details infomation about the config
    GET/v2/​{tenant_id}​/os-pci-config/{config-object-name}/uuid/<UUID>
    i.e {config-object-name} is pci_whitelist:
     {'product_id':'8086', 'group':'nic-g1', 'compute-node-id':"3" ....}
  • update a exist config
   UPDATE//v2/​{tenant_id}​/os-pci-config/{config-object-name}/<UUID>
   with data :
       {  'sriov-type':"macvtap" ....}
   or with array data :
       {  'address': ['a1', 'a2'...] ....}
  • unset a exist config's key value (the value support list)
   UPDATE//v2/​{tenant_id}​/os-pci-config/{config-object-name}/<UUID>/unset
   with data :
       {  'address':"0000:01:00:7"}
    this wil remove the "0000:01:00:7" from address key's value set.
 
  • delete a whole config item
   DELETE/v2/​{tenant_id}​/os-pci-config/{config-object-name}/<UUID>