Difference between revisions of "HeterogeneousGpuAcceleratorSupport"
(→Summary) |
|||
| (6 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
| − | + | ||
| − | * '''Launchpad Entry''': | + | * '''Launchpad Entry''': NovaSpec:heterogeneous-gpu-accelerator-support |
* '''Created''': [https://launchpad.net/~bfschott Brian Schott] | * '''Created''': [https://launchpad.net/~bfschott Brian Schott] | ||
* '''Current maintainer''': [https://launchpad.net/~jwalters-isi John Paul Walters] | * '''Current maintainer''': [https://launchpad.net/~jwalters-isi John Paul Walters] | ||
| Line 7: | Line 7: | ||
== Summary == | == Summary == | ||
| − | This blueprint proposes to add support for GPU- | + | This blueprint proposes to add support for GPU-accelerated machines as an alternative machine type in [[OpenStack]]. |
| − | The target release for this is | + | The target release for this is Grizzly. We plan to have a stable branch at https://code.launchpad.net/~usc-isi/nova/hpc-testing. |
The USC-ISI team has a functional prototype here: | The USC-ISI team has a functional prototype here: | ||
* https://code.launchpad.net/~usc-isi/nova/hpc-trunk | * https://code.launchpad.net/~usc-isi/nova/hpc-trunk | ||
| − | |||
This blueprint is related to the [[HeterogeneousInstanceTypes]] blueprint here: | This blueprint is related to the [[HeterogeneousInstanceTypes]] blueprint here: | ||
* http://wiki.openstack.org/HeterogeneousInstanceTypes | * http://wiki.openstack.org/HeterogeneousInstanceTypes | ||
| − | |||
| − | |||
| − | |||
| − | |||
An etherpad for discussion of this blueprint is available at http://etherpad.openstack.org/heterogeneousultravioletsupport | An etherpad for discussion of this blueprint is available at http://etherpad.openstack.org/heterogeneousultravioletsupport | ||
| Line 36: | Line 31: | ||
== User stories == | == User stories == | ||
| − | Jackie has a CUDA-accelerated application and wants to run it on an instance that has access to GPU hardware. She chooses a cg1. | + | Jackie has a CUDA-accelerated application and wants to run it on an instance that has access to GPU hardware. She chooses a cg1.xlarge instance type, that provides access to two NVIDIA Fermi GPUs: |
<pre><nowiki> | <pre><nowiki> | ||
| − | + | $ nova flavor-list | |
| + | +----+-----------+-----------+------+-----------+------+-------+-------------+-----------+----------------------------------------------+ | ||
| + | | ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public | extra_specs | | ||
| + | +----+-----------+-----------+------+-----------+------+-------+-------------+-----------+----------------------------------------------+ | ||
| + | | 9 | cg1.xlarge | 16384 | 160 | 0 | | 8 | 1.0 | True | {u'hypervisor': u's== LXC', u'gpus': u'= 2', u'gpu_arch':u's== fermi'} | | ||
| + | +----+-----------+-----------+------+-----------+------+-------+-------------+-----------+----------------------------------------------+ | ||
| + | |||
| + | $ nova boot --flavor 9 --key-name mykey --image 2b1509fe-b573-488a-be4d-d61d25c7ab4f gpu_test | ||
</nowiki></pre> | </nowiki></pre> | ||
| Line 45: | Line 47: | ||
== Assumptions == | == Assumptions == | ||
| − | + | The only approach that has been successful for CUDA access from a kvm virtual machine that we know of is gVirtuS [http://osl.uniparthenope.it/projects/gvirtus/]. | |
| − | + | Here we propose direct access of gpus from LXC instances. | |
| − | The only approach that has been successful for CUDA access from a kvm virtual machine that we know of is gVirtuS [http://osl.uniparthenope.it/projects/gvirtus/]. | + | We assume that the host system's kernel supports 'lxc-attach', and the utilities for 'lxc-attach' are installed. |
== Design == | == Design == | ||
| − | We | + | We have have new nova.virt.GPULibvirt which is an extension of nova.virt.libvirt to instantiate a GPU-enabled virtual machine when requested. |
| − | |||
| − | |||
| − | * When an instance is spawned (or rebooted), nova starts | + | * When an instance is spawned (or rebooted), nova starts an LXC VM |
| − | * The | + | * The requested gpu(s) is(are) marked as allocated and its(their) device(s) is(are) created inside LXC using 'lxc-attach' |
| − | * | + | * Access permission to the gpu(s) is added to /cgroup |
* Boot finalizes | * Boot finalizes | ||
| − | * When an instance is terminated (destroyed) | + | * When an instance is terminated (destroyed), the gpu(s) are deallocated. |
=== Schema Changes === | === Schema Changes === | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
== Implementation == | == Implementation == | ||
| Line 108: | Line 67: | ||
The USC-ISI team has a functional prototype: | The USC-ISI team has a functional prototype: | ||
https://code.launchpad.net/~usc-isi/nova/hpc-trunk | https://code.launchpad.net/~usc-isi/nova/hpc-trunk | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
== GPUs (NVIDIA Teslas) == | == GPUs (NVIDIA Teslas) == | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
==== Code Changes ==== | ==== Code Changes ==== | ||
| − | * | + | * added nova/virt/gpu/driver.py |
| − | + | Inherits [[LibvirtDriver]] and extends a few methods to provision gpus | |
| − | + | Adds a few flags to describe gpu architecture, number of gpus, device ids, etc. | |
| − | |||
| − | |||
| − | |||
| − | * | + | * added nova/virt/gpu/utils.py |
| + | Gpu provisioning routines | ||
=== Migration === | === Migration === | ||
| − | + | Unless the migration target supports gpus of the same indices, it may not work. | |
== Test/Demo Plan == | == Test/Demo Plan == | ||
| Line 196: | Line 88: | ||
== Unresolved issues == | == Unresolved issues == | ||
| − | |||
| − | |||
| − | |||
| − | |||
== BoF agenda and discussion == | == BoF agenda and discussion == | ||
Latest revision as of 02:49, 13 June 2015
- Launchpad Entry: NovaSpec:heterogeneous-gpu-accelerator-support
- Created: Brian Schott
- Current maintainer: John Paul Walters
- Contributors: USC Information Sciences Institute
Contents
Summary
This blueprint proposes to add support for GPU-accelerated machines as an alternative machine type in OpenStack.
The target release for this is Grizzly. We plan to have a stable branch at https://code.launchpad.net/~usc-isi/nova/hpc-testing.
The USC-ISI team has a functional prototype here:
This blueprint is related to the HeterogeneousInstanceTypes blueprint here:
An etherpad for discussion of this blueprint is available at http://etherpad.openstack.org/heterogeneousultravioletsupport
Release Note
Nova has been extended to make NVIDIA GPUs available to provisioned instances for CUDA programming.
Rationale
See HeterogeneousInstanceTypes.
The goal of this blueprint is to allow GPU-accelerated computing in OpenStack.
User stories
Jackie has a CUDA-accelerated application and wants to run it on an instance that has access to GPU hardware. She chooses a cg1.xlarge instance type, that provides access to two NVIDIA Fermi GPUs:
$ nova flavor-list
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+----------------------------------------------+
| ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public | extra_specs |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+----------------------------------------------+
| 9 | cg1.xlarge | 16384 | 160 | 0 | | 8 | 1.0 | True | {u'hypervisor': u's== LXC', u'gpus': u'= 2', u'gpu_arch':u's== fermi'} |
+----+-----------+-----------+------+-----------+------+-------+-------------+-----------+----------------------------------------------+
$ nova boot --flavor 9 --key-name mykey --image 2b1509fe-b573-488a-be4d-d61d25c7ab4f gpu_test
Assumptions
The only approach that has been successful for CUDA access from a kvm virtual machine that we know of is gVirtuS [1]. Here we propose direct access of gpus from LXC instances. We assume that the host system's kernel supports 'lxc-attach', and the utilities for 'lxc-attach' are installed.
Design
We have have new nova.virt.GPULibvirt which is an extension of nova.virt.libvirt to instantiate a GPU-enabled virtual machine when requested.
- When an instance is spawned (or rebooted), nova starts an LXC VM
- The requested gpu(s) is(are) marked as allocated and its(their) device(s) is(are) created inside LXC using 'lxc-attach'
- Access permission to the gpu(s) is added to /cgroup
- Boot finalizes
- When an instance is terminated (destroyed), the gpu(s) are deallocated.
Schema Changes
Implementation
The USC-ISI team has a functional prototype: https://code.launchpad.net/~usc-isi/nova/hpc-trunk
GPUs (NVIDIA Teslas)
Code Changes
- added nova/virt/gpu/driver.py
Inherits LibvirtDriver and extends a few methods to provision gpus Adds a few flags to describe gpu architecture, number of gpus, device ids, etc.
- added nova/virt/gpu/utils.py
Gpu provisioning routines
Migration
Unless the migration target supports gpus of the same indices, it may not work.
Test/Demo Plan
This need not be added or completed until the specification is nearing beta.
Unresolved issues
BoF agenda and discussion
Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected.
