Summary

This blueprint proposes to add support for the SGI Ultra Violet (UV) shared memory machines as an alternative machine type in OpenStack. This blueprint is dependent on the schema changes described in the HeterogeneousInstanceTypes blueprint and the scheduler in HeterogeneousArchitectureScheduler.

The target release for this is Diablo. The USC-ISI team has a stable test branch and deployment at https://code.launchpad.net/~usc-isi/nova/hpc-testing.

The USC-ISI team has a functional prototype here:

This blueprint is related to the HeterogeneousInstanceTypes blueprint here:

We are also drafting blueprints for other machine types:

An etherpad for discussion of this blueprint is available at http://etherpad.openstack.org/heterogeneousultravioletsupport

Release Note

Nova has been extended to provide support for the SGI UV family of shared memory machines.

Rationale

See HeterogeneousInstanceTypes.

The goal is to allow openstack to launch virtual machines on SGI Altix UV machines. There are many user scenarios that involve subdividing this large SMP machine across multiple users with different configuration requirements. Even with a standard X86_64 virtual machine image taken from a stock openstack glance repository, an instance could be provisioned with 64 cores and 256GB of memory without modification.

User stories

Jackie has a Linux virtual machine image running on a blade server that runs out of memory. She sees an SGI UV is available. She chooses a uv1.8xlarge with 128GB of memory and 64 cores. 

euca-run-instances -t uv1.8xlarge -k jackie -keypair emi-12345678

Assumptions

This blueprint is dependent on uv1.8xlarge being a selectable instance type and that the scheduler knows this instance must get routed to a SGI UV machine. We assume that cpu_arch, cpu_info, xpu_arch, xpu_info, xpus, net_arch, net_info, and net_mbps as attributes to instance_types, instances, and compute_nodes tables are provided. See HeterogeneousArchitectureScheduler.

Design

The UV machine is unique in that it supports X86_64 cpu_arch type and runs standard Linux binaries, but normally runs a largesmp kernel to enable more memory and more cores in a single system image. There are also special OpenMP, MPI, and other libraries that take advantage of the NUMAlink5 distributed memory fabric. You can manipulate the memory and processor locality/affinity to performance tune parallel applications.

We have several configuration approaches. Only the first needs to be successful for this project to be worthwhile.

Schema Changes

See HeterogeneousInstanceTypes.

We're proposing the following default values added to the instance_types table. 

    # Shared-memory (SGI UV)
    uv1.small': dict(memory_mb=2048, vcpus=1, local_gb=20,
                      flavorid=200,
                      cpu_arch="x86_64",
                      cpu_info='{"feature":"NUMAlink5"}'),
    'uv1.medium': dict(memory_mb=4096, vcpus=2, local_gb=40,
                       flavorid=201,
                       cpu_arch="x86_64",
                       cpu_info='{"feature":"NUMAlink5"}'),
    'uv1.large': dict(memory_mb=8192, vcpus=4, local_gb=80,
                       flavorid=202,
                      cpu_arch="x86_64",
                      cpu_info='{"feature":"NUMAlink5"}'),
    'uv1.xlarge': dict(memory_mb=16384, vcpus=8, local_gb=160,
                       flavorid=203,
                       cpu_arch="x86_64",
                       cpu_info='{"feature":"NUMAlink5"}'),
    'uv1.2xlarge': dict(memory_mb=32768, vcpus=16, local_gb=320,
                        flavorid=204,
                        cpu_arch="x86_64", 
                        cpu_info='{"feature":"NUMAlink5"}'),
    'uv1.4xlarge': dict(memory_mb=65536, vcpus=32, local_gb=320,
                        flavorid=205,
                        cpu_arch="x86_64", 
                        cpu_info='{"feature":"NUMAlink5"}'),
    'uv1.8xlarge': dict(memory_mb=131072, vcpus=64, local_gb=500,
                        flavorid=206,
                        cpu_arch="x86_64", 
                        cpu_info='{"feature":"NUMAlink5"}'),
    'uv1.16xlarge': dict(memory_mb=262144, vcpus=128, local_gb=500,
                         flavorid=207,
                         cpu_arch="x86_64", 
                         cpu_info='{"feature":"NUMAlink5"}'),
    'uv1.32xlarge': dict(memory_mb=524288, vcpus=256, local_gb=1000,
                         flavorid=208,
                         cpu_arch="x86_64",
                         cpu_info='{"feature":"NUMAlink5"}'),
    'uv1.64xlarge': dict(memory_mb=1048576, vcpus=512, local_gb=1000,
                         flavorid=209,
                         cpu_arch="x86_64",
                         cpu_info='{"feature":"NUMAlink5"}'),
    'uv1.128xlarge': dict(memory_mb=2097152, vcpus=1024, local_gb=2000,
                         flavorid=209,
                         cpu_arch="x86_64",
                         cpu_info='{"feature":"NUMAlink5"}'),
    'uv1.256xlarge': dict(memory_mb=4194304, vcpus=2048, local_gb=4000,
                         flavorid=209,
                         cpu_arch="x86_64",
                         cpu_info='{"feature":"NUMAlink5"}'),

Implementation

The USC-ISI team has a functional prototype: https://code.launchpad.net/~usc-isi/nova/hpc-trunk

UI Changes

The following will be available as new default instance types.

UV small

UV medium

UV large

UV xlarge

UV 2xlarge

UV 4xlarge

UV 8xlarge

UV 16xlarge

UV 32xlarge

UV 64xlarge

UV 128xlarge

UV 256xlarge

Code Changes

Migration

Very little needs to change in terms of the way deployments will use this if we set sane defaults like "x86_64" as assumed today.

Test/Demo Plan

This need not be added or completed until the specification is nearing beta.

Unresolved issues

One of the challenges we have is that the flavorid field in the instance_types table isn't auto-increment. We've selected high numbers to avoid collisions, but the community should discuss how flavorid behaves and the best approach for adding future new instance types.

BoF agenda and discussion

Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected.

CategorySpec

Wiki: HeterogeneousSgiUltraVioletSupport (last edited 2011-04-21 14:10:18 by DavidKang)