Jump to: navigation, search


Revision as of 15:15, 15 April 2011 by MikyungKang (talk)


This blueprint proposes to add support for the Tilera tiled-processor machines as an alternative machine type in OpenStack. This blueprint is dependent on the schema changes described in the HeterogeneousInstanceTypes blueprint and the scheduler in HeterogeneousArchitectureScheduler.

The target release for this is Diablo, however the USC-ISI team intends to have a stable test branch and deployment at Cactus release.

The USC-ISI team has a functional prototype here:

This blueprint is related to the HeterogeneousInstanceTypes blueprint here:

We are also drafting blueprints for other machine types:

An etherpad for discussion of this blueprint is available at http://etherpad.openstack.org/heterogeneoustilerasupport

Release Note

Nova has been extended to allow support for the Tilera TILEmpower Platform (TILEPro64 Processor).


See HeterogeneousInstanceTypes.

User stories

Jackie has an application and wants to run on a CPU that does not support KVM or XEN. As an example, she wants to run the application on a TILEmpower board that does not support KVM or XEN. She chooses tp64.8x8 instance type that provides access to a TILEmpower board with TILEPro64 processors having 64 cores in it.

$ euca-add-keypair $key > $key.pem
$ chmod 600 $key.pem
$ euca-run-instances -k $key -t tp64.8x8 ami-5c86a016

$ ssh -i key.pem -p 52108 $Given_Tilera_IP_address


This blueprint is dependent on tp64.8x8 being a selectable instance type and that the scheduler knows this instance must get routed to a TILEmpower board. See HeterogeneousArchitectureScheduler.


The tilera machine is unique in that it has a NxN mesh of processor cores. Currently there is no support for KVM or XEN virtualization, so we are doing a bare metal approach.

Supporting Non-virtualizable Architecture

For supporting non-virtualizable architecture (ex. TILERA), Proxy Compute Node should be designed.

An x86 Proxy Compute Node is connected to the Tilera boards through network. A Proxy Compute Node may handle multiple Tilera boards. Tilera boards are connected to the network such that a cloud user can ssh into them directly after an instance starts on the Tilera boards. A Tilera board is configured to be tftp bootable. Proxy Compute Node behaves as the tftp server for the Tilera boards. After Proxy Compute node receives instance images from the image server, it wakes up a Tilera board and passes the images to the Tilera board using tftp protocol, and control their booting. Once a Tilera board is booted, Proxy Compute Node doesn't do anything except turning off/on the board. Once Tilera instance is running, user can access the TILEmpower board, not Proxy Compute Node, through ssh. Here, we assume that Proxy Compute Node can power on/off Tilera boards remotely using PDU(Power Distribute Unit). The finite state machine shown below describes the procedure in detail.

File:HeterogeneousTileraSupport$Proxy Compute Node.png

Proxy Compute Node keeps the information of the systems that depends on the Proxy Compute Node in the file /tftpboot/architecure_information_file (ex. tilera_boards). The file contains static information of the boards such as MAC address, processor type, memory size, disk size. It is used for handling dynamic information such as the status of the board (power off/booting/running/shutting down), and the instance_id if a board is running an instance. It is Proxy Compute Node's responsibility to keep track of the status of the dependent systems.

1. TFTP setting for Proxy Compute Node

$ vi /etc/xinetd.d/tftp 
service tftp
protocol = udp
port = 69
socket_type = dgram
wait = yes
user = root 
server = /usr/sbin/in.tftpd
server_args = /tftpboot
disable = no
$ /etc/init.d/xinetd restart

2. File preparation in Proxy Compute Node:/tftpboot

Copy the following files to /tftpboot:
• vmlinuz   // linux image
• initrd        // init ramfile system
• disk          // file system image
• architecture_information_file (ex. tilera_boards)
• pdu_mgr

 2.1 Example of architecture_information_file: /tftpboot/tilera_boards
 : Proxy Compute Node manges Tilera board information (board_id, board_ip_address, board_mac_address, board_hw_description, etc.) using this tilera_boards file.
# board_id  ip_address mac_address 
0     00:1A:CA:00:57:90 10 16218 917 476 1 tilera_hv 1 
1     00:1A:CA:00:58:98 10 16218 917 476 1 tilera_hv 1       
2     00:1A:CA:00:58:50 10 16218 917 476 1 tilera_hv 1      
3     00:1A:CA:00:57:A8 10 16218 917 476 1 tilera_hv 1    
4     00:1A:CA:00:58:AA 10 16218 917 476 1 tilera_hv 1 
5     00:1A:CA:00:58:2C 10 16218 917 476 1 tilera_hv 1 
6     00:1A:CA:00:58:5C 10 16218 917 476 1 tilera_hv 1       
7     00:1A:CA:00:58:A4 10 16218 917 476 1 tilera_hv 1 
8     00:1A:CA:00:58:1A 10 16218 917 476 1 tilera_hv 1 
9    00:1A:CA:00:58:38 10 16218 917 476 1 tilera_hv 1  

 2.2 pdu-mgr
 : PDU(Power Distribute Unit)-controlling EXPECT script for remote power-on/off/reboot

Schema Changes in Bare metal implementation for TILEmpower boards

We're proposing the following default values added to the instance_types table.

   't64.8x8':  dict(memory_mb=16384, vcpus=1, local_gb=500,
   'tp64.8x8': dict(memory_mb=16384, vcpus=1, local_gb=500,
   'tgx.4x4':  dict(memory_mb=16384, vcpus=1, local_gb=500,
   'tgx.6x6':  dict(memory_mb=16384, vcpus=1, local_gb=500,
   'tgx.8x8':  dict(memory_mb=16384, vcpus=1, local_gb=500,
   'tgx.10x10':  dict(memory_mb=16384, vcpus=1, local_gb=500,

Proxy Compute Node in Bare metal implementation for TILEmpower boards (nova/virt/tilera.py)

Tilera Compute Node shown in below is an example of Proxy Compute Node. After setting the status of instance and domain as Pending, Proxy Compute Node copies vmlinux_x_1 to vmlinux_x. The x means the board_id and 1 means the first boot with vmlinux image which sets TLR_ROOT=tmpfs. By default the rootfs is copied to a tmpfs whose size limit is half of total memory. After 1st mboot-run through tftp download of vmlinux, Proxy Compute Node uploads compressed tilera file system into memory, mounts /dev/sda1 to /mnt, and uncompresses the uploaded tilera file system into /mnt disk space. And then Proxy Compute Node copies vmlinux_x_2 to vmlinux_x. The x means the board_id and 2 means the second boot with vmlinux image which sets TLR_ROOT=/dev/sda1. After second mboot-run, rootfs is set as /dev/sda1. After that, Proxy Compute Node sets the status of instance and domain as Running. Then user can access the board through ssh.

File:HeterogeneousTileraSupport$Tilera Compute.png


The USC-ISI team has a functional prototype: https://code.launchpad.net/~usc-isi/nova/hpc-testing

Proxy Compute Node should be implemented as proxy_baremetal.py and specific_architecture.py (for example, tilera.py or arm.py). The proxy_baremetal.py code may describe Connection class of the non-virtualizable architecture and specific architecture calls may be invoked within the proxy_baremetal.py code. Architecture specific routines are described in specific_architecture.py. In current implementation, tilera.py code covers all of them and it will be separated into two parts(proxy_baremetal.py and tilera.py) later.

File:HeterogeneousTileraSupport$Next Baremetal.png

UI Changes

The following will be available as new default instance types.

Tilera TILEPro64

  • API name: tp64.8x8
  • TILEPro64 processor: 64 (8x8) cores
  • 16 GB RAM (16384 MB)
  • 1 TB of instance storage
  • http://www.tilera.com/

(Only one Tilera instance type for now. When KVM support appears, we will add additional types to support partitioning into smaller instances)

Code Changes

  • nova/db/sqlalchemy/migrate_repo/versions/013_add_architecture_to_instance_types.py
  - add default instance types for tilera
  • nova/virt/connection.py
  - add tilera connection_type
  • nova/virt/tilera.py
  - new tilera connection code for launching instances on TILEmpower board
  • nova/virt/tilera.xml.template and nova/virt/tilera_interfaces.template
  - new tilera template files
  - add tilera related template files


Very little needs to change in terms of the way deployments will use this if we set sane defaults like "x86_64" as assumed today.

Test/Demo Plan

This need not be added or completed until the specification is nearing beta.

Unresolved issues

One of the challenges we have is that the flavorid field in the instance_types table isn't auto-increment. We've selected high numbers to avoid collisions, but the community should discuss how flavorid behaves and the best approach for adding future new instance types.

BoF agenda and discussion

Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected.