Jump to: navigation, search

Difference between revisions of "HeterogeneousTileraSupport"

Line 34: Line 34:
  
 
Only tp64.8x8 instance type is chosen with 16GB of memory and 64 cores.
 
Only tp64.8x8 instance type is chosen with 16GB of memory and 64 cores.
 +
 +
To make and register tilera_vmlinux kernel image,
  
 
<pre><nowiki>
 
<pre><nowiki>
Line 41: Line 43:
 
</nowiki></pre>
 
</nowiki></pre>
  
 +
To make and register tilera_fs file system image,
  
 
<pre><nowiki>
 
<pre><nowiki>
Line 48: Line 51:
 
</nowiki></pre>
 
</nowiki></pre>
  
 +
To verify the registered tilera kernel and file system,
  
 
<pre><nowiki>
 
<pre><nowiki>
Line 55: Line 59:
 
</nowiki></pre>
 
</nowiki></pre>
  
 +
To edit/check the tilera image (ex. ami-5c86a016),
  
 
<pre><nowiki>
 
<pre><nowiki>
Line 61: Line 66:
 
</nowiki></pre>
 
</nowiki></pre>
  
 +
To run tilera instances,
  
 
<pre><nowiki>
 
<pre><nowiki>
Line 68: Line 74:
 
</nowiki></pre>
 
</nowiki></pre>
  
 +
To access tilera instances,
  
 
<pre><nowiki>
 
<pre><nowiki>
Line 119: Line 126:
 
=== Tilera Compute Node's Design ===
 
=== Tilera Compute Node's Design ===
  
All 10 TILEmpower boards (with TILEPro64 processor) are connected to one x86 Tilera Compute Node through USB-to-16 Serial Adapter. The vmlinux file is downloaded from Tilera Compute Node to each TILEmpower board through TFTP which is setup in boot parameters.
+
All 10 TILEmpower boards #0~9 (with TILEPro64 processor) are connected to one x86 Tilera Compute Node through USB-to-16 Serial Adapter. The vmlinux file is downloaded from Tilera Compute Node to each TILEmpower board through TFTP which is setup in boot parameters on each board. Once tilera file system is ready on TILEmpower board, Tilera Compute Node doesn't do anything except turning off/on the board. Once tilera instance is running, user can access the TILEmpower board, not Tilera Compute Node, through ssh.
  
1. TFTP setting for Tilera Compute Node
+
'''1. TFTP setting for Tilera Compute Node'''
  
 
<pre><nowiki>
 
<pre><nowiki>
Line 140: Line 147:
  
  
2. File preparation in Tilera Compute Node:/tftpboot
+
'''2. File preparation in Tilera Compute Node:/tftpboot'''
 +
 
 +
- Different vmlinux and file system are used for each TILEmpower board.
  
 
<pre><nowiki>
 
<pre><nowiki>
Line 158: Line 167:
  
  
2.1 Tilera boards information: /tftpboot/tilera_boards
+
'''2.1 Tilera boards information: /tftpboot/tilera_boards'''
 +
 
 +
- Tilera Compute Node manges board information (board_id, board_ip_address, board_mac_address, board_hw_description, etc.) using this tilera_boards file.
  
 
<pre><nowiki>
 
<pre><nowiki>
Line 175: Line 186:
  
  
2.2 pdu_mgr: PDU(Power Distribute Unit)-controlling EXPECT script for remote turn-on/off/reboot of TILEmpower board
+
'''2.2 pdu_mgr'''
 +
 
 +
- PDU(Power Distribute Unit)-controlling EXPECT script for remote turn-on/off/reboot of TILEmpower board
  
3. Bare metal approach without supporting KVM or XEN virtualization (nova/virt/tilera.py)
+
'''3. Bare metal approach without supporting KVM or XEN virtualization (nova/virt/tilera.py)'''
 
[[Image:HeterogeneousTileraSupport$Tilera_Compute.png]]
 
[[Image:HeterogeneousTileraSupport$Tilera_Compute.png]]
  
Line 195: Line 208:
 
* 16 GB RAM (16384 MB)
 
* 16 GB RAM (16384 MB)
 
* 1 TB of instance storage
 
* 1 TB of instance storage
 +
* http://www.tilera.com/
  
 
(Only one Tilera instance type for now. When KVM support appears, we will add additional types to support partitioning into smaller instances)
 
(Only one Tilera instance type for now. When KVM support appears, we will add additional types to support partitioning into smaller instances)

Revision as of 18:34, 12 April 2011

Summary

This blueprint proposes to add support for the Tilera tiled-processor machines as an alternative machine type in OpenStack. This blueprint is dependent on the schema changes described in the HeterogeneousInstanceTypes blueprint and the scheduler in HeterogeneousArchitectureScheduler.

The target release for this is Diablo, however the USC-ISI team intends to have a stable test branch and deployment at Cactus release.

The USC-ISI team has a functional prototype here:

This blueprint is related to the HeterogeneousInstanceTypes blueprint here:

We are also drafting blueprints for other machine types:

An etherpad for discussion of this blueprint is available at http://etherpad.openstack.org/heterogeneoustilerasupport

Release Note

Nova has been extended to allow support for the Tilera TILEmpower Platform (TILEPro64 Processor).

Rationale

See HeterogeneousInstanceTypes.

User stories

Only tp64.8x8 instance type is chosen with 16GB of memory and 64 cores.

To make and register tilera_vmlinux kernel image,

$ euca-bundle-image -i tilera_vmlinux --kernel true
$ euca-upload-bundle -b kernel-bucket -m /tmp/tilera_vmlinux.manifest.xml
$ euca-register kernel-bucket/tilera_vmlinux.manifest.xml

To make and register tilera_fs file system image,

$ euca-bundle-image -i tilera_fs --kernel tilera_vmlinux
$ euca-upload-bundle -b image-bucket -m /tmp/tilera_fs.manifest.xml
$ euca-register image-bucket/tilera_fs.manifest.xml

To verify the registered tilera kernel and file system,

$ euca-describe-images
IMAGE   aki-71046db2    kernel-bucket/tilera_vmlinux.manifest.xml admin   available       private         tilera  kernel
IMAGE   ami-5c86a016    image-bucket/tilera_fs.manifest.xml     admin available       private         tilepro64       machine aki-71046db2

To edit/check the tilera image (ex. ami-5c86a016),

$ cat /home/images/5c86a016/info.json 
{"status": "active", "location": "/home/images/5c86a016/image", "is_public": true, "container_format": "ami", "disk_format": "ami", "properties": {"image_location": "image-bucket/tilera_fs.manifest.xml", "image_state": "available", "kernel_id": 1896115634, "architecture": "tilepro64", "is_public": false, "type": "machine", "owner_id": "admin"}, "id": 1552326678}

To run tilera instances,

$ euca-add-keypair $key > $key.pem
$ chmod 600 $key.pem
$ euca-run-instances -k $key -t tp64.8x8 ami-5c86a016

To access tilera instances,

$ ssh -i key.pem -p 52108 $Given_Tilera_IP_address


Assumptions

This blueprint is dependent on tp64.8x8 being a selectable instance type and that the scheduler knows this instance must get routed to a TILEmpower board. See HeterogeneousArchitectureScheduler.

Design

See HeterogeneousInstanceTypes. The tilera machine is unique in that it has a NxN mesh of processor cores. Currently there is no support for KVM or XEN virtualization, so we are doing a bare metal approach.

Schema Changes

See HeterogeneousInstanceTypes.

We're proposing the following default values added to the instance_types table.


   't64.8x8':  dict(memory_mb=16384, vcpus=1, local_gb=500,
                    flavorid=301,
                    cpu_arch="tile64",
                    cpu_info='{"geometry":"8x8"}'),
   'tp64.8x8': dict(memory_mb=16384, vcpus=1, local_gb=500,
                    flavorid=302,
                    cpu_arch="tilepro64",
                    cpu_info='{"geometry":"8x8"}'),
   'tgx.4x4':  dict(memory_mb=16384, vcpus=1, local_gb=500,
                    flavorid=303,
                    cpu_arch="tile-gx16",
                    cpu_info='{"geometry":"4x4"}'),
   'tgx.6x6':  dict(memory_mb=16384, vcpus=1, local_gb=500,
                    flavorid=304,
                    cpu_arch="tile-gx36",
                    cpu_info='{"geometry":"6x6"}'),
   'tgx.8x8':  dict(memory_mb=16384, vcpus=1, local_gb=500,
                    flavorid=305,
                    cpu_arch="tile-gx64",
                    cpu_info='{"geometry":"8x8"}'),
   'tgx.10x10':  dict(memory_mb=16384, vcpus=1, local_gb=500,
                      flavorid=306,
                      cpu_arch="tile-gx100",
                      cpu_info='{"geometry":"10x10"}')


Tilera Compute Node's Design

All 10 TILEmpower boards #0~9 (with TILEPro64 processor) are connected to one x86 Tilera Compute Node through USB-to-16 Serial Adapter. The vmlinux file is downloaded from Tilera Compute Node to each TILEmpower board through TFTP which is setup in boot parameters on each board. Once tilera file system is ready on TILEmpower board, Tilera Compute Node doesn't do anything except turning off/on the board. Once tilera instance is running, user can access the TILEmpower board, not Tilera Compute Node, through ssh.

1. TFTP setting for Tilera Compute Node

$ vi /etc/xinetd.d/tftp 
service tftp
{
protocol = udp
port = 69
socket_type = dgram
wait = yes
user = root 
server = /usr/sbin/in.tftpd
server_args = /tftpboot
disable = no
}
$ /etc/init.d/xinetd restart


2. File preparation in Tilera Compute Node:/tftpboot

- Different vmlinux and file system are used for each TILEmpower board.

Copy the following files to /tftpboot:
• vmlinux_0_1, vmlinux_1_1, vmlinux_2_1, vmlinux_3_1, vmlinux_4_1, vmlinux_5_1, vmlinux_6_1, vmlinux_7_1, vmlinux_8_1, vmlinux_9_1
• vmlinux_0_2, vmlinux_1_2, vmlinux_2_2, vmlinux_3_2, vmlinux_4_2, vmlinux_5_2, vmlinux_6_2, vmlinux_7_2, vmlinux_8_2, vmlinux_9_2
• mboot.bootrom
• pdu_mgr
• tilera_boards
• tilera_fs_1G
Copy tilera_fs_1G to tilera_fs_0, ..., tilera_fs_9. At the bash prompt, do:
for i in {0..9}
do
cp /tftpboot/tilera_fs_1G /tftpboot/tilera_fs_$i
done


2.1 Tilera boards information: /tftpboot/tilera_boards

- Tilera Compute Node manges board information (board_id, board_ip_address, board_mac_address, board_hw_description, etc.) using this tilera_boards file.

# board_id  ip_address mac_address 
0            10.0.2.1   00:1A:CA:00:57:90 10 16218 917 476 1 tilera_hv 1 
1            10.0.2.2   00:1A:CA:00:58:98 10 16218 917 476 1 tilera_hv 1       
2            10.0.2.3   00:1A:CA:00:58:50 10 16218 917 476 1 tilera_hv 1      
3            10.0.2.4   00:1A:CA:00:57:A8 10 16218 917 476 1 tilera_hv 1    
4            10.0.2.5   00:1A:CA:00:58:AA 10 16218 917 476 1 tilera_hv 1 
5            10.0.2.6   00:1A:CA:00:58:2C 10 16218 917 476 1 tilera_hv 1 
6            10.0.2.7   00:1A:CA:00:58:5C 10 16218 917 476 1 tilera_hv 1       
7            10.0.2.8   00:1A:CA:00:58:A4 10 16218 917 476 1 tilera_hv 1 
8            10.0.2.9   00:1A:CA:00:58:1A 10 16218 917 476 1 tilera_hv 1 
9            10.0.2.10  00:1A:CA:00:58:38 10 16218 917 476 1 tilera_hv 1  


2.2 pdu_mgr

- PDU(Power Distribute Unit)-controlling EXPECT script for remote turn-on/off/reboot of TILEmpower board

3. Bare metal approach without supporting KVM or XEN virtualization (nova/virt/tilera.py) File:HeterogeneousTileraSupport$Tilera Compute.png

Implementation

The USC-ISI team has a functional prototype: https://code.launchpad.net/~usc-isi/nova/hpc-trunk

UI Changes

The following will be available as new default instance types.

Tilera TILEPro64

  • API name: tp64.8x8
  • TILEPro64 processor: 64 (8x8) cores
  • 16 GB RAM (16384 MB)
  • 1 TB of instance storage
  • http://www.tilera.com/

(Only one Tilera instance type for now. When KVM support appears, we will add additional types to support partitioning into smaller instances)

Code Changes

  • nova/db/sqlalchemy/migrate_repo/versions/013_add_architecture_to_instance_types.py
  - add default instance types for tilera
  • nova/virt/connection.py
  - add tilera connection_type
  • nova/virt/tilera.py
  - new tilera connection code for launching instances on TILEmpower board
  • nova/virt/tilera.xml.template and nova/virt/tilera_interfaces.template
  - new tilera template files
  • MANIFEST.in
  - add tilera related template files

Migration

Very little needs to change in terms of the way deployments will use this if we set sane defaults like "x86_64" as assumed today.

Test/Demo Plan

This need not be added or completed until the specification is nearing beta.

Unresolved issues

One of the challenges we have is that the flavorid field in the instance_types table isn't auto-increment. We've selected high numbers to avoid collisions, but the community should discuss how flavorid behaves and the best approach for adding future new instance types.

BoF agenda and discussion

Use this section to take notes during the BoF; if you keep it in the approved spec, use it for summarising what was discussed and note any options that were rejected.