Jump to: navigation, search


OpenStack/SmartOS Compute

This is a work-in-progress page about OpenStack and SmartOS.

Getting a Development Environment Going for SmartStack

TODO: Put this under control of Chef/Puppet

  • dev: need a vagrant smartos box - use veewee
  • prod: DHCP, PXE boot, foreman, razor

Current assumption for now is that there is a controller node running with all necessary services (like keystone). This could be a devstack environment. It is also possible to get these service running in separate zones on SmartOS btw. I currently use a devstack (Grizzly) environment on the controller node. The IPs which interconnect the two VMs are (SmartOS) and

I will not go into detail on how to run devstack on the controller node - it's straight forward.

The basics

Getting SmartOS

Check out:

If you use the script above, modify the SmartOS VirtualBox VM to have two NICs configured, one with Internet access (NAT) the other one host-only.


Then get `pkgin` running in the global zone:

   $ cd /
   $ curl -k http://pkgsrc.joyent.com/packages/SmartOS/bootstrap/bootstrap-2013Q1-x86_64.tar.gz | gzcat | tar -xf -
   $ pkg_admin rebuild
   $ pkgin -y up

Installed some packages which are needed or come in handy (mc, vim) during the next steps:

   $ pkgin install mc vim scmgit python2.7 py27-pip py27-expat py27-sqlite2 py27-mysqldb gcc47 gnu-binutils libxslt

Get the code & install the dependencies needed:

   $ mkdir -p /zones/workspace
   $ cd /zones/workspace
   $ git clone git://github.com/tmetsch/nova.git -b smartos/grizzly
   $ cd nova/tools
   $ export PATH=/opt/local/gcc47/bin/:$PATH
   $ export CFLAGS="-D_XPG6 -std=c99"
   $ pip install -r pip-requires

Configuring the SmartOS Compute Node

Edit sample `./etc/nova/nova.conf` file:

   # network_manager=nova.network.manager.FlatDHCPManager

Create a symlink `/opt/local/bin/python` to `/opt/local/bin/python2.7`

   $ ln -s /opt/local/bin/python2.7 /opt/local/bin/python

Run `nova-compute`:

   $ export LD_LIBRARY_PATH=/opt/local/lib
   $ bin/nova-compute --config-file etc/nova/nova.conf

Run `nova-network`:

   $ export LD_LIBRARY_PATH=/opt/local/lib
   $ bin/nova-network --config-file etc/nova/nova.conf

At this point you will have the set of services necessary to begin provisioning VMs.

Starting a SmartOS image

Get an image (make sure it is f9e4be48-9466-11e1-bc41-9f993f5dff36 for now - smartos64 v1.6.3 - See this line):

   $ imgadm import f9e4be48-9466-11e1-bc41-9f993f5dff36
   $ zfs snapshot zones/f9e4be48-9466-11e1-bc41-9f993f5dff36@now
   $ zfs send zones/f9e4be48-9466-11e1-bc41-9f993f5dff36@now > /zones/workspace/smartos.img

Now on the controller node - register it with glance:

   $ cd /zones/workspace
   $ glance image-create --name 'smartos' --is-public 'true' --container-format 'bare' --disk-format 'raw' --property 'zone=true' < smartos.img"

Boot a instance:

   $ nova boot --flavor=m1.nano --image=smartos testserver --availability-zone nova:08-00-27-ee-14-2a

Note: It might be needed to change to policies for the user you are using to start the instance - must allow for forced_hosts.

Check instance:

   $ zoneadm list -cv
   ID NAME             STATUS     PATH                           BRAND    IP
   0 global           running    /                              liveimg  shared
   1 e99947b5-3578-4f4b-b102-d6ef5706b173 running    /zones/e99947b5-3578-4f4b-b102-d6ef5706b173 joyent   excl

Coding against it with PyCharm's remote interpreter option

PyCharm offers a nice remote interpreter features. To set it up do:

  • Configure Remote Interpreter: Settings > Project Interpreters > Python Interpreters > Add (Remote)
  • Configure Deployment Settings: Settings > Deployment > Add
    • Do not forget to set proper path mappings
    • Include .git in the deployment options tab
  • Right click in the project tree > Upload to **your name here**
  • Create a new Python run configuration with the Remote Interpreter
    • Do not forget to set the '--config-file' script parameter
    • Alter the Path environment and add: /smartdc/bin

Now you can run nova-compute and nova-network from you IDE on the SmartOS node.

Some helpers

Getting the Code

The code is currently not in the mainline code base. The latest version is available at:

Once the code is mature enough it should be easy to integrate the code into the main.

Setting up an environment

There is also setup script that you can use to create an environment; similar to devstack but focussed on SmartOS: https://github.com/hvolkmer/openstack-smartos

This is a work in progess. YMMV. Additions welcome.

This Metsch also has a setup script.

Running the metadata service

1. Add these options to compute.conf: metadata_listen= and metadata_listen_port=80 2. Configure metadata interface like this:

dladm create-vnic -l e1000g0 meta0
ifconfig meta0 plumb
ifconfig meta0
ifconfig meta0 up

3. Then start /openstack/nova/bin/nova-api-metadata --config-file=/openstack/cfg/compute.conf in the global (compute) zone

4. When booting VMs I currently force the default route IP to be the IP of the compute zone, so that the metadata IP is reachable

Install Newrelic

   $ pkgin install nrsysmond
   $ vi /opt/local/etc/nrsysmond.cfg
   $ svcadm enable nrsysmond:default

Install Puppet

   $ pkgin in ruby19-puppet


   $ pkgin in ruby18-puppet

Install Chef

   $ pkgin in ruby19
   $ gem install chef

Next development steps

As Basic VM handling is working now the following dev steps should happen:

  • add unit tests to fixate the current VM start/stop behaviour for SmartOS
  • add networking manager based on the VLANManager but integrate with vmadm/crossbow instead of the Linux way