Jump to: navigation, search



A webUI to run integration tests for OpenStack branches.

Source Code


Online at



SmokeStack is a Rails 3 application that provides a webUI and REST API to install, configure, and test branches of Openstack. It specializes in using packages (currently Centos and Fedora) with config management (currently Puppet). The system has been in use and running against upstream branches since Cactus.

It currently supports custom branches of Nova, Keystone, Glance, Swift, Cinder, Neutron, and Ceilometer. Additionally SmokeStack is well suited to testing/reviewing configuration management branches and is currently tracking/reviewing the stackforge/puppet modules as well for each core project that it supports.


  • Custom branches: Each test configuration allows you to customize the branches and provides an option to merge/rebase with trunk.
  • Packages: currently uses Fedora (maintained by dprince)
  • Config management: currently supports Puppet (stackforge)
  • Pluggable test suite support: By default Torpedo (uses Fog)
  • Unit test runners: Runs unit tests for each branch
  • Multiple configurations: Libvirt, XenServer 6.1, MySQL, Postgres, MongoDB, etc.
  • A templated job runner: defaults to a puppet runner which runs tests on server groups configured with puppet. Can be customized to your liking.
  • Resque: a redis backed job queue

SmokeStack automation

SmokeStack is automated so that every single branch in merge prop for upstream master gets tested.

1. A command line tool called Bellows queries the OpenStack Gerrit server for the latest branches of nova, glance, and keystone.

2. Bellows creates test configurations in SmokeStack for any new merge proposals. Existing merge proposals are updated with new refspecs. Bellows then schedules jobs for any test configs (merge proposals) that have been modified.

3. A SmokeStack job worker pulls a job from the queue and runs it. A custom shell script is generated and written to disk for each job which contains all the information about custom branches, package installers, etc.

4) FireStack creates a custom cloud server group where the job is executed. Packages are built. Puppet is used to install and configure packages. Tests suites like Torpedo and the Nova smoke tests are executed. When the test run is finished results are harvested by SmokeStack and available for consumption by git hash via the API.

5) ReviewDay generates a consolidated report of merge proposals (Gerrit), by priority (launchpad tickets/blueprints), with tests results (SmokeStack).


Each job goes into a Redis backed job queue. Smokestack currently has 10 job workers (which allows for 6 concurrent jobs to be running at any time). 3 workers run Libvirt configurations in the cloud and 3 workers use a hybrid cloud/bare metal server group to test XenServer. There are 4 unit test workers on isolated nodes which are available to run unit tests. Job workers are easily scalable by adding more workers, hardware, and/or OpenStack accounts.

Each job makes use of a FireStack project template which is used to automate and control the setup of things in the cloud. The FireStack project is also a handy tool for developers who want to spin up groups of servers to have direct access to a fully working Openstack (Nova/Keystone/Glance/Cinder/Swift/Quantum) system which uses packages.

Sample Job run:

1 ) A "server group" is created. This is done by FireStack/Kytoon and could be a libvirt/openstack/or XenServer based test group.

2) Packages are built (within the group) based on the configured SmokeStack branches(Nova/Keystone/Glance). An API repo is created which is used to deploy these packages.

3) Puppet is installed and each node configured according to the selected 'configuration' (see below).

4) Tests are executed: first Torpedo, then the nova smoketests (the job template controls this ordering).

5) Results, logs, git revisions are harvested and stored in the SmokeStack DB.

Other links

Fedora upstream packages built by SmokeStack