Smokestack

SmokeStack

A webUI to run integration/unit tests for OpenStack branches.

Source Code

https://github.com/dprince/smokestack

Online at

http://smokestack.openstack.org/

Description

SmokeStack is a Rails 3 application that provides a webUI and REST API to install, configure, and test branches of Openstack.

It currently supports custom branches of Nova, Keystone, and Glance.

Features

• Custom branches: Each test configuration allows you to customize the Nova, Keystone, and Glance branch and provides an option to merge/rebase with trunk.
• Packages: currently Ubuntu upstream packagers to build custom packages for each job
• Config management: currently supports Chef
• Pluggable test suite support: By default Torpedo (Ruby bindings tests for the v1.1 OSAPI) and the Nova smoke tests (EC2 API) are used
• Unit test runners: Runs unit tests for each branch
• Multiple configurations: Libvirt, XenServer 5.6 SP2, MySQL, Postgres, etc.
• A templated job runner: defaults to an openstack_vpc runner which runs tests on the Rackspace Cloud. Can be modified to your liking...
• Resque: a redis backed job queue (thanks GitHub!)

SmokeStack automation

SmokeStack is automated so that every single branch in merge prop for upstream master gets tested.

Here is an example flow of how it works.

1. A command line tool called Bellows queries the OpenStack Gerrit server for the latest branches of nova, glance, and keystone.

2. Bellows creates test configurations in SmokeStack for any new merge proposals. Existing merge proposals are updated with new refspecs. Bellows then schedules jobs for any test configs (merge proposals) that have been modified.

3. A SmokeStack job worker pulls a job from the queue and runs it. A custom shell script is generated and written to disk for each job which contains all the information about custom branches, package installers, etc.

4) Openstack VPC creates a custom cloud server group where the job is executed. Packages are built. Chef is used to install and configure packages. Tests suites like Torpedo and the Nova smoke tests are executed. When the test run is finished results are harvested by SmokeStack and available for consumption by git hash via the API.

5) ReviewDay generates a consolidated report of merge proposals (Gerrit), by priority (launchpad tickets/blueprints), with tests results (SmokeStack).

Creating a new test configuration

A test config for a developer branch in merge prop might look something like this:

Advanced options are provided to specify custom:

• Debian package builder URLs
• Git Hash/Bzr revision: provide a way to test a previous revision of the source code.

NOTE: You usually don't have to do this manually. Bellows automatically configures branches that are in merge prop for you.

Jobs

Each job goes into a Redis backed job queue. Smokestack currently has 10 job workers (which allows for 6 concurrent jobs to be running at any time). 3 workers run Libvirt configurations in the cloud and 3 workers use a hybrid cloud/bare metal server group to test XenServer. There are 4 unit test workers on isolated nodes which are available to run unit tests. Job workers are easily scalable by adding more workers, hardware, and/or Cloud Servers accounts.

Each job makes use of an openstack_vpc project template which is used to automate and control the setup of things in the cloud. The openstack_vpc project is also a handy tool for developers who want to spin up groups of servers to have direct access to a fully working Openstack (Nova/Keystone/Glance) system in the cloud.

Here is what happens each time a job runs:

1) A Cloud Servers VPC server group is created.

2) Packages are built (in the cloud) based on the configured SmokeStack branches(Nova/Keystone/Glance). An API repo is created which is used to deploy these packages.

3) Chef is installed and each node configured according to the selected 'configuration' (see below).

4) Tests are executed: first Torpedo, then the nova smoketests (the job template controls this ordering).

5) Results, logs, bzr/git revisions are harvested and stored in the SmokeStack DB.

Configurations

SmokeStack 'configurations' are groups of settings that define how each VPC group cloud is configured. Kind of like Amazon's "Cloud Formations" for functional testing. The following configuration options are available:

1) The type and size of Cloud Servers used to build the server group

2) The Chef cookbook repo used

3) The Chef roles and recipes that should be used for each node.

SmokeStack currently contains three configurations: Libvirt MySQL, Libvirt Postgres, and XenServer 5.6 SP2 w/ MySQL.

In the works

• Run SmokeStack on stable OpenStack release commits (Diablo, etc.)
• Lots of UI stuff (see the GitHub issues list)
• Future configurations might support testing different nova configurations (zones), glance caching, larger groups of servers, etc.
• Integrate with something like Dell's crowbar project to provide a clean/flexible UI for testing on bare metal.
• Additional hypervisor support.