Openstack-integration-test-suites

Integration Test Suites

Kong

Black Box

Uses httplib2.

Uses a config file (INI-style) to tell the test runner about the environment it is running against.

Capable of testing Swift + Nova + Glance + Keystone

Uses unittest2 and nose wrapper for testing framework.

Simple shell script wrapper (run_tests.sh) around its Python-based test runner (run_tests.py). Similar to the runners for Glance/Nova.

Uses "tagging" (a unittest2/nose feature) to mark individual test cases as belonging to a group of tests (nova, glance, rabbitmq, etc)

Tests are NOT self-contained. They have side-effects and each test case is numbered (like 001_test_auth) to ensure that tests are run in an appropriate order.

Zodiac

Black Box

NOTE: Last patch to https://github.com/rohit-k/zodiac was 9/9/2011

Uses httplib2. Has directories for Nova and Swift.

Torpedo

"Fire when ready. Fast ruby integration tests for OpenStack"

https://github.com/dprince/torpedo

• Features:
  • Uses Ruby bindings (openstack-compute).
  • Smoke Testing (deep system feature/integration tests which run reasonably fast)
  • Command line tool built w/ a Thor wrapper around test-unit tests
  • Focus on the Openstack Compute API (Nova)
  • Release based (install via Gem)

First line of defense for SmokeStack... built specifically to hit upstream branches and blow up when something breaks.

Backfire

NOTE: last commit to https://github.com/ohthree/backfire was 9/9/2011

Uses DTest to fire tests in multiple eventlet threads.

Uses novaclient to execute commands against the OpenStack API

Uses a BaseIntegrationTest class that houses a number of methods that fire novaclient commands. The test cases themselves verify the results from the novaclient calls.

Tests ARE self-contained. Because of the use of DTest, any test case can be run simultaneously which means test cases clean up after themselves.

There are a couple of simplistic stress tests included (in /stress directory) that count how many

Some decent documentation on how to run tests and, importantly, how to create/add new tests.

Lettuce

This is a Behavior Driven Development tool for Python, having support for functional, integration and unit tests. Has CLI and Web UI modes. Uses the concepts features(behavior of a project), scenarios(describe the feature), and steps for creating test cases in a natural language.

Reddwarf Integration Tests

The Reddwarf Project (Database as a service) has a suite of integration and functional tests which are executed constantly by the developers and in CI. These tests model complex usage scenarios and run against a fully stood up REST API using the python-novaclient. They utilize the actual Nova codebase to execute code a normal user never could for the sake of adding additional assertions. For example they check the state of the database following certain calls to make sure things are being updated (they don't do anything the user couldn't do to make the tests pass).

Vagrant along with some shell scripts are used to set up and configure Nova with Reddwarf. However the tests use a config file and can also run against real-hardware.

The test code uses Proboscis, a test framework which sits on top of Nose and adds features from TestNG, such as advanced grouping (similar to Nose's "tagging"), ordering of tests by adding dependencies (which means test names do *NOT* have to be numbered or otherwise named strangely and the module they live in has no effect on the order they run). Proboscis is backwards compatable with unittest / Nose and can run tests extending unittest.TestCase the same way they'd usually run. However tests which do not extend unittest.TestCase run in style of TestNG, where the same instance of the test class is passed between all the inner test methods of that class. This allows you to avoid using global variables to stich complex test scenarios together.

Certain features which in other frameworks would be done through a shell script or non-test setup or fixture code are first-class tests in Proboscis. An example is that the tests start up the service daemons and asserts they are responding. This is done in a test group. If it fails the majority of tests, which depend on this group, are merely skipped, making the real error easy to indentify.

Additionally the tests are driven by a config file and can be run against various deployments. For example, before the Vagrant approach was discovered the tests ran against real-hardware. However by utilizing Vagrant every developer on the team can run the tests whenver they need to.

StackTester

NOTE: Last commit to https://github.com/rackspace-titan/stacktester was 9/7/2011

Black Box

Uses httplib2.

Uses a config file (INI-style) to tell the test runner about the environment it is running against.

Capable of testing Swift + Nova + Glance + Keystone

Uses unittest2 and nose wrapper for testing framework.

Python test runner (bin/stacktester).

Tests are self contained.