Jump to: navigation, search


< Mistral
Revision as of 14:09, 10 February 2015 by Akuznetsova (talk | contribs) (Rally gate for Mistral:)

Different types of tests

On Mistral project we have two separate test suites:

  • Unit tests - executed by Jenkins CI job in OpenStack gerrit (python-style checks and execution of all unit tests)
  • Integration tests - executed by Devstack Gate job in OpenStack gerrit (integration tests for Mistral after the OpenStack deployment with devstack)

Where we can find automated tests



How to execute tests manually

Almost all existing automated tests can be executed manually on the developer's desktop (except those which check OpenStack actions). To do this you should clone mistral repository (or python-mistralclient) and perform the following shell commands:

git clone https://github.com/stackforge/mistral
cd mistral

Unit tests

To run unit tests suite


To run unit tests suite against a specific python version

tox -e py27

To run tests from a specific test class (using a specific python version)

tox -e py27 -- 'DataFlowEngineTest'

Integration tests

There are several test suites of intergation tests in both repositories:


  • mistral/tests/functional/api/v1/test_mistral_basic.py - contains tests which check Mistral API v1 (workbooks, executions, tasks endpoints)
  • mistral/tests/functional/api/v1/test_workflow_execution.py - contains tests which check execution of standard scenarios, task results and dependencies in v1
  • mistral/tests/functional/api/v2/test_mistral_basic.py - contains tests which check Mistral API v2 (workbooks, workflows, executions, tasks, actions and cron triggers endpoints)
  • mistral/tests/functional/engine/actions - contains tests which check Mistral integration with OpenStack components (Nova, Glance, Keystone)


  • mistralclient/tests/functional/cli/ - contains test suites for v1 and v2 which check interaction with Mistral using CLI
  • mistralclient/tests/functional/client/ - contains test suites which check integration and interaction of Mistral client and API

To run integration test suite:

  • in OpenStack mode (when auth in Mistral is enabled and Mistral integrates with OpenStack components)
pip install git+http://github.com/openstack/tempest.git
nosetests <mistral or mistralclient>/tests/functional
  • in Non-OpenStack mode:
set auth_enable=false in the mistral.conf
restart Mistral server
execute: ./run_functional_tests

Test Plans

Discussion of the general direction of testing can be found here: https://etherpad.openstack.org/p/MistralTests

More detailed test plan will be available later.

Load and Performance Testing

Mistral server consists of three main components api, engine and executor, which can be run in one process or every component can be run in its own one. In a production environment (OpenStack cloud or not) it is more convenient and natural next situation. A lot of users with their Mistral client and connect to Mistral server, there a few variants are possible. First variant is when there is one Mistral server with 1 api, 1 engine and 1 executor, it is not a good decision for a big environment and complex Mistral scenarios, but why not? Another one is when there are a few instances of api servers, engines and executors. Mistral has to work in both situations and we need to know what is its limit.

Rally is a tool that will help us to make a stress and performance Mistral testing.

Load/performance test plan:

  • Prepare simple Rally scenarios which will execute main Mistral actions: get list of workbooks, create and delete workbook, workflow, create execution and wait for its success.

Status: Done

  • Measure the time of one request (time of getting the list of objects, time of creating/deleting different objects, time of the simplest workflow execution) to have the initial value for the next experiments.

Status: Good progress

  • Make a series of measurements for all mentioned above scenarios increasing number of requests (1, 10, 100, 1000, 10000 times for 1 tenant)

Status: Good progress

  • Make the same steps as in 3) but now with increasing number of tenants and accordingly use the concurrency.

Status: Not started

  • Next step is to get time of workflow execution increasing number of Mistral engines and executors and to analyze what is going on with time.

Status: Not started

  • Prepare complex workflow and check Mistral performance in conditions close to reality.

Status: Not started


Results will be here: https://etherpad.openstack.org/p/mistral-rally-testing-results

(if this format is not very suitable then results can be published in other way)

Rally gate for Mistral:

Now we have special gate in the 'mistral' repository that runs Mistral Rally scenarios against OpenStack deployed by DevStack with installed Rally & Mistral.