Fuel CI Overview
Fuel project uses the OpenStack Gerrit infrastructure and follows its Development workflow.
Additionally to that, there is a Fuel CI -- third-party CI service which runs additional checks and tests which are not yet supported in OpenStack Infra gate.
Fuel CI consists of one Jenkins master node, which is connected to Openstack Gerrit via Gerrit Trigger plugin using service account Fuel CI. There is also about 30 bare-metal Jenkins slaves. All Jenkins slaves are running Ubuntu 14.04 and perform unit tests and deployment tests via fuel-devops/fuel-qa framework.
Generic deployment test
To run deployment for Fuel you need:
- ISO image for Fuel node 
- fuel-devops  - the cli tool which manages virtual machines and stores state (vm names, network interfaces..) in a PostgreSQL database
- fuel-qa  - test framework based on proboscis
Basic setup is described in .
And the test flow is as follows:
- with fuel-devops tool:
- create several vm's connected via internal network - a so-called devops environment
- with fuel-qa framework:
Step 1. install Fuel node on first vm using the ISO image provided by the local path on the host server Step 2. bootstrap other vms with basic OS image provided on Fuel ISO Step 3. configure Fuel environment via API according to certain scenario Step 4. run deployment
Test scenarios are described in fuel-qa documentation, see for example .
Deployment test on CI
fuel-library code is essentially a set of puppet manifests which are used to deploy the enviroment configuration defined via Fuel interface. These manifests are delivered to Fuel node as RPM package .
To save time and resources on CI we don't recreate environment from scratch for every tests but regularly take a "stable enough" ISO, upload it to Jenkins slaves, create base environment (steps 1. and 2.) and snapshot all its vms.
Then, on every commit we
- rebuild a fuel-library package in a CentOS-based docker container
- revert devops environment from snapshot
- upload and install package on Fuel node
- run the deployment test scenario (steps 3. and 4.)
You can refer to detailed logs in 
-  nightly ISO builds
-  fuel-devops
-  fuel-qa
-  test environment setup
-  test scenario
-  fuel-library spec-fiel
-  test run example
-  YAML-based templates for devops environments
Fuel Library repo
The heaviest tests are those for fuel-library changes. Each commit to fuel-library repo run two deployments with three virtual machines each. These jobs use preconfigured environment (master node is already deployed and three slaves enabled) and don't do clean start for every run, which may affect the result.