Fuel/CI

Fuel CI Overview
Fuel project uses the OpenStack Gerrit infrastructure and follows its Development workflow.

Additionally to that, there is a Fuel CI -- third-party CI service which runs additional checks and tests which are not yet supported in OpenStack Infra gate.

Fuel CI consists of one Jenkins master node, which is connected to Openstack Gerrit via Gerrit Trigger plugin using service account Fuel CI. There is also about 30 bare-metal Jenkins slaves. All Jenkins slaves are running Ubuntu 14.04 and perform unit tests and deployment tests via fuel-devops/fuel-qa framework.

Links

 * Jenkins Job Builder configuration
 * Puppet manifests
 * Jenkins instance

Generic deployment test
To run deployment for Fuel you need:
 * ISO image for Fuel node [1]
 * fuel-devops [2] - the cli tool which manages virtual machines and stores state (vm names, network interfaces..) in a PostgreSQL database
 * fuel-qa [3] - test framework based on proboscis

Basic setup is described in [4].

And the test flow is as follows: Step 1. install Fuel node on first vm using the ISO image provided by the local path on the host server Step 2. bootstrap other vms with basic OS image provided on Fuel ISO Step 3. configure Fuel environment via API according to certain scenario Step 4. run deployment
 * with fuel-devops tool:
 * create several vm's connected via internal network - a so-called devops environment
 * with fuel-qa framework:

Test scenarios are described in fuel-qa documentation, see for example [5].

Deployment test on CI
fuel-library code is essentially a set of puppet manifests which are used to deploy the enviroment configuration defined via Fuel interface. These manifests are delivered to Fuel node as RPM package [6].

To save time and resources on CI we don't recreate environment from scratch for every tests but regularly take a "stable enough" ISO, upload it to Jenkins slaves, create base environment (steps 1. and 2.) and snapshot all its vms.

Then, on every commit we


 * 1) build a fuel-library package in a CentOS-based docker container using mock
 * 2) build deb package in Ubuntu-based container via sbuild
 * 3) revert devops environment from snapshot
 * 4) upload and install package on Fuel node
 * 5) run the deployment test scenario (steps 3. and 4.)

You can refer to detailed logs in [7]


 * [1] nightly ISO builds
 * [2] fuel-devops
 * [3] fuel-qa
 * [4] test environment setup
 * [5] test scenario
 * [6] fuel-library spec-file
 * [7] test run example
 * [8] YAML-based templates for devops environments

CI for Puppet OpenStack
Fuel CI acts as a third-party CI for a set of Puppet OpenStack project modules used in fuel-library:
 * puppet-aodh
 * puppet-ceilometer
 * puppet-cinder
 * puppet-glance
 * puppet-ironic
 * puppet-heat
 * puppet-horizon
 * puppet-keystone
 * puppet-murano
 * puppet-neutron
 * puppet-nova
 * puppet-openstacklib
 * puppet-sahara
 * puppet-swift

For every commit to these projects we start a set of builds which currently includes fuel-library noop tests and 2 deployment tests. Deployment tests share the same workflow and test scenarios with the deployment tests we're running for fuel-library. See [|design specification] for details.

To prevent Puppet OpenStack changes from introducing regressions into Fuel CI, and to assist Puppet OpenStack developers with investigating Fuel CI failures, Fuel team commits to monitoring and investigating failures of Fuel CI for PuppetOpenStack.

For any questions regarding Fuel CI for Puppet OpenStack please contact #fuel-dev IRC channel.

Packaging CI Overview
Packaging CI for Fuel builds RPM-packages for all Fuel repositories. Packages are built and published using Perestroika.

Packaging CI consists of one Jenkins master node packaging-ci.fuel-infra.org, which is connected to Zuul instance gate.fuel-infra.org via Gearman plugin. Zuul is attached to Openstack Gerrit using service account Fuel Packaging CI.

There are also several Jenkins slaves running Ubuntu 14.04 (LTS):
 * one publisher and mirror host (VM)
 * package building slave(s) (HW)
 * slave(s) for install tests (HW)
 * slave(s) for deployment (system) tests (HW)

Built packages are available at packages.fuel-infra.org site.

Packaging CI Links

 * Packaging CI Spec
 * Jenkins Job Builder configuration
 * Zuul layout
 * Puppet manifests
 * Package repository for Ubuntu containing Zuul package and its dependencies
 * packaging-ci.fuel-infra.org - Jenkins
 * gate.fuel-infra.org - Zuul
 * packages.fuel-infra.org - packages repository