Jump to: navigation, search

Neutron/FunctionalGateSetup

< Neutron
Revision as of 21:45, 3 February 2015 by Paul Michali (talk | contribs) (Add Hooks and Test)

Setting Up Functional Gate

This wiki contains notes on how to setup functional test gate for a repository. Using the neutron-vpnaas repo as an example.

Preparation

Clone the project-config repo, so that a commit can be formed for changes to support the repo.

    git clone https://github.com/openstack-infra/project-config

Setup Non-voting Experimental Test Runs

In the openstack-infra project-config repo, modify the zuul/ayout.yaml for this project. First, there should be a project entry created, under ‘projects:’. In VPNaaS, there was already entry:

   projects:
   ...
     - name: openstack/neutron-vpnaas
       template:
         - name: check-requirements
         - name: integrated-gate-neutron
         - name: merge-check
         - name: python-jobs
         - name: openstack-server-release-jobs

With this project, there are only the templates, and no check, gate, post, or experimental sections. For now, we’ll add a new test, under the experimental section, so that the test will only run when the user adds a Gerrit comment with “check experimental"

   experimental:
     - check-neutron-vpnaas-dsvm-functional

In the “jobs:” section, we need to mark this test as non-voting: for the check queue:

   jobs:
   ...
     - name: check-neutron-vpnaas-dsvm-functional
       voting: false

Next, a job must be created for the VPNaaS repo in jenkins/jobs/. To speed things up, and because the needs are similar, the neutron-functional.yaml is copied to neutron-vpnaas-functional.yaml. The changes are made to the name, which is at the top of the file:

   - job-template:
       name: '{pipeline}-neutron-vpnaas-dsvm-functional{branch-designator}'

and change the path for the gate_hook.sh and post_test_hook.sh, from neutron/neutron/ to neutron-vpnaas/neutron_vpnaas/. Since the VPN repo only has master branch, and not older branches that have hooks in different paths, the conditional logic for the hook scripts is also removed. The whole file looks like this (in case you don’t have access to the Neutron YAML file):

   - job-template:
       name: '{pipeline}-neutron-vpnaas-dsvm-functional{branch-designator}'
       node: '{node}'
   
       wrappers:
         - build-timeout:
             timeout: 125
         - timestamps
   
       builders:
         - link-logs
         - net-info
         - devstack-checkout
         - shell: |
             #!/bin/bash -xe
             export PYTHONUNBUFFERED=true
             export DEVSTACK_GATE_UNSTACK=1
             export DEVSTACK_GATE_TIMEOUT=120
             export DEVSTACK_GATE_TEMPEST=0
             export DEVSTACK_GATE_EXERCISES=0
             export DEVSTACK_GATE_NEUTRON=1
             export DEVSTACK_GATE_INSTALL_TESTONLY=1
             export BRANCH_OVERRIDE={branch-override}
             if [ "$BRANCH_OVERRIDE" != "default" ] ; then
                 export OVERRIDE_ZUUL_BRANCH=$BRANCH_OVERRIDE
             fi
   
             function gate_hook Template:Bash -xe $BASE/new/neutron-vpnaas/neutron vpnaas/tests/contrib/gate hook.sh dsvm-functional
             export -f gate_hook
   
             function post_test_hook Template:Bash -xe $BASE/new/neutron-vpnaas/neutron vpnaas/tests/contrib/post test hook.sh dsvm-functional
             export -f post_test_hook
   
             cp devstack-gate/devstack-vm-gate-wrap.sh ./safe-devstack-vm-gate-wrap.sh
             ./safe-devstack-vm-gate-wrap.sh
   
       publishers:
         - test-results
         - devstack-logs
         - console-log

A quick check was made in openstack-infra/devstack-gate/features.yaml to make sure that the q-vpn service was started up for neutron. It was, no nothing was needed there.

Pushed up for review. Did get failure of gate-project-config-layout test. Log was truncated, so didn’t know what the error was and had to ask infra folks. They pointed me to console log to see the error (can go to https://jenkins06.openstack.org/job/gate-project-config-laylout/. There is not a check-neutron-vpnaas-dsvm-functional defined. Needed to add to project.yaml in jenkins/jobs/. Under the neutron-vpnaas project, I added job definitions:

   - project:
       name: neutron-vpnaas
       node: 'bare-precise || bare-trusty'
       tarball-site: tarballs.openstack.org
       doc-publisher-site: docs.openstack.org
   
       jobs:
         - python-jobs
         - openstack-publish-jobs
         - '{pipeline}-neutron-vpnaas-dsvm-functional{branch-designator}':
             pipeline: gate
             node: 'devstack-precise || devstack-trusty'
             branch-designator: 
             branch-override: default
         - '{pipeline}-neutron-vpnaas-dsvm-functional{branch-designator}':
             pipeline: check
             node: 'devstack-precise || devstack-trusty'
             branch-designator: 
             branch-override: default

Made sure that Jenkins passes for the review and get approval from Infra folks. This goes under openstack-infra review obviously.

Ref: Review 148616

Quick Test On Local Host (optional)

In a neutron-vpnaas view, make sure the development environment is setup for tox (e.g. “tox -r -e py27 —notest”). Then, a functional.filter can be added to /etc/neutron/rootwrap.d/ with perms 644 and containing:

   # neutron-rootwrap command filters to support functional testing.  It
   # is NOT intended to be used outside of a test environment.
   #
   # This file should be owned by (and only-writeable by) the root user
   
   [Filters]
   # '$BASE_PATH' is intended to be replaced with the expected tox path
   # (e.g. /opt/stack/new/neutron/.tox/dsvm-functional) by the neutron
   # functional jenkins job.  This ensures that tests can kill the
   # processes that they launch with their containing tox environment's
   # python.
   kill_tox_python: KillFilter, root, /opt/stack/neutron-vpnaas/.tox/dsvm-functional/bin/python, -9
   
   # enable ping from namespace
   ping_filter: CommandFilter, ping, root
   
   # enable curl from namespace
   curl_filter: CommandFilter, curl, root

Ran ‘tox -e dsvm-functional’ on a repo with a functional test added. This worked, however, I did get a warning about test command found, but not installed in testenv.

When the project-config changes are used up for review, several tests are run on the changes, and the Jenkins results should be checked for any failures.

Add Hooks and Test

The easiest way here, is to copy the files from neutron/neutron/tests/contrib, into the repo (neutron-vpnaas/neutron_vpnaas/tests/contrib), and tweak them (yeah, cut and paste :). The devstack-vm-gate.sh and functions.sh from Neutron should not be used. The gate_hook.sh should run devstack-vm-date.sh in openstack-infra/devstack-gate/ project area.

For the neutron-vpnaas repo, the following changes were made to the files.

  • Indicated neutron-vpnaas repo in README
  • Update the path from neutron/neutron to neutron-vpnaas/neutron_vpnaas and from new/neutron to new/neutron-vpnaas in gate_hook.sh and post_test_hook.sh
  • Call devstack-vm-gate.sh from gate_hook.sh for both dsvm-functional and api cases (change the API one).

Here is gate_hook.sh (see repo for latest):

   #!/bin/bash
   
   set -ex
   
   CONTRIB_DIR="$BASE/new/neutron-vpnaas/neutron_vpnaas/tests/contrib"
   
   $BASE/new/devstack-gate/devstack-vm-gate.sh
   
   # Add a rootwrap filter to support test-only
   # configuration (e.g. a KillFilter for processes that
   # use the python installed in a tox env).
   FUNC_FILTER=$CONTRIB_DIR/filters.template
   sed -e "s+\$BASE_PATH+$BASE/new/neutron-vpnaas/.tox/dsvm-functional+" \
       $FUNC_FILTER | sudo tee /etc/neutron/rootwrap.d/functional.filters > /dev/null
   
   # Use devstack functions to install mysql and psql servers
   TOP_DIR=$BASE/new/devstack
   source $TOP_DIR/functions
   source $TOP_DIR/lib/config
   source $TOP_DIR/stackrc
   source $TOP_DIR/lib/database
   source $TOP_DIR/localrc
   
   disable_service postgresql
   enable_service mysql
   initialize_database_backends
   install_database
   
   disable_service mysql
   enable_service postgresql
   initialize_database_backends
   install_database
   
   # Set up the 'openstack_citest' user and database in each backend
   tmp_dir=`mktemp -d`
   
   cat << EOF > $tmp_dir/mysql.sql
   CREATE DATABASE openstack_citest;
   CREATE USER 'openstack_citest'@'localhost' IDENTIFIED BY 'openstack_citest';
   CREATE USER 'openstack_citest' IDENTIFIED BY 'openstack_citest';
   GRANT ALL PRIVILEGES ON *.* TO 'openstack_citest'@'localhost';
   GRANT ALL PRIVILEGES ON *.* TO 'openstack_citest';
   FLUSH PRIVILEGES;
   EOF
   /usr/bin/mysql -u root < $tmp_dir/mysql.sql
   
   cat << EOF > $tmp_dir/postgresql.sql
   CREATE USER openstack_citest WITH CREATEDB LOGIN PASSWORD 'openstack_citest';
   CREATE DATABASE openstack_citest WITH OWNER openstack_citest;
   EOF
   # User/group postgres needs to be given access to tmp_dir
   setfacl -m g:postgres:rwx $tmp_dir
   sudo -u postgres /usr/bin/psql --file=$tmp_dir/postgresql.sql

Here is post_test_hook.sh (again, check the repo for the latest):

   #!/bin/bash
   
   set -xe
   
   NEUTRON_DIR="$BASE/new/neutron-vpnaas"
   TEMPEST_DIR="$BASE/new/tempest"
   SCRIPTS_DIR="/usr/local/jenkins/slave_scripts"
   
   function generate_testr_results {
       # Give job user rights to access tox logs
       sudo -H -u $owner chmod o+rw .
       sudo -H -u $owner chmod o+rw -R .testrepository
       if [ -f ".testrepository/0" ] ; then
           .tox/$venv/bin/subunit-1to2 < .testrepository/0 > ./testrepository.subunit
           .tox/$venv/bin/python $SCRIPTS_DIR/subunit2html.py ./testrepository.subunit testr_results.html
           gzip -9 ./testrepository.subunit
           gzip -9 ./testr_results.html
           sudo mv ./*.gz /opt/stack/logs/
       fi
   }
   
   function dsvm_functional_prep_func {
       :
   }
   
   owner=stack
   prep_func="dsvm_functional_prep_func"
   
   # Set owner permissions according to job's requirements.
   cd $NEUTRON_DIR
   sudo chown -R $owner:stack $NEUTRON_DIR
   # Prep the environment according to job's requirements.
   $prep_func
   
   # Run tests
   echo "Running neutron $venv test suite"
   set +e
   sudo -H -u $owner tox -e $venv
   testr_exit_code=$?
   set -e
   
   # Collect and parse results
   generate_testr_results
   exit $testr_exit_code

In addition to the hook support, create a dummy functional test in tests/functional. Push this for review, and from Gerrit, enter in a comment with ‘check experimental’ to run the functional test from the experimental pipeline.

Ref: Review 150990

Move To Check Queue

Once the the experimental queue run works and the hooks commit is merged, the check-neutron-vpnaas-dsvm-functional job can be moved to the check queue (still as non-voting). It will now run on all commits.

This is as simple as changing experimental entry to check for the project in zuul/layout.yaml

   diff --git a/zuul/layout.yaml b/zuul/layout.yaml
   index d587a43..684c035 100755
   --- a/zuul/layout.yaml
   +++ b/zuul/layout.yaml
   @@ -1350,7 +1350,7 @@ projects:
          - name: merge-check
          - name: python-jobs
          - name: openstack-server-release-jobs
   -    experimental:
  +    check:
          - check-neutron-vpnaas-dsvm-functional
   
      - name: openstack/nova

Now, the test will run for every patch reviewed for the repo, but the testing will not be voting.

Ref: Review 152602

Add to Gate and Check Queue With Voting

Once everything looks good, the check-neutron-vpnaas-dsvm-functional job can be made voting and a gate-neutron-vpnaas-dsvm-functional job can be added to the gate pipeline.

For this, zulu/layout.yaml is again modified to remove the lines that make the check non-voting, and add a gate entry for the project.

TODO: Code...