Difference between revisions of "XenServer/XenServer CI/AdminTips"

Revision as of 12:22, 30 September 2014

Status

http://f0ab4dc4366795c303a0-8fd069087bab3f263c7f9ddd524fce42.r22.cf1.rackcdn.com/ci_status/results.html

Main box

This is the instance which orchestrates the execution of tests.

• Authentication via ssh keys. For access, join the XenAPI team meeting on IRC.
• All the components are deployed to this instance.
• 166.78.153.192
• created via jenkins job at Citrix

openstack-citrix-ci

• source code: https://github.com/citrix-openstack/openstack-citrix-ci
• branch: master
• cloned to: /root/src/openstack-citrix-ci/

• to update:
  • Stop all its services

cd /root/src/openstack-citrix-ci
git pull
pip install -e .

• • Start the required services

• configuration file: /root/osci.config

service: citrix-ci-gerritwatch

This service watches the gerrit stream and adds jobs to the queue

• Logs: /var/log/citrix-ci-gerritwatch.log
• (Re)start: (re)start citrix-ci-gerritwatch

service: citrix-ci

This service progresses jobs through the lifecycle (see below)

• Logs: /var/log/citrix-ci.log
• (Re)start: (re)start citrix-ci

• three threads:
  • main (run through state machine, see below)
  • collect results (long blocking process, so its split out so main stuff makes progress -- service net?)
  • delete node (go to collected, delete is a pain, tries up to to 10 mins, need to fix the delete process!!!!!)

Utilities

• osci-check-connection
  • checks to node, to xenserver, then logs uploaded to swift from both
• osci-manage
  • what citrix-ci service runs
  • runs main lifecycle
  • can also manually add a job to the queue
• osci-upload
  • upload logs from local host directory up to swift
  • called from osci-manage (via python not cli)
• osci-watch-gerrit
  • what citrix-ci-gerrit-watch runs
  • reads gerrit stream and adds jobs to DB
• osci-create-dbschema
  • creates job DB
• osci-run-tests
  • e.g. osci-run-tests print user host ref
  • calls out to host to run tests
  • used (via python not cli) by osci-manage
• osci-view
  • prints out the job DB in useful ways

To report status:

• osci-view is called and content uploaded to swift
• /src/openstack-citrix-ci/upload-ci-status

nodepool

• source code: https://github.com/citrix-openstack/nodepool
• branch: master
• cloned to: /root/src/nodepool
• configuration file: /etc/nodepool/nodepool.yaml

service: nodepool

Provisions VMs to use in the tests

• Logs: /var/log/nodepool/nodepool.log, /var/log/nodepool/debug.log
• (Re)start: killall nodepool; rm /var/run/nodepool/nodepool.pid; start nodepool

Utilities

This could be used to get information/control the nodepool service.

• To list the images:

nodepool image-list

Useful commands

• osci-view list: Gives current queue, what is running etc. Shouldn't have jobs in here that are 'older' than 2 hours unless they are 'Finished'.
• nodepool list: Gives a list of the currently available nodes. Should have some nodes that are 'Ready' or 'Building'
• eval `ssh-agent`; ssh-add ~/.ssh/citrix_gerrit; osci-manage -c 12345/1; ssh-agent -k: Queue job 12345, patchset 1

VM lifecycle

• Queued -> Running: citrix-ci job has got a new node from nodepool (nodepool list will show it as 'held') does osci-run-tests to hand over to xenapi-os-testing
• Running -> Collecting: Job has finished; citrix-ci has changed state to Collecting - waiting on log collection thread
• Collecting -> Collected: Log collection thread has posted logs to swift and updated job with logs URL
• Collected -> Finished: Citrix-ci has posted to gerrit and the job is now complete
• <anything> -> Obsolete: a new job for the same change (recheck or new patchset) has been added

Code

• http://git.openstack.org/cgit/stackforge/xenapi-os-testing/ (although currently using https://github.com/citrix-openstack/xenapi-os-testing)
  • Actual job runner; downloaded by openstack-citrix-ci/osci/tests/test_instructions.py
• citrix-ci: https://github.com/citrix-openstack/openstack-citrix-ci
  • Workflow manager
• TODO - move to upstream, but for now... https://github.com/citrix-openstack/devstack-gate
• TODO - move to upstream, but for now... https://github.com/citrix-openstack/install-nodepool

Node pool

Currently using master branch of nodepool:

• https://github.com/citrix-openstack/nodepool/commit/5f85541e44184b115cc10515bc8ee65e059d75dd

Configuration is here:

• /etc/nodepool/nodepool.yaml

It is all installed using this script:

• https://github.com/citrix-openstack/install-nodepool/blob/master/inp/osci_installscript.sh

The ssh-keys given to created xenserver boxes are configured in the jenkins job that runs the above script to create the xenserver-ci box.

To see how its all doing, run this command:

• nodepool list

To claim a node, you can do:

• nodepool hold id

Above CI scripts do that via a python interface to claim workers from nodepool.

TODO list

• start using stackforge http://git.openstack.org/cgit/stackforge/xenapi-os-testing/
• reduce number of excluded tests, using above stackforge integration
• stop forking infra projects, where possible
• consider moving to zuul, talk to turbo hipster folks
• create new environment to test out config changes
• create system to construct dev environments
• trial out neutron based tests (with quark?)
• BobBall to send johnthetubaguy an example on how to deploy the Main Box