Debugging XenServer CI failures

Introduction to results

When the job is initially run, all the results go to run_tests.log - you can find most failures in here, although at different points during the process other scripts are run which might get passed to other jobs. The other common log file with errors in the process is the devstacklog.txt.gz.

If the job has completed, there will be a test_results.html.gz file showing which tempest jobs passed or failed. From there on, debugging the failures is the same as in the gate.

Fix suggestions

The primary fix for unknown issues is to rebase on the latest master. The XenServer CI currently fetches the change from review.openstack.org; this means that all of the dependencies are pulled in, but it is _NOT_ merged with the latest master repository. This is in contrast to the stock CI environment where a merge is performed.

There have been various cross-project issues that cause failures in jobs in the XenServer CI, such as the requirements.txt file being "out of date". As we pull in the master of all other repositories, the global requirements can move forward while the XenServer CI still pulls the old requirements from the version that your change was based off.

By rebasing on the latest master many of the unexplained failures have disappeared. We are, of course, going to try to fix this generically.

Contact details

The primary maintainer of the CI is BobBall on IRC who can also be reached at the openstack@citrix.com mailing alias.