Jump to: navigation, search

Difference between revisions of "Nova/Turbo-Hipster"

(Database CI)
(Database CI)
 
Line 5: Line 5:
 
Doing a large OpenStack deployment has always been hard when it came to database migrations. Running a migration requires downtime, and when you have giant datasets that downtime could be hours. To help catch these issues [http://josh.people.rcbops.com/2013/09/building-a-zuul-worker/ Turbo-Hipster] will now run your patchset’s migrations against copies of real databases. This will give you valuable feedback on the success of the patch, and how long it might take to migrate.
 
Doing a large OpenStack deployment has always been hard when it came to database migrations. Running a migration requires downtime, and when you have giant datasets that downtime could be hours. To help catch these issues [http://josh.people.rcbops.com/2013/09/building-a-zuul-worker/ Turbo-Hipster] will now run your patchset’s migrations against copies of real databases. This will give you valuable feedback on the success of the patch, and how long it might take to migrate.
  
[[ThirdPartySystems/TurboHipster|3rd Party CI information at ThirdPartySystems/TurboHipster]]
+
[[ThirdPartySystems/DB Datasets CI|3rd Party CI information at ThirdPartySystems/DB Datasets CI]]
  
 
=== What should I do if Turbo-Hipster fails? ===
 
=== What should I do if Turbo-Hipster fails? ===

Latest revision as of 03:05, 14 August 2014

Database CI

Turbo-Hipster is continuous integration for OpenStack nova database migrations. For each new patchset for Nova the database migrations are ran against real user datasets and timed to catch performance issues that may cause significant maintenance times for operators.

Doing a large OpenStack deployment has always been hard when it came to database migrations. Running a migration requires downtime, and when you have giant datasets that downtime could be hours. To help catch these issues Turbo-Hipster will now run your patchset’s migrations against copies of real databases. This will give you valuable feedback on the success of the patch, and how long it might take to migrate.

3rd Party CI information at ThirdPartySystems/DB Datasets CI

What should I do if Turbo-Hipster fails?

That depends on why it has failed. Here are some scenarios and steps you can take for different errors:


 FAILURE – Did not find the end of a migration after a start

If you look at the log you should find that a migration began but never finished. Hopefully there’ll be a traceroute for you to follow through to get some hints about why it failed.


 WARNING – Migration %s took too long

In this case your migration took a long time to run against one of our test datasets. You should reconsider what operations your migration is performing and see if there are any optimisations you can make, or if each step is really necessary. If there is no way to speed up your migration you can email us at rcbau@rcbops.com for an exception.


 FAILURE – Final schema version does not match expectation

Somewhere along the line the migrations stopped and did not reach the expected version. The datasets start at previous releases and have to upgrade all the way through. If you see this inspect the log for traceroutes or other hints about the failure.


 FAILURE – Could not setup seed database. FAILURE – Could not find seed database.

These two are internal errors. If you see either of these, contact us at rcbau@rcbops.com to let us know so we can fix and rerun the tests for you.


 FAILURE – Could not import required module.

This error probably shouldn’t happen as Jenkins should catch it in the unit tests before Turbo-Hipster launches. If you see this, please contact us at rcbau@rcbops.com and let us know.


If you receive an error that you think is a false positive, leave a comment on the review with the sole contents of "recheck migrations".


Escalation

If you see any continued false positives or have any questions or problems please contact us on rcbau@rcbops.com