Heat/Blueprints/RollingUpdates
- Launchpad Entry: HeatSpec:rolling-updates
- Created: 07 Feb 2013
- Contributors: Clint Byrum
Summary
While managing a large group of instances, I may want to roll out changes in topology and/or configuration to a limited percentage of these instances and then wait to see if those initial rollouts produced failures or successes before deploying a larger percentage. This is known as a "canary" deployment strategy, after the old mining practice of carrying a canary in a lantern to test for air quality.
Release Note
Multi-Instance resources may now specify a property which causes them to apply updates using a rolling or canary strategy.
Rationale
With large scale deployments, updating configuration on all machines at once without testing may result in downtime. Being able to control the deployment will lead to more reliability for users who implement it.
User stories
As an operations engineer I want to roll out a change to topology or configuration on a very large resource without the risk of significant downtime or error rates.
Assumptions
Design
A single new property will be introduced to OS::Heat::InstanceGroup:
The string argument will be one of "rolling", "canary", or "immediate". If this property is not specified, 'immediate' is assumed.
rolling
Metadata updates will be performed on one resource at a time. Any WaitCondition that depends on the InstanceGroup will be waited on before continuing to the next resource. Any failure of said WaitCondition will result in rolling back to the previous Metadata.
canary
Identical to rolling except WaitCondition will be waited on in groups of instances rather than 1 at a time. The progression is 1 instance, then 1%, 5%, 20%, then the remainder.
immediate
The new Metadata is copied to all instances immediately without waiting.
Implementation
Currently an update stack just tries to update the metadata for each resource. To facilitate the rollback capability of rolling/canary upgrades, a new column will be needed in the resource table previous_rsrc_metadata. As machines are updated, their previous metadata will need to be stored so that they can be rolled back. Updates to stacks are already protected by code that will not let another update happen in parallel, so there is no need for any joining to a table of versioned metadata. Once an InstanceGroup has been updated, all of the previous_rsrc_metadata should be set to NULL.
UI Changes
N/A
Code Changes
TBD
Migration
- Adding a column means a schema change, and so would have to be handled in database migrations.
Test/Demo Plan
TBD'
