Heat/Blueprints/as-update-policy

Summary

The following is the proposed solution for the as-update-policy blueprint. We want to add an UpdatePolicy attribute that can be used with InstanceGroup and AutoScalingGroup to specify how changes to the launch configuration or subnet are rolled out. The UpdatePolicy attribute can be introduced to an existing stack during a request for a stack update. For the InstanceGroup resource type, we want to add the following snippet at the cfn template.

  "UpdatePolicy" : {
     "RollingUpdate" : {
        "MinInstancesInService" : "1",
        "MaxBatchSize" : "12",
        "PauseTime" : "PT60S"
     }
  }

MinInstancesInService indicates the number of instances that must be in service while other instances are being replaced.
MaxBatchSize indicates the maximum number of instances to roll out with each batch.
PauseTime indicates the wait time between each change.

The example below is a cfn template for an InstanceGroup with UpdatePolicy. This snippet is a revision of the sample heat template @ https://github.com/openstack/heat-templates/blob/master/cfn/F17/InstanceGroup.template. In the example below, update to the LaunchConfiguration in the JobServerGroup for an existing stack will trigger the specific RollingUpdate policy under the UpdatePolicy attribute. The name of the entry under UpdatePolicy is not significant. The InstanceGroup resource only expects one entry within the UpdatePolicy attribute. During the update, the JobServerGroup must have at least one instance in service. The update will be rolled out in batches of 5 instances. For each batch, new instances will be created first in parallel prior to terminating the old instances. There will be a 30 seconds pause before each batch is rolled out.

 "Resources" : {
   "JobServerGroup" : {
     "UpdatePolicy" : {
       "RollingUpdate" : {
         "MinInstancesInService" : "1",
         "MaxBatchSize" : "5",
         "PauseTime" : "PT30S"
       }
     },
     "Type" : "OS::Heat::InstanceGroup",
     "Properties" : {
       "LaunchConfigurationName" : { "Ref" : "JobServerConfig" },
       "Size" : {"Ref": "NumInstances"},
       "AvailabilityZones" : { "Fn::GetAZs" : "" }
     }
   },
   "JobServerConfig" : {
     "Type" : "AWS::AutoScaling::LaunchConfiguration",
     "Properties": {
       "ImageId"           : { "Ref" : "ImageId" },
       "InstanceType"      : { "Ref" : "InstanceType" },
       "KeyName"           : { "Ref" : "KeyName" },
       "NovaSchedulerHints": [ {"Key": "part", "Value": "long"},
                               {"Key": "ready", "Value": "short"} ],
       "UserData"          : { "Fn::Base64" : { "Fn::Join" : ["", [
         "#!/bin/bash -v\n"
       ]]}}
     }
   }
 }

The current Heat engine does not support changes in the underlying resource reference (i.e. LaunchConfiguration). Given the above example, if JobServerConfig is updated, when checking JobServerGroup for update, the changes to JobServerConfig is not recognized at _update_resource of the StackUpdate class. Therefore, a resource update for the InstanceGroup would not get triggered. The LaunchConfiguration resource will recognize the update but currently there's no update handler implemented and it seems more appropriate to let the InstanceGroup handle its own instance updates. So currently, the only way to trigger a change in the InstanceGroup as a result of the LaunchConfiguration change is if we rename the LaunchConfiguration JobServerConfig in the cfn template. Since LaunchConfigurationName is not in the update_allowed_properties of InstanceGroup, this will lead to a replace (destroy follow by create) of the existing InstanceGroup. This is not the desire solution as we want the update to the LaunchConfiguration to be rolled out in a controlled fashion.

Implementation

The following are changes proposed for implementation of this blueprint. The goal is to allow a resource to recognize that there's an update with the resources that it references. The resource should continue to make the decision on how to handle its own update. Currently, the template differences and property differences are passed into the update function and the resource makes the decision what to do with the differences. We want to extend and include resource differences.

Modify LaunchConfiguration class

Add properties of LaunchConfiguration to update_allowed_properties
Add handle_update for LaunchConfiguration to update difference in properties

Modify StackUpdate class

Modify the _udpate_resource method and add a check to determine if resources referenced by the current resource have changed between existing and new templates

Modify Resource class

Add update_template_diff_resources method that returns a dictionary of resources referenced by self that have been changed
Modify the update method in Resource
- Include call to update_template_diff_resources
- Pass the dict returned from update_template_diff_resources into the handle_update method as optional arg rsrc_diff
- To avoid making changes to handle_update of all resources, use inspect.getargspec to determine if rsrc_diff is supported for the resource's handle_update method
This potentially paves the way to support triggering of update of other resource types such as changes to subnet or security group that are referenced by other resources.

Add UpdatePolicy class

Put this new class in the autoscaling module under the engine module

Modify InstanceGroup

Add UpdatePolicy to updated_allowed_keys in InstanceGroup and modify handle_update to property differences in UpdatePolicy
- Changes to the UpdatePolicy is only property changes
- Changes to the UpdatePolicy alone does not trigger InstanceGroup to update/replace its instances
Modify handle_update to handle rolling update
- The rolling update will only be triggered if there's an UpdatePolicy defined and that the LaunchConfiguration is listed in rsrc_diff