Jump to: navigation, search

Heat/AutoScaling

< Heat
Revision as of 14:59, 17 September 2013 by Radix (talk | contribs) (CeilometerAlarm already seems like it'll work)

Summary

This is a proposal for a new design for Heat autoscaling. The existing AWS-based design is described at Heat/AWSAutoScaling.

The design is currently reflected in this blueprint: https://blueprints.launchpad.net/heat/+spec/autoscaling-api-resources

Use Cases

  1. Users want to use AutoScale without using Heat templates.
  2. Users want to use AutoScale *with* Heat templates.
  3. Administrators or automated processes want to add or remove *specific* instances from a scaling group. (one node was compromised or had some critical error?)
  4. Users want to scale arbitrary resources, not just instances.

AutoScaling API

The general ideas of this proposal are as follows:

  • Implement new resources for scaling groups and policies in terms of a new, separate API (implemented in the Heat codebase)
  • That separate API will be usable by end-users directly, or via Heat resources.
  • That API will create a Heat template and its own Heat stack whenever n scaling group is created within it.
  • As events happen which trigger a policy that changes the number of instances in a scaling group, the autoscale API will generate a new template, and update-stack the stack that it manages.
  • The existing CeilometerAlarm resource will be able to be used with the URL from a WebhookTrigger resource.

The AutoScaling Resources

There are a number of resources associated with autoscaling:

  • ScalingGroup - a group that can scale an arbitrary set of heat resources.
  • ScalingPolicy - affects the number of scaling units in a group (+1, -10%, etc)
  • WebHookTrigger - creates a new webhook that can be used to execute a ScalingPolicy
  • ScheduleTrigger - schedules execution of a ScalingPolicy for certain times

ScalingGroup

A scaling group that can manage the scaling of arbitrary Heat resources.

  • Properties:
    • name
    • max_size
    • min_size
    • cooldown
    • resources: The mapping of resources that will be duplicated in order to scale.

The 'resources' mapping is duplicated for each scaling unit. For example, if the 'resources' property is specified as follows:

mygroup:
    type: OS::Heat::ScalingGroup
    properties:
        resources:
            my_web_server: {type: AWS::EC2::Instance}

then if we scale to "2", the concrete resources included in the private stack's template will be as follows:

my_web_server-1: {type: AWS::EC2::Instance}
my_web_server-2: {type: AWS::EC2::Instance}
    ...

And multiple resources are supported and scaled in lockstep. For example, if the 'resources' property is specified as follows:

resources:
    my_web_server: {type: AWS::EC2::Instance}
    my_db_server: {type: AWS::EC2::Instance}

Then the resulting template (when scaled to "2") will be

my_web_server-1: {type: AWS::EC2::Instance}
my_db_server-1: {type: AWS::EC2::Instance}
my_web_server-2: {type: AWS::EC2::Instance}
my_db_server-2: {type: AWS::EC2::Instance}

Special 'Ref' behavior

Consider a scenario where we want to scale some Nova servers and also place them in a load balancer. Assuming a resource OS::Nova::Server and OS::Neutron::LBMember (for "Load Balancer Member" -- a resource that associates a server with a load balancer), which are used in the following way:

my_lb: {type: OS::Neutron::LoadBalancer}
my_server: {type: OS::Nova::Server}
lb_member:
    type: OS::Neutron::LBMember
    properties:
        server: {Ref: my_server}
        lb: {Ref: my_lb}

Now, if we want to use this construct in a ScalingGroup, we have a small issue. Consider the following template:

my_lb: {type: OS::Neutron::LoadBalancer}
my_group:
    type: OS::Heat::ScalingGroup
    properties:
        resources:
            my_server: {type: OS::Nova::Server}
            lb_member:
                type: OS::Neutron::LBMember
                server: {Ref: my_server}
                lb: {Ref: my_lb}

The problem is that "my_lb" lives in the stack that is directly controlled by the user (represented by this template), but "my_server" and "lb_member" are not resources in *this* stack, but rather will be copied into the template managed by the autoscale API and live in an entirely other stack. So the "server" {Ref} is in the AS stack, and the "lb" {Ref} is in the user's stack.

It will be fairly simple to support this case by overriding the behavior of "Ref" in the context of ScalingGroup, to allow "delayed dereferencing". If a {Ref} target is not found, we can check to see if it is declared as one of the members of the "resources" property -- if so, we can ignore it, passing it along literally to the AS API. Then, when the AS API processes its template, it can rewrite the {Ref} recipients to point to the indexed resources that it generates (e.g. "my_server-52").

ScalingPolicy

A scaling policy describes a particular type of change to a scaling group, such as "add -1 capacity" or "add +10% capacity" or "set 5 capacity".

  • Properties:
    • name
    • group_id: the ID of the group that this policy will affect
    • cooldown
    • change: a number that has an effect based on change_type.
    • change_type: one of "change_in_capacity", "percentage_change_in_capacity", or "exact_capacity" -- describes what this policy does (and the meaning of "change")

WebHookTrigger

Represents a revokable webhook endpoint for executing a policy.

For example, when you create a webhook for a policy, a new URL endpoint will be created in the form of http://as-api/webhooks/<random_hash>. When that URL is requested, the policy will be executed.

This resource will be useful in combination with a CeilometerAlarm resource that knows how to set up Ceilometer to execute a webhook when an alert happens.

  • Properties:
    • policy_id: The ID of the policy.
  • Attributes:
    • webhook_url: The webhook URL.

ScheduleTrigger

Defines a trigger that will execute the policy at scheduled times. Useful for scaling up before regular expected load.

  • Properties:
    • policy_id: The ID of the policy.
    • cron: a cron-style schedule string (optional; exclusive with 'at')
    • at: an at-style schedule string (optional; exclusive with 'cron')

Authentication

  • how do we authenticate the request from ceilometer to AS?
  • is this a special unprivileged user "ceilometer-alarmer" that we trust?
  • The AS API should have access to a Trust for the user who owns the resources it manages, and pass that Trust to Heat.

Securing Webhooks

Many systems just treat the webhook URL as a secret (with a big random UUID in it, generated *per client*). I think think this is actually fine, but it has two problems we can easily solve:

  • there are lots of places other than the actual SSL stream that URLs can be seen. Logs of the Autoscale HTTP server, for example.
  • it's susceptible to replay attacks (if sniff one request, you can send the same request to keep doing the same operation, like scaling up or down)


The first one is easy to solve by putting some important data into the POST body. The second one can be solved with a nonce with timestamp component.

The API for creating a webhook in the autoscale server should return two things, the webhook URL and a random signing secret. When Ceilometer (or any client) hits the webhook URL, it should do the following:

  • include a "timestamp" argument with the current timestamp
  • include another random nonce
  • sign the request with the signing secret

(to solve the first problem from above, the timestamp and nonce should be in the POST request body instead of the URL)

And anytime the AS service receives a webhook it should:

  • verify the signature
  • ensure that the timestamp is reasonably recent (no more than minutes old, and no more than minutes into the future)
  • check to see if the timestamp+nonce has been used recently (we only need to store the nonces used within that "reasonable" time window)


On top of all of this, of course, webhooks should be revokable.


[Qu] if we do this in the context of Heat (db not accessible from the API daemon).

  1. We are going to have to send all webhooks to the heat-engine for verification.
  2. This is because we can't check the uuid in the API, thus making it very easy for a DOS attack. Any idea on how to solve this?

[An] This doesn't sound like a unique problem, which should be solved by rate limiting, as other parts of OpenStack do.

[Qu] Why make Autoscale a separate service?

[An] To clarify, service = REST server (to me)

Initially because someone wanted it separate (rackers). But I think it is the right approach long term.

Heat should not be in the business of implementing too many services internally, but rather having resources to orchestrate them.

monitoring <> Xaas.policy <> heat.resource.action()

Some cool things we could do with this:

  1. better instance HA (restarting servers when they are ill) - and smarter logic defining what is "ill"
  2. autoscaling
  3. energy saving (could be linked to autoscaling)
  4. automated backup (calling snapshots at regular time periods)
  5. autoscaling using shelving? (maybe for faster response)

I guess we could put all this into one service (an all purpose policy service)?