Heat/AutoScaling

Heat Autoscaling now and beyond

AS == AutoScaling

Now

The AWS AS is broken into a number of logical objects

• AS group (heat/engine/resources/autoscaling.py)
• AS policy (heat/engine/resources/autoscaling.py)
• AS Launch Config (heat/engine/resources/autoscaling.py)
• Cloud Watch Alarms (heat/engine/resources/cloud_watch.py, heat/engine/watchrule.py)

Dependencies

Note the in template resource dependencies are:

• Alarm
  • Group
  • Policy
    • Group
      • Launch Config
      • [Load Balancer] - optional

This mean the creation order should be [LB, LC, Group, Policy, Alarm].

When a stack is created with these resources the following happens:

1. Alarm: the alarm rule is written into the DB
2. Policy: nothing interesting
3. LaunchConfig: it is just storage
4. Group: the Launch config is used to create the initial number of servers.
5. the new server starts posting samples back to the cloud watch API

When an alarm is triggered in watchrule.py the following happens:

1. the periodic task runs the watch rule
2. when an alarm is triggered it calls (python call) the policy resource (policy.alarm())
3. the policy figures out if it needs to adjust the group size, if it does it calls (via python again) group.adjust()

Beyond

• make the AS policy a service
• use ceilometer (pluggable monitoring)

Basically the same steps happen just instead of python calls they will be REST calls.

When a stack is created with these resources the following happens:

1. LaunchConfig: it is just config storage
2. Group: the Launch config is used to create the initial number of servers. (note we need to start adding a tag to the server with probably the group name, this is used to aggregate the metrics in ceilometer)
3. Policy: the policy is created in the new policy resource via a POST to the autoscaling service: TODO(radix;). We at this point know the AS-Group so we can pass a webhook to the AS service with the Group resource ID in it. It will need to provide the webhook/url to post alarm notifications to.
4. Alarm: the alarm is created in Ceilometer via a POST (https://github.com/openstack/python-ceilometerclient/blob/master/ceilometerclient/v2/alarms.py). We pass it the above webhook.
5. ceilometer then starts collecting samples and calculating the thresholds.

When an alarm is triggered in Ceilometer the following happens:

1. Ceilometer will post a webhook to what ever we supply (I'd guess a url to the policy API)
2. the policy figures out if it needs to adjust the group size, if it does it calls a heat-api webhook (something like PUT.../resources/<as-group>/adjust)

Authentication

• how do we authenticate the request from ceilometer to AS?
• is this a special unprivileged user "ceilometer-alarmer" that we trust?
• at some point we need to get the correct user credentials to scale in the AS group.

Securing Webhooks

Many systems just treat the webhook URL as a secret (with a big random UUID in it, generated *per client*). I think think this is actually fine, but it has two problems we can easily solve:

• there are lots of places other than the actual SSL stream that URLs can be seen. Logs of the Autoscale HTTP server, for example.
• it's susceptible to replay attacks (if sniff one request, you can send the same request to keep doing the same operation, like scaling up or down)

The first one is easy to solve by putting some important data into the POST body. The second one can be solved with a nonce with timestamp component.

The API for creating a webhook in the autoscale server should return two things, the webhook URL and a random signing secret. When Ceilometer (or any client) hits the webhook URL, it should do the following:

• include a "timestamp" argument with the current timestamp
• include another random nonce
• sign the request with the signing secret

(to solve the first problem from above, the timestamp and nonce should be in the POST request body instead of the URL)

And anytime the AS service receives a webhook it should:

• verify the signature
• ensure that the timestamp is reasonably recent (no more than minutes old, and no more than minutes into the future)
• check to see if the timestamp+nonce has been used recently (we only need to store the nonces used within that "reasonable" time window)

On top of all of this, of course, webhooks should be revokable.

[Qu] if we do this in the context of Heat (db not accessible from the API daemon).

1. We are going to have to send all webhooks to the heat-engine for verification.
2. This is because we can't check the uuid in the API, thus making it very easy for a DOS attack. Any idea on how to solve this?

[An] This doesn't sound like a unique problem, which should be solved by rate limiting, as other parts of OpenStack do.

[Qu] Why make Autoscale a separate service?