Jump to: navigation, search


Blueprint Instance Users


Problem Statement

We have a need to enable API authentication from within instances launched as part of a Heat stack. The instances, once launched, are not directly controlled by heat, so we have to treat them as being implicitly untrusted - we need to limit privileges of whatever credentials are deployed inside the instance to the absolute minimum possible that allow authentication with our API, ideally also reinforced via policy RBAC to minimise the API surface these credentials can access.

The key requirement is that we should be able to validate the identity of the agent connecting to the API, and map that to a specific Heat resource

  • The lifecycle of the credentials should be manageable on a per-resource basis
  • Some use-cases may require impersonation of the stack owning user (e.g posting metrics to ceilometer?_
  • Many other use cases do not require impersonation (e.g signals for wait condition notification, where we can map the identity inside heat)

What we do now

Currently, if we create any of the following resources, we create a new keystone user, in the same tenant as the user creating the stack, and assign that user to the "heat_stack_user" role, which is restricted to specific API actions via policy.json.

  • AWS::AutoScaling::ScalingPolicy
  • AWS::CloudFormation::WaitConditionHandle
  • OS::Heat::HARestarter
  • AWS::IAM::User

The problem with this is that every user creating a heat stack containing one of the following resources needs to be a keystone admin, otherwise they can't create the users. This is a big problem for "real world" deployments of Heat, where most users are not expected to have keystone administrative privileges.

Additionally, the reliance on the ec2tokens keystone functionality is problematic for some users, as their environments do not enable this keystone extension.

Proposed Solutions

1. Use credentials associated with a trust

  • Create one dedicated "heat_proxy_user" in a dedicated project, which has no roles assigned in any project
  • When we create a heat stack, we create a trust between the user creating the stack, and the heat_proxy_user, enabling impersonation and delegating only the "heat_stack_user" role, which can be used to restrict access via policy.json. The stack owning user is not restricted because we could use a rule like "if heat_stack_user and not Member"
  • Each resource which requires credentials creates an ec2 keypair using a trust-scoped token, meaning the resulting credentials are associated with the trust_id, so they can be used to perform role-limited impersonation of the stack-owning user

Pros: Solves the admin requirement, could work with the v2 keystone API (so backportable, possibly, to Havana)

Cons: Doesn't solve the dependence on ec2tokens, arguably complex, security implications of having shared proxy user (despite impersonation, because...bugs happen)

2. Use Domain separation and username/password

This suggestion was made by ayoung (initially in the context of using trusts, but I'm not sure we actually need to do impersonation in all cases)

  • Create a keystone domain specific to heat
  • For each heat stack, we create a new project in the heat domain
  • For each resource requiring credentials, we create a keystone user, in the stack project of the heat domain
  • The user has a randomly created password, which we can deploy in the instance
  • The user can be assigned the heat_stack_user role to limit the API surface accessible
  • Optionally, we could create a trust between the stack owning user and the heat "resource user", where impersonation is required (still not sure we actually need it in this case)

Pros: Solves the admin requirement (keystone user creation is performed by the heat service admin user), relatively simple, password auth solves dependency on ec2tokens auth.

Cons: Requires v3 keystone API (probably can't backport to Heat stable/havana)

Additional info/ideas:

  • Could add a "deploy_credentials" property to OS::Nova::Server, so the username/password are provided to the instance via nova metadata key/value pairs
  • We could use python-heaclient as the agent which talks to heat, e.g heat stack-signal mystack --data "complete!"
  • Could still derive an ec2-keypair from the heat-domain-resource-user to support compatibility with current heat-cfntools/boto
  • heat-cfntools could potentially be ported to use the ReST API and python-heaclient's python bindings instead of boto

3. Use OAuth access/secret keys and signed requests

OAuth has been mentioned as a possible solution to this problem, but currently it looks like there is too much missing from the keystone implementation to consider it a viable option, in particular the client support, request signing/validation and auth middleware do not yet exist.

4. x509 certificates

TODO: ayoung mentioned this as a possible solution, need more info

5. Resource-scoped virtual keypairs

  • Create and store random strings in resource data to be used as a virtual keypair. (keystone creates these with uuid.uuid4().hex)
  • Use this virtual keypair for any URL or request signing, just as we do now for real ec2 keypairs
  • Create a heat engine service RPC call which uses keystoneclient Ec2Signer to check request signature with the virtual keypair and return a token using the configured stack credentials (trusts or stored creds)
  • Write new middleware or modify EC2Token to attempt to get a token via the heat RPC call, with fallback to ec2token API


  • No need to create keystone users, hence no need for heat user to be admin
  • No requirement for any recent keystone features (trusts, domains etc) so Icehouse heat should work on many openstack clouds
  • No need to change any existing templates or agent configuration
  • Openstack-infra can potentially start using heat for infra workloads on HPCloud and Rackspace (this has been a huge blocker for them)
  • This approach can be implemented (or at least tried) with relatively few code changes


  • Potential to perform authenticated operations with a compromised keypair (possibly the risk is the same as the current solution, except that this requires new codepaths outside of keystone. Also possibly mitigated by policy.json)

(shardy): Question - *whose* token does the engine return after validating a signature? Since the virtual keypairs aren't backed by any keystone user, it is unclear to me what user the token would be bound to. Also, since the keypair isn't associated with a real user who has keystone role assignments, I don't currently understand how we could control access via policy.json?

Additional info/ideas:

  • ec2token middleware could always call the heat-engine rpc to get a token, and the first implementation of the rpc method could just delegate to the ec2token API
  • maybe some policy could be written to limit operations to the resource that the keypair belongs to
  • this could be developed alongside another of the above options (such as 2.)