Jump to: navigation, search


Secure Secret Storage


The blueprint: https://blueprints.launchpad.net/nova/+spec/secure-secret-storage


It would be great to be able to encrypt sensitive data e.g. in Swift.

We would have to store the keys securely.

The proposal is to store the secret in Keystone, so that we need only get it right once.

The secrets in Keystone are unlocked by the action of authenticating, so that the secret is not actually stored (unencrypted) anywhere.

When a service verifies the token, it can also get the secret, so that it can then decrypt information secured to that user/tenant.

Password login

Consider first simple password login: the user passes (over SSL) their username & password. The Keystone backend is pluggable, but consider simple database-style backends.

In order to generate a secure key, we could simply take the password and run it through PBKDF2: http://en.wikipedia.org/wiki/PBKDF2

This would be a secure key that is not stored anywhere (ignoring the fact that we're storing the hash - we'll come to that later). We could pass this key to Swift (more details later) and use it to encrypt files.

Changing passwords

If we're directly using the password -> key then we'd have to re-encrypt all the files when the user changes their password.

So, we introduce a layer of indirection. The password -> password key. The actual secret key is stored encrypted with the password key, in the user database row. When a user logs in, you take the password, PBKDF2 gives us the password key, then we use that to decrypt the secret key as stored in the user row.

Now, if the user needs to change their password, we only need to re-encrypt one value: the secret key in the user row encrypted with the password-derived key.

We'll call that sequence of unlock operations the chain; we'll use the chain concept again and again.


Turns out that Keystone doesn't really deal in users, but instead deals with tenants. Each user can be a member of one or more tenants. So, we need per-tenant secret keys, for information which is shared across all users in a tenant.

In a database, we might have a "user_tenant" membership table with a structure like ( user_id, tenant_id ).

We just add a column to the membership table: (user_id, tenant_id, `tenant_secret_by_user_secret` )

The pithily named tenant_secret_by_user_secret encrypts the tenant secret with the user secret.

So now, our chain looks like this:

password -> password secret -> user secret -> tenant secret

And we can pass the tenant secret to Swift.


Services like swift can also continue the chain. For example, each container in Swift might be available to one or more tenants. We'd want a per-container secret.

Again, we'd have a table like "tenant_container_rights". We'd extend this with a container_secret_by_tenant_secret.

Strong ACLs

What the chain does here is to take ACLs, and cryptographically enforces them. If you don't have the secret which proves that you are a member of the ACL, you can't get the rights/secret the ACL grants.


I've glossed over a detail here... when the user authenticates, we give them a token, the user presents that token to the service, the service presents the token to Keystone, and then Keystone must give the service the secret.

If tokens are stored in the database (UUID tokens), then this is weak. The user secret would have to be stored unencrypted (or encrypted with an available token).

There are two approaches here:

Use UUID tokens, but break them into two components: <UUID>/<token secret>. The token secret can be used as the secret to unlock the user secret stored in the token row.

Use cryptographically signed tokens (not stored in the database), and store the user secret directly in the token (in plaintext). This has the downside that user secrets are exposed to authenticated users, although I don't believe this is actually a security issue. We could (if desired) have a time-based encryption key which can be stored by Keystone either per-user or service-wide, which could be used to encrypt the tokens. One example implementation would be to generate a new key every minute; we store the ID of the key in the token so that we know which key to use to decrypt the token; we only need to keep 60 keys around if tokens expire every hour. For version 1, I propose _not_ encrypting the secret in the token.

API Keys or other auth mechanisms

Other authentication mechanisms (other than passwords) are possible, though none are yet supported by Keystone.

For each, we'd have to figure out how to generate a secret.

For example, for API keys, we might have an id and a secret, and we might sign the message with the secret, and so we store the secret in the database. One approach would be to split the ID <uuid>/<secret2>; we would only store uuid in the database, and we would derive the key from secret2.


Glance: Encrypted images Quantum: VPN keys Keystone: 'Foreign' passwords (Windows?) Nova: Encrypted volumes XaaS: Application credentials

Weakness: Two hashes

Currently Keystone hashes the password and compares it to a stored hash, to determine whether to grant access.

The current algorithm is to use repeated SHA-512: http://packages.python.org/passlib/lib/passlib.hash.sha512_crypt.html

This hash is stored in the database, so obviously we can't use the same algorithm with the same salt to derive the key!

It should be secure to use the same algorithm with a different value e.g. <username>||<password>, or even just a different salt.

I'd be happier if we used a different hash function entirely (as PBKDF2 normally uses SHA as well). Maybe bcrypt or scrypt?

I think I will allow this to be configurable, based on your level of paranoia. This means storing the algorithm in use, which I believe is done anyway.

The weakest link will always remain brute-forcing the passwords, particularly if users are allowed to choose them.


Steal the Keystone DB: Still easiest to brute force the passwords.

Steal the Keystone server: Maybe a few keys in-memory (if you're really good). Only those users compromised.

Steal a Swift server: Maybe a few keys in-memory. Only those containers compromised.

Steal tokens: Gain access to that user's information, which you have access to via the token anyway.

Other attacks?


  1. Implement basic secure secret storage in Keystone, for password authentication
  2. Extend the token-backends to securely store the secret using the 'split token' approach
  3. Implement the secret chain in Keystone
  4. Pass the secret to services (that request it?)
  5. Use the secret somewhere (encrypt files in swift gateway?)