Swift/ideas/hierarchical keymaster

Hierarchical Keymaster

The encryption functionality recently merged into the swift master branch has the following features:

• Keymaster providing the keys, encrypter and decrypter functionality in the encryption filter in the wsgi pipeline
• Static root secret in the proxy-server.conf file
• Encryption keys for containers and objects derived using the root secret and the path of the container and object
• Randomly generated body key used to encrypt the actual object data, and body key wrapped with object key provided by keymaster
• Container and object keys provided by the keymaster for encrypting metadata

As an extension to this encryption functionality, hereafter known as the base functionality, this proposal contains a high-level design of a hierarchical keymaster that provides functionality such as:

• Root secret stored in an external key management system such as Barbican
• Randomly generated data encryption keys (DEKs) for accounts, containers, objects (commonly referred to as entities), which are provided by the hierarchical keymaster to the encrypter and decrypter functionality provided by the base. The DEKs are generated once at creation time of the entities, and are never changed.
• Randomly generated key encryption keys (KEKs) that are used to wrap the DEKs, and which are stored in the metadata of the entity that they protect.
• Each KEK is wrapped by the KEK in the next-highest level in the hierarchy (e.g., an object KEK is wrapped by the KEK of its container), with the account KEK being wrapped with the root secret.
• Key rotation can be performed on the KEKs as a way of providing cryptographic, secure deletion of user data.

The hierarchical keymaster is designed to be a drop-in replacement for the base keymaster when configuring an encryption environment - in particular, the base encryption and decryption functionality, along with which data is and is not to be encrypted, will not change.

The work on the functionality described above can be separated into three parts:

1. Support for storing a root secret key in Barbican, while still deriving the other keys (container, object) based on the path, as in the base implementation.
2. Root secret in Barbican, and randomly generated KEKs and DEKs stored in the entity metadata.
3. Support for rotating the root secret and KEKs (secure deletion)

Some of the main challenges that have been identified in earlier discussions are:

• Using client-controlled keys, e.g., in Barbican, while preserving the API. This impacts e.g., container-sync and other background processes, which will not have the necessary client credentials required to access the key(s).
• Using random keys, since for some scenarios, it is then necessary to retrieve metadata from the object server (using "HEAD"), which incurs a performance penalty. In particular, this impacts update operations ("POSTs") and conditional GETs.
• Eventual consistency brings challenges to the key rotation, since it is difficult, or even impossible, to conclude that an object is no longer encrypted with an old key.

Root secret in Barbican

The first step in moving towards a hierarchical keymaster with support for key rotation, is providing a mechanism for storing the root secret somewhere else than in the proxy-server.conf configuration file. The base keymaster is replaced with a new secdel_keymaster in the WSGI pipeline, and the system also needs to be configured with a Barbican URL, e.g.:

... secdel_keymaster encryption proxy-logging proxy-server

[filter:secdel_keymaster]
use = egg:swift#secdel_keymaster
barbican_url = http://192.168.50.11:9311/v1

Internally, the secdel_keymaster retrieves a root secret key from Barbican, or if one does not exist, one is created and stored in Barbican. The lookup is done using a name generated from, among others, the project ID.

Alternatively, the account key could (also) be stored in Barbican, which may facilitate future moves to BYOK.

Considerations

For a BYOK deployment, the Barbican key could be one provided by the user(s), in which case it could still be a root secret (in case there is one user per cluster), or an account secret (for a one user per account scenario). With BYOK, additional questions regarding the configuration, addition of new API calls and/or modification of existing ones etc. need to be answered.

Randomly generated keys

The next step is using randomly generated keys, both for the DEKs provided by the secdel_keymaster to the encrypter and decrypter, as well as for the KEKs that are used to wrap the DEKs and the lower-level KEKs. The pair KEK-DEK is termed an entity key, and an entity key also contains the parent KEK ID, i.e., the ID of the key used to wrap the KEK of the entity key. The ID of the entity keys are uunique identifiers in the UUID1 format. In particular, the ID needs to have the following properties:

• Be unique, so that e.g., multiple proxy nodes do not need to coordinate for creating the IDs, which would be the case for IDs based on incrementing counters.
• Contain a timestamp, so that it can be determined which is the most recent key.

The keys are stored in the metadata of the entity in the following secdel-metadata format:

X-<Entity>-[Transient-]-Sysmeta-Secdelkeywrapinfo:
  { uuid1 :
    { "kek_wrapping_key_id" : wrapping_key_uuid1,
      "kek_payload" : wrapped_kek_material1,
      "dek_payload" : wrapped_dek_material1
    },
    uuid2 :
    { "kek_wrapping_key_id" : wrapping_key_uuid2,
      "kek_payload" : wrapped_kek_material2,
      "dek_payload" : wrapped_dek_material2
    }
  }

Where uuidN is the unique identifier of the Nth entity key. In addition to using IDs to identify which wrapping keys are used for wrapping and unwrapping, the wrapping algorithm itself also should contain integrity checking so that attempts to unwrap a key with the wrong wrapping key are caught already at the time of unwrapping, and the error is not propagated to the end user in the form of corrupted user data. This can be achieved e.g., by using AES key wrap (see RFC3394).

Key Generation for Encryption

When creating a new entity, an entity key is generated. The KEK of the entity key is chosen to be KEK of the most recent entity key of the next-higher level.

Key Retrieval for Decryption

When retrieving an entity, the secdel-metadata is retrieved and the kek wrapping key ID is read to determine the ID of the KEK used to wrap the entity KEK. This is done for all higher-level entities, all the way up to the unwrapped root secret. Once all the necessary entity keys have been gathered, the unwrapping is performed from the root secret all the way down to the entity being retrieved.

Considerations

The secdel-metadata from the response headers are needed by the secdel-keymaster to be able to determine the wrapping keys and to retrieve the key material necessary for unwrapping the entity DEKs for providing to the decrypter. The secdel-metadata is provided through the callback function to the keymaster, which is called by the decrypter to retrieve the keys.

Key rotation

Key rotation of the root secret and entity KEKs is to be used to be able to securely delete user data. To rotate the root secret and all KEKs, effectively securely deleting all user data whose keys are not rotated, the following actions are performed:

Create a new root secret
For each account:
  Unwrap the DEK from the latest entity key
  Create a new account KEK, use it to wrap the DEK
  Wrap the new account KEK with the new root secret
  Store the new account entity key along the existing ones in secdel-metadata
  Repeat for each container, and each object in containers

During the key rotation process, entities are accessible using either new or old keys. Once the key rotation process is complete, the old keys can be deleted from the secdel-metadata and Barbican.

Considerations

The following points need to be addressed for the key rotation:

• How to efficiently update secdel-metadata in all entities?
• How to know when the key rotation process (creation of new keys, deletion of old keys) has completed?
• How to trigger the (starting, finalization) key rotation (background) process?
• How is the access control/account information provided to the (background) process that rotates the keys?

Contact: mathiasb, kmARC, jrichli on IRC