Swift/ideas/keymaster v2

Keymaster v2

The original keymaster uses a single root encryption secret stored in the proxy server configuration file to generate keys for the accounts, containers and objects, based on the path to the account/container/object.

Summary of features to be added to the keymaster

This design aims to add the following features to the keymaster:

Support multiple root encryption keys

Supporting multiple root encryption keys includes:

• Ability to switch to a new root encryption key for encrypting new objects
• Do not implement re-keying/re-wrapping of existing data, but
• Design and implement changes in a way that would allow for re-keying later

Support root encryption keys stored in an external key management server

Update: This is implemented in https://review.openstack.org/#/c/364878 Steps necessary to setup for testing are at https://etherpad.openstack.org/p/swift-kms-keymaster-setup

• In practice, a Barbican server accessed using the Castellan library
• Implementation and tests structured such that someone who does not want Barbican, does not need to install dependencies (e.g., oslo)

Support multiple instances of key servers

• Multiple locations for locally stored root secrets
• Multiple external Barbican servers

System Model

The assumptions regarding the swift storage system are as follows:

• Current Swift administrator is trusted
• Encryption key storage is trusted and securely erasable, e.g., a key management server backed by an HSM, or removable storage plugged into proxy server
• Servers (proxy, account, container, object) are trusted, including the processes running in the servers, memory
• All storage is untrusted

Attack scenarios

The following attack scenarios are considered and protected against:

Gaining access to decommissioned disks from proxy, account, container, object servers:

• Encryption keys stored wrapped in the metadata, data not accessible
• Brute-forcing the encryption keys takes a long time
• Key rotation can reduce exposure in case a key is brute-forced

Attacker is a former Swift admin with key server credentials, attempts to access keys by connecting to the key management server:

• Credential rotation when personnel changes limits access to keys

The following threats are not protected against:

• Malicious Swift admin
• Attacker gaining administrator access to Swift servers, in addition to the storage

Motivation

The current inability to rotate keys is a limitation

• PCI DSS requirements state that keys should be rotated every time someone with knowledge of the keys leaves or changes position
• NIST SP 800-57 says symmetric key encryption keys can have crypto periods up to two years, or as short as a day or a week, depending on the number of keys that are encrypted
• Also rotate credentials for accessing keys

For secure, cryptographic deletion, the key(s) to be rotated should be the ones stored on securely erasable media:

• Removable local media
• External key management server (Barbican)

Re-wrapping of old keys could be deferred, since re-wrapping is technically challenging:

• As long as we only use a key to encrypt/wrap for a limited amount of time, we can still decrypt for a while before the key needs to be rewrapped

Storing root secret in local storage on proxy server is not great:

• Pointer to separate config file from proxy-server.conf allows for storing root secret on external media
• Separating key material from rest of config info might make sense, but leads to yet another indirection

Client-managed / bring your own keys:

• Keys on different local media, different external key management servers
• Local media for regular users, HSM-backed Barbican for premium users
• Client-managed Barbican server per account

Details on new features

Support multiple root encryption keys

To support multiple root encryption keys, the following is needed:

• Multiple root encryption keys, e.g., in external key management server; in local configuration file, or; a combination of the two
• Pointer to the current root encryption key to be used to encryption new data
• Pointer to a default root encryption key to be used to decrypt data that does not indicate which key should be used; this is needed for backwards compatibility with current implementation

The following changes would be made in how the keymaster is configured in keymaster.conf:

[keymaster]
current_root_secret=key3
default_root_secret=key1

root_secrets=key1,key2,key3

[key1]
key_material=d2qkoer+g4S+s2tbt1ZKJl9EfMUyMfT9BNdIXU2HI2s=

[key2]
key_material=3uVi489y9sL2GHHzqNgzJk2f0BnIFIC0SQFuvkdElBY=

[key3]
key_material=IIQf3dasef1tiqqXUIOYBP/1HKqPiP6V67JKyr8JYqo=

The metadata would look like this:

X-Object-Sysmeta-Crypto-Body-Meta:
    {"iv": body_iv,
     "cipher": "AES_CTR_256",
     "body_key": {"key": wrapped_body_key,
                  "iv": body_key_iv},
     "root_secret_id": root_secret_id"}

Where:

• root_secret_id is the ID of one of the keys configured in keymaster.conf, e.g., “key3”

Support root encryption keys stored on external key management system

The configuration of external key management system needs the following parameters:

• Username, password, other credentials
• Barbican server host, port
• Keystone server host, port
• etc

The above could be configured in keymaster.conf as follows:

[keymaster]
external_key_server=barbicanserver1
current_root_secret=barbican_key3
default_root_secret=barbican_key1

root_secrets=barbican_key1,barbican_key2,barbican_key3

[barbicanserver1]
server_type = barbican
barbican_username = swift
barbican_password = swift
barbican_project_name = service
barbican_auth_url = http://192.168.50.11:5000/v3

The configurations would be referenced in metadata as follows:

X-Object-Sysmeta-Crypto-Body-Meta:
    {"iv": body_iv,
     "cipher": "AES_CTR_256",
     "body_key": {"key": wrapped_body_key,
                  "iv": body_key_iv},
     "root_secret_id": root_secret_id}

Where:

• root_secret_id is the ID of one of the keys configured in keymaster.conf, e.g., “key3”

The metadata-changes are the same as before, but keymaster needs to know that root_secret_id is in Barbican, and not in local storage

Support multiple instances of key servers

Needed:

• Separation of configuration for servers and keys
• Configuration and metadata needs to point to <server,key> tuple

The changes to how keys are referenced in keymaster.conf:

[keymaster]
key_servers=barbicanserver1,localfile1
current_root_secret=barbicanserver1:key3
default_root_secret=localfile1:key1

[localfile1]
server_type = localfile
key_file = /mnt/usb1/swift_keys1.conf

[barbicanserver1]
server_type = barbican
barbican_username = swift
barbican_password = swift
barbican_project_name = service
barbican_auth_url = http://192.168.50.11:5000/v3

The local keys would be specified as follows in e.g., /mnt/usb1/swift_keys.conf:

[key1]
key_material=d2qkoer+g4S+s2tbt1ZKJl9EfMUyMfT9BNdIXU2HI2s=

[key2]
key_material=3uVi489y9sL2GHHzqNgzJk2f0BnIFIC0SQFuvkdElBY=

The metadata changes would look like this:

X-Object-Sysmeta-Crypto-Body-Meta:
    {"iv": body_iv,
     "cipher": "AES_CTR_256",
     "body_key": {"key": wrapped_body_key,
                  "iv": body_key_iv},
     "root_secret_id": server_id:root_secret_id}

Where:

• root_secret_id is the ID of one of the keys configured in keymaster.conf, e.g., “key3”
• server_id is one of the servers defined in keymaster.conf, e.g., “barbicanserver1”

Future Work

Rotate existing keys:

X-Object-Sysmeta-Crypto-Body-Meta:
    {"iv": body_iv,
     "cipher": "AES_CTR_256",
     "body_key": {"key": wrapped_body_key,
                  "iv": body_key_iv},
     "root_secret_id": server_id:root_secret_id}

• Point to new server_id:root_secret_id
• Replace wrapped_body_key with the existing body_key, wrapped with the new root key

Challenges include eventual consistency, when old key can be deleted

Barcelona Summit Feedback

• Storing the root encryption key on removable media (and removing it after the proxy server has been started) is not advised, since the proxy server may restart/be restarted at any time.
• Look into using Hashicorp Vault for talking to external key management systems (e.g., AWS) and for auditing key access
• Concerns regarding storing additional information (e.g., key unique identifier and server identifier) in object metadata. An alternative would be to have keys per epoch, and maintain an epoch-to-key mapping in the keymaster, so lookups can be performed to find the key to use based on the creation time of an object.
• Storing a key ID in the object metadata could be avoided by storing a wrapped random, immutable key in the container, and use that to derive the object key. Alternatively, the immutable key could be at account level, with the container and object keys being derived from that, similarly to how a/c/o keys are being derived from the root secret now. With these approaches, the benefits from key rotation would be greater than simply rotating the root secret, but key rotation at only account, or account and container level would be more manageable than rewrapping the object keys (just the object key encryption keys, never the body keys used to encrypt the actual object data).
• Concerns about conditionally importing the castellan/barbican modules in a function in the existing keymaster; an alternative approach would be to have a separate, new keymaster.
• If adopting a bring-your-own-key (BYOK) approach similar to the one used by Amazon, what happens if the client does not provide a key, but the Swift proxy server has encryption enabled? In this case, we should either use the cluster key to encrypt, or we should always encrypt with the cluster key, and if a BYOK is provided, then there would be two layers of encryption.
• If no account or container key exists, and two clients create the account/container key at once (on different proxy servers), we need to be able to store both keys, since we may end up with one object encrypted with one key (or a key derived from one key), and another object encrypted with another key.
• Do not add support for only add new, rotated keys, without at the same time adding support for cleaning up and removing no longer used keys.

Contact: mathiasb on IRC.