OpenStack - User contributions [en]

Swift/ideas/keymaster v2

2016-11-01T08:41:52Z

Mbj: Added feedback from Barcelona summit design discussion

= Keymaster v2 =

The original keymaster uses a single root encryption secret stored in the proxy server configuration file to generate keys for the accounts, containers and objects, based on the path to the account/container/object.
 

== Summary of features to be added to the keymaster ==

This design aims to add the following features to the keymaster:
 

=== Support multiple root encryption keys ===

Supporting multiple root encryption keys includes:

* Ability to switch to a new root encryption key for encrypting new objects
* Do not implement re-keying/re-wrapping of existing data, but
* Design and implement changes in a way that would allow for re-keying later
 

=== Support root encryption keys stored in an external key management server ===

* In practice, a Barbican server accessed using the Castellan library
* Implementation and tests structured such that someone who does not want Barbican, does not need to install dependencies (e.g., oslo)
 

=== Support multiple instances of key servers ===

* Multiple locations for locally stored root secrets
* Multiple external Barbican servers
 

== System Model ==

[[File:Swift Encryption KeyMaster v2-system model.png|none|left|System model]]

The assumptions regarding the swift storage system are as follows:

* Current Swift administrator is trusted
* Encryption key storage is trusted and securely erasable, e.g., a key management server backed by an HSM, or removable storage plugged into proxy server
* Servers (proxy, account, container, object) are trusted, including the processes running in the servers, memory
* All storage is untrusted
 

=== Attack scenarios ===

The following attack scenarios are considered and protected against:
 

Gaining access to decommissioned disks from proxy, account, container, object servers:
* Encryption keys stored wrapped in the metadata, data not accessible
* Brute-forcing the encryption keys takes a long time
* Key rotation can reduce exposure in case a key is brute-forced
 

Attacker is a former Swift admin with key server credentials, attempts to access keys by connecting to the key management server:
* Credential rotation when personnel changes limits access to keys
 

The following threats are not protected against:
* Malicious Swift admin
* Attacker gaining administrator access to Swift servers, in addition to the storage

== Motivation ==

The current inability to rotate keys is a limitation
* PCI DSS requirements state that keys should be rotated every time someone with knowledge of the keys leaves or changes position
* NIST SP 800-57 says symmetric key encryption keys can have crypto periods up to two years, or as short as a day or a week, depending on the number of keys that are encrypted
* Also rotate credentials for accessing keys
 

For secure, cryptographic deletion, the key(s) to be rotated should be the ones stored on securely erasable media:
* Removable local media
* External key management server (Barbican)
 

Re-wrapping of old keys could be deferred, since re-wrapping is technically challenging:
* As long as we only use a key to encrypt/wrap for a limited amount of time, we can still decrypt for a while before the key needs to be rewrapped
 

Storing root secret in local storage on proxy server is not great:
* Pointer to separate config file from proxy-server.conf allows for storing root secret on external media
* Separating key material from rest of config info might make sense, but leads to yet another indirection
 

Client-managed / bring your own keys:
* Keys on different local media, different external key management servers
* Local media for regular users, HSM-backed Barbican for premium users
* Client-managed Barbican server per account

== Details on new features ==

=== Support multiple root encryption keys ===

To support multiple root encryption keys, the following is needed:
* Multiple root encryption keys, e.g., in external key management server; in local configuration file, or; a combination of the two
* Pointer to the current root encryption key to be used to encryption new data
* Pointer to a default root encryption key to be used to decrypt data that does not indicate which key should be used; this is needed for backwards compatibility with current implementation

The following changes would be made in how the keymaster is configured in keymaster.conf:
<pre>[keymaster]
current_root_secret=key3
default_root_secret=key1

root_secrets=key1,key2,key3

[key1]
key_material=d2qkoer+g4S+s2tbt1ZKJl9EfMUyMfT9BNdIXU2HI2s=

[key2]
key_material=3uVi489y9sL2GHHzqNgzJk2f0BnIFIC0SQFuvkdElBY=

[key3]
key_material=IIQf3dasef1tiqqXUIOYBP/1HKqPiP6V67JKyr8JYqo=</pre>

The metadata would look like this:
<pre>X-Object-Sysmeta-Crypto-Body-Meta:
{"iv": body_iv,
"cipher": "AES_CTR_256",
"body_key": {"key": wrapped_body_key,
"iv": body_key_iv},
"root_secret_id": root_secret_id"}</pre>

Where:
* ''root_secret_id'' is the ID of one of the keys configured in keymaster.conf, e.g., “key3”

=== Support root encryption keys stored on external key management system ===

The configuration of external key management system needs the following parameters:
* Username, password, other credentials
* Barbican server host, port
* Keystone server host, port
* etc

The above could be configured in keymaster.conf as follows:
<pre>[keymaster]
external_key_server=barbicanserver1
current_root_secret=barbican_key3
default_root_secret=barbican_key1

root_secrets=barbican_key1,barbican_key2,barbican_key3

[barbicanserver1]
server_type = barbican
barbican_username = swift
barbican_password = swift
barbican_project_name = service
barbican_auth_url = http://192.168.50.11:5000/v3</pre>

The configurations would be referenced in metadata as follows:
<pre>X-Object-Sysmeta-Crypto-Body-Meta:
{"iv": body_iv,
"cipher": "AES_CTR_256",
"body_key": {"key": wrapped_body_key,
"iv": body_key_iv},
"root_secret_id": root_secret_id}</pre>

Where:
* ''root_secret_id'' is the ID of one of the keys configured in keymaster.conf, e.g., “key3”

The metadata-changes are the same as before, but keymaster needs to know that ''root_secret_id'' is in Barbican, and not in local storage

=== Support multiple instances of key servers ===

Needed:
* Separation of configuration for servers and keys
* Configuration and metadata needs to point to <server,key> tuple

The changes to how keys are referenced in keymaster.conf:
<pre>[keymaster]
key_servers=barbicanserver1,localfile1
current_root_secret=barbicanserver1:key3
default_root_secret=localfile1:key1

[localfile1]
server_type = localfile
key_file = /mnt/usb1/swift_keys1.conf

[barbicanserver1]
server_type = barbican
barbican_username = swift
barbican_password = swift
barbican_project_name = service
barbican_auth_url = http://192.168.50.11:5000/v3</pre>

The local keys would be specified as follows in e.g., /mnt/usb1/swift_keys.conf:
<pre>[key1]
key_material=d2qkoer+g4S+s2tbt1ZKJl9EfMUyMfT9BNdIXU2HI2s=

[key2]
key_material=3uVi489y9sL2GHHzqNgzJk2f0BnIFIC0SQFuvkdElBY=</pre>

The metadata changes would look like this:
<pre>X-Object-Sysmeta-Crypto-Body-Meta:
{"iv": body_iv,
"cipher": "AES_CTR_256",
"body_key": {"key": wrapped_body_key,
"iv": body_key_iv},
"root_secret_id": server_id:root_secret_id}</pre>

Where:
* ''root_secret_id'' is the ID of one of the keys configured in keymaster.conf, e.g., “key3”
* ''server_id'' is one of the servers defined in keymaster.conf, e.g., “barbicanserver1”

== Future Work ==

Rotate existing keys:
<pre>X-Object-Sysmeta-Crypto-Body-Meta:
{"iv": body_iv,
"cipher": "AES_CTR_256",
"body_key": {"key": wrapped_body_key,
"iv": body_key_iv},
"root_secret_id": server_id:root_secret_id}</pre>

* Point to new ''server_id:root_secret_id''
* Replace ''wrapped_body_key'' with the existing body_key, wrapped with the new root key

Challenges include eventual consistency, when old key can be deleted

== Barcelona Summit Feedback ==

* Storing the root encryption key on removable media (and removing it after the proxy server has been started) is not advised, since the proxy server may restart/be restarted at any time.
* Look into using Hashicorp Vault for talking to external key management systems (e.g., AWS) and for auditing key access
* Concerns regarding storing additional information (e.g., key unique identifier and server identifier) in object metadata. An alternative would be to have keys per epoch, and maintain an epoch-to-key mapping in the keymaster, so lookups can be performed to find the key to use based on the creation time of an object.
* Storing a key ID in the object metadata could be avoided by storing a wrapped random, immutable key in the container, and use that to derive the object key. Alternatively, the immutable key could be at account level, with the container and object keys being derived from that, similarly to how a/c/o keys are being derived from the root secret now. With these approaches, the benefits from key rotation would be greater than simply rotating the root secret, but key rotation at only account, or account and container level would be more manageable than rewrapping the object keys (just the object key encryption keys, never the body keys used to encrypt the actual object data).
* Concerns about conditionally importing the castellan/barbican modules in a function in the existing keymaster; an alternative approach would be to have a separate, new keymaster.
* If adopting a bring-your-own-key (BYOK) approach similar to the one used by Amazon, what happens if the client does not provide a key, but the Swift proxy server has encryption enabled? In this case, we should either use the cluster key to encrypt, or we should always encrypt with the cluster key, and if a BYOK is provided, then there would be two layers of encryption.
* If no account or container key exists, and two clients create the account/container key at once (on different proxy servers), we need to be able to store both keys, since we may end up with one object encrypted with one key (or a key derived from one key), and another object encrypted with another key.
* Do not add support for only add new, rotated keys, without at the same time adding support for cleaning up and removing no longer used keys.

Contact: mathiasb on IRC.

Swift/ideas

2016-10-19T14:05:37Z

Mbj: Adding link to keymaster v2 page

= Ideas for OpenStack Swift =

This page is a collection of "brain dumps" for ideas about features in OpenStack Swift. If you're working on something, it's a very good idea to write down what you're thinking about. This lets others get up to speed, helps you collaborate, and serves as a great record for future reference. Write down your thoughts somewhere and put a link to it here. It doesn't matter what form your thoughts are in; use whatever is best for you. Your document should include why your idea is needed and your thoughts on particular design choices and tradeoffs. Please include some contact information (ideally, your IRC nick) so that people can collaborate with you.

''Historic "specs" are available at https://specs.openstack.org/openstack/swift-specs/''

== Ideas ==

Format: '''Idea''' -- ''link to your brain dump''

* '''Small file optimization''' -- [[Swift/ideas/small files]]
* '''Reduce memcache lookups''' -- [[Swift/ideas/memoize lookups]]
* '''Improve internal network security''' -- [[Swift/ideas/network_security]]
* '''Metrics around rate-limiting''' -- [[Swift/ideas/ratelimiting_metrics]]
* '''Hierarchical keymaster''' -- [[Swift/ideas/hierarchical_keymaster]]
* '''sync_method = repconn''' -- https://etherpad.openstack.org/p/hummingbird-replication-upgrade
* '''Container Sharding''' -- [https://docs.google.com/document/d/1-0wUnaRf2TTA0KQGk301BPR8-5wzuyA1r4tLSEh-cbw/edit?usp=sharing container-sharding.odt] [https://etherpad.openstack.org/p/container-sharding-SAT-2016 Etherpad notes] [https://trello.com/b/8p2iJ9RR/swift-container-sharding trello] [https://docs.google.com/document/d/16rPdDrT-4G6hcen_6no8cr8KbxUqbcvsSkOfxxu0EBE/edit?usp=sharing Initial locking db shard approach idea]
* '''High-latency media / Tape support for Swift''' -- [[Swift/HighLatencyMedia]]
* '''The Archival Storage extension to the Swift API''' -- [https://etherpad.openstack.org/p/high-latency-storage-policy]
* '''Symlinks''' -- https://etherpad.openstack.org/p/swift_symlinks
* '''Metadata index (Elasticsearch)''' -- [[Swift/ideas/metadata-sync]]
* '''Keymaster v2''' -- [[Swift/ideas/keymaster_v2]]

----

[[Category:ObjectStorage]]

Swift/ideas/keymaster v2

2016-10-19T14:03:51Z

Mbj: Swift keymaster v2 proposal for discussion at Barcelona design summit

File:Swift Encryption KeyMaster v2-system model.png

2016-10-19T13:54:37Z

Mbj:

Swift keymaster v2 system model

Swift/ideas

2016-07-05T14:02:24Z

Mbj: Adding link to hierarchical keymaster page

Swift/ideas/hierarchical keymaster

2016-07-05T14:01:05Z

Mbj: Initial version of hierarchical keymaster for swift

= Hierarchical Keymaster =

The encryption functionality recently merged into the swift master branch has the following features:

* Keymaster providing the keys, encrypter and decrypter functionality in the encryption filter in the wsgi pipeline
* Static root secret in the proxy-server.conf file
* Encryption keys for containers and objects derived using the root secret and the path of the container and object
* Randomly generated body key used to encrypt the actual object data, and body key wrapped with object key provided by keymaster
* Container and object keys provided by the keymaster for encrypting metadata

As an extension to this encryption functionality, hereafter known as the base functionality, this proposal contains a high-level design of a hierarchical
keymaster that provides functionality such as:

* Root secret stored in an external key management system such as Barbican
* Randomly generated data encryption keys (DEKs) for accounts, containers, objects (commonly referred to as entities), which are provided by the hierarchical keymaster to the encrypter and decrypter functionality provided by the base. The DEKs are generated once at creation time of the entities, and are never changed.
* Randomly generated key encryption keys (KEKs) that are used to wrap the DEKs, and which are stored in the metadata of the entity that they protect.
* Each KEK is wrapped by the KEK in the next-highest level in the hierarchy (e.g., an object KEK is wrapped by the KEK of its container), with the account KEK being wrapped with the root secret.
* Key rotation can be performed on the KEKs as a way of providing cryptographic, secure deletion of user data.

The hierarchical keymaster is designed to be a drop-in replacement for the base keymaster when configuring an encryption environment - in particular, the base encryption and decryption functionality, along with which data is and is not to be encrypted, will not change.

The work on the functionality described above can be separated into three parts:

# Support for storing a root secret key in Barbican, while still deriving the other keys (container, object) based on the path, as in the base implementation.
# Root secret in Barbican, and randomly generated KEKs and DEKs stored in the entity metadata.
# Support for rotating the root secret and KEKs (secure deletion)

Some of the main challenges that have been identified in earlier discussions are:

* Using client-controlled keys, e.g., in Barbican, while preserving the API. This impacts e.g., container-sync and other background processes, which will not have the necessary client credentials required to access the key(s).
* Using random keys, since for some scenarios, it is then necessary to retrieve metadata from the object server (using "HEAD"), which incurs a performance penalty. In particular, this impacts update operations ("POSTs") and conditional GETs.
* Eventual consistency brings challenges to the key rotation, since it is difficult, or even impossible, to conclude that an object is no longer encrypted with an old key.

== Root secret in Barbican ==

The first step in moving towards a hierarchical keymaster with support for key rotation, is providing a mechanism for storing the root secret somewhere else than in the proxy-server.conf configuration file. The base keymaster is replaced with a new secdel_keymaster in the WSGI pipeline, and the system also needs to be configured with a Barbican URL, e.g.:

<pre>... secdel_keymaster encryption proxy-logging proxy-server

[filter:secdel_keymaster]
use = egg:swift#secdel_keymaster
barbican_url = http://192.168.50.11:9311/v1</pre>
Internally, the secdel_keymaster retrieves a root secret key from Barbican, or if one does not exist, one is created and stored in Barbican. The lookup is done using a name generated from, among others, the project ID.

Alternatively, the account key could (also) be stored in Barbican, which may facilitate future moves to BYOK.

=== Considerations ===

For a BYOK deployment, the Barbican key could be one provided by the user(s), in which case it could still be a root secret (in case there is one user per cluster), or an account secret (for a one user per account scenario). With BYOK, additional questions regarding the configuration, addition of new API calls and/or modification of existing ones etc. need to be answered.

== Randomly generated keys ==

The next step is using randomly generated keys, both for the DEKs provided by the secdel_keymaster to the encrypter and decrypter, as well as for the KEKs that are used to wrap the DEKs and the lower-level KEKs. The pair KEK-DEK is termed an entity key, and an entity key also contains the parent KEK ID, i.e., the ID of the key used to wrap the KEK of the entity key. The ID of the entity keys are uunique identifiers in the UUID1 format. In particular, the ID needs to have the following properties:

* Be unique, so that e.g., multiple proxy nodes do not need to coordinate for creating the IDs, which would be the case for IDs based on incrementing counters.
* Contain a timestamp, so that it can be determined which is the most recent key.

The keys are stored in the metadata of the entity in the following secdel-metadata format:

<pre>X-<Entity>-[Transient-]-Sysmeta-Secdelkeywrapinfo:
{ uuid1 :
{ "kek_wrapping_key_id" : wrapping_key_uuid1,
"kek_payload" : wrapped_kek_material1,
"dek_payload" : wrapped_dek_material1
},
uuid2 :
{ "kek_wrapping_key_id" : wrapping_key_uuid2,
"kek_payload" : wrapped_kek_material2,
"dek_payload" : wrapped_dek_material2
}
}</pre>
Where uuidN is the unique identifier of the Nth entity key. In addition to using IDs to identify which wrapping keys are used for wrapping and unwrapping, the wrapping algorithm itself also should contain integrity checking so that attempts to unwrap a key with the wrong wrapping key are caught already at the time of unwrapping, and the error is not propagated to the end user in the form of corrupted user data. This can be achieved e.g., by using AES key wrap (see RFC3394).

=== Key Generation for Encryption ===

When creating a new entity, an entity key is generated. The KEK of the entity key is chosen to be KEK of the most recent entity key of the next-higher level.

=== Key Retrieval for Decryption ===

When retrieving an entity, the secdel-metadata is retrieved and the kek wrapping key ID is read to determine the ID of the KEK used to wrap the entity KEK. This is done for all higher-level entities, all the way up to the unwrapped root secret. Once all the necessary entity keys have been gathered, the unwrapping is performed from the root secret all the way down to the entity being retrieved.

=== Considerations ===

The secdel-metadata from the response headers are needed by the secdel-keymaster to be able to determine the wrapping keys and to retrieve the key material necessary for unwrapping the entity DEKs for providing to the decrypter. The secdel-metadata is provided through the callback function to the keymaster, which is called by the decrypter to retrieve the keys.

== Key rotation ==

Key rotation of the root secret and entity KEKs is to be used to be able to securely delete user data. To rotate the root secret and all KEKs, effectively securely deleting all user data whose keys are not rotated, the following actions are performed:

<pre>Create a new root secret
For each account:
Unwrap the DEK from the latest entity key
Create a new account KEK, use it to wrap the DEK
Wrap the new account KEK with the new root secret
Store the new account entity key along the existing ones in secdel-metadata
Repeat for each container, and each object in containers</pre>
During the key rotation process, entities are accessible using either new or old keys. Once the key rotation process is complete, the old keys can be deleted from the secdel-metadata and Barbican.

=== Considerations ===

The following points need to be addressed for the key rotation:

* How to efficiently update secdel-metadata in all entities?
* How to know when the key rotation process (creation of new keys, deletion of old keys) has completed?
* How to trigger the (starting, finalization) key rotation (background) process?
* How is the access control/account information provided to the (background) process that rotates the keys?

Contact: mathiasb, kmARC, jrichli on IRC