Latest revision as of 12:05, 20 November 2013

Server Side Encryption

Abstract

The general scheme is to create a swift proxy middleware that will encrypt and sign the object data during PUT and check the signature + decrypt it during GET. The target is to create two domains - the user domain between the client and the middleware where the data is decrypted and the system domain between the middleware and the data at rest (on the device) where the data is encrypted.

Design goals include: (1) Extend swift as necessary but without changing existing swift behaviors and APIs; (2) Support encrypting data incoming from unchanged clients

Top Level Design

The design described here encrypts once at the proxy. To enable encryption, the admin needs to add the encryption middleware to the pipeline. Support is given to non greenfield installations by:

1. A container would either be created as an encrypted container or a non encrypted container and will its encryption bit will not change throughout the life of the container.
2. A PUT object would result in encrypting and signing the object data by the middleware if the container is marked for encryption.
3. If middleware is removed, keys would not be exposed.
4. Each object would have its own key for encrypting the data.
5. The object key of 'AUTH_myaccount/mycontainer/myobject' would be stored as system metadata x-object-sysmeta-key-XXX where XXX is base64(hash('AUTH_myaccount/mycontainer')) with a value of enc(containerkey, objkey) - i.e. objkey encrypted using the container key. This would allow COPY operation without decrypting objects as described below. The hash used would be the Swift system hash (MD5 be default).
6. Non encrypted objects would have no x-object-sysmeta-key-* metadata.
7. New objkey would be randomly chosen during PUT object
8. new containerkey, accountkey would be randomly chosen during the creation of the container/account respectively.
9. The container key is stored as system metadata x-conaitner-sysmeta-key with a value of enc(accountkey, containerkey) - i.e. containerkey encrypted using the account key.
10. The decrypted container key is cached together with the container metadata in memcache (may require enhancing Swift core or maintaining a separate cache space in memcache)
11. The account key is stored as system metadata x-account-sysmeta-key with a value of enc(masterkey, accountkey) - i.e. accountkey encrypted using the master key (master key is stored per account by the key manager, e.g. Barbican).
12. The decrypted account key may be optionally cached together with the account metadata in memcache (requires enhancing Swift core)
13. Manifests are never encrypted
14. When an object is encrypted, it is divided to blocks where each block is encrypted than signed. When an object is decrypted each block signature is verified before decryption takes place.
15. Range queries are supported by decrypting only the blocks that include the relevant offset and sending only the data requested.

Design Details

Etag issues

Current Swift behavior:

• During PUT object, at the object server, while the object server writes the chunks to the DiskFile it computes an MD5 checksum of the chunks.
• If the proxy had sent an etag header, the object server will compare the computed etag to the one sent by the proxy. If the etag mismatch, the object server return with HTTPUnprocessableEntity... in which case the etag metadata would not be stored as metadata of the object which presumably would result in the object be discarded at some future time via the auditor.
• else: the etag is stored under the Etag metadata key of the object.

We name here the MD5 of the stored object 'de-etag' (for decrypted etag) and the MD5 of the encrypted object 'en-etag'. The en-etag is stored in the etag-field of the object allowing auditors to continue working unchanged. This is achieved by removing the etag provided by the client (if provided) and thus allowing the object server to calculate the en-etag field and store it.

During a PUT, the middleware is tasked with:

1. Calculating the de-etag while chunks are sent to the object server
2. If the client provided etag, comparing the etag to the calculated de-etag, if not matched responding with HTTPUnprocessableEntity(?) to the client. This would leave an encrypted object without x-object-sysmeta-etag in the system which should be ignored during a GET/HEAD. Eventual consistency should be added to resolve this issue (updating the auditors to discard encrypted objects without x-object-sysmeta-etag)
3. Else: perform a POST and store the de-etag under x-object-sysmeta-etag. If the proxy fails after the PUT and before the POST succeeded, the eventual consistency discussed above would resolve the issue.

During a HEAD, the middleware would send the x-object-sysmeta-etag as the etag (or would indicate that the object does not exist if the x-object-sysmeta-etag is missing).

Another issue with Etags is that they are also stored and reported as part of Container listing. The etag provided during container listing should be the de-etag one. To achieve that, the object server needs to transfer the value in the 'x-object-sysmeta-etag' object metadata to the container server instead of the one in the 'Etag' object metadata (Note that the de-etag becomes available at the object server during the POST and is not available during the PUT).

Size issues

Similar to the Etag issues above, the object size is also transformed when the object is stored.

We name here the size of the stored object 'de-size' (for decrypted size) and the size of the encrypted object 'en-size'. The en-size is stored in the size-field of the object allowing auditors to continue working unchanged.

During a PUT, the middleware is tasked with:

1. Calculating the size of the encrypted and signed object.
2. Than perform a POST and store the de-size under x-object-sysmeta-size. If the proxy fails after the PUT and before the POST succeeded, the eventual consistency discussed under 'etag issues' subsection above would resolve the issue.

During a HEAD, the middleware would send the x-object-sysmeta-size as the etag (or would indicate that the object does not exist if the x-object-sysmeta-etag is missing).

Another issue with object sizes is that they are also stored and reported as part of Container listing. The size provided during container listing should be the de-size one. To achieve that, the object server needs to transfer the value in the 'x-object-sysmeta-size' object metadata to the container server instead of the one in the 'Size' object metadata (Note that this de-size becomes available at the object server during the POST and is not available during the PUT).

Keys

As discussed above, each object data and user metadata is encrypted with its own encryption key named here objkey. The objkey is randomly created during an object PUT and is stored in an encrypted form as part of the object system metadata under x-object-sysmeta-key-XXX where XXX is the base64(hash('AUTH_myaccount/mycontainer')). This unusual key name structure was chosen in order to allow support in COPY operations in Swift. During a copy operation the object may move between accounts and containers and as a result, the objkey would need to be re-encrypted. As the object is being copied it requires access via two separate path under two separate master keys etc. The same object may be copied repeatedly. in order to overcome the eventual consistency issues that may result with the need to update the key of an object after it was copied or to introduce changes into swift, it is suggested to use the above unusual key name thus allowing each path to maintain an individual copy of the objkey in the object system metadata. See COPY below for more details.

Note that the objkey is never changed and the object is never re-encrypted during COPY operations or during master key changes. The objkey is stored in encrypted form as the value of the object sysmeta key x-object-sysmeta-key-XXX. The objkey is encrypted with the key of the container. The decrypted container key would be cached using memcache to enable rapid access to multiple objects of the same container at minimal overhead.

The container key is randomly chosen when the container is created and is stored in the container system metadata field x-container-sysmeta-key after being encrypted by the account key. The decrypted account key would be cached using memcache to enable rapid access to multiple containers of the same account at minimal overhead. The container key is not reencrypted during master key changes.

The account key is randomly chosen when the account is created and is stored in the account system metadata field x-account-sysmeta-key after being encrypted by the master key of the account owner. The master key is retrieved from a key manager to decrypt the account key and is never cached. The account key is not reencrypted during master key changes.

The account master key is stored by Barbican or an alternative key manager.

Signatures

The Object is stored as a series of blocks where each block is encrypted than signed. Before decrypting a block of an object, its signature is verified.

Blob Encryption

The original plan is to use M2Crypto library for the crypto operations. Yet, M2Crypto has a BSD license. pyCrypto seems more suitable and is used in Keystone.

It is TBD if to make the library pluggable and which library to use by default.

API Implementation

During a PUT account or container the middleware:

1. Choose a random key and would set the sysmeta to store the key

During a PUT object the middleware:

1. Look at the sysmeta of the container: if x-container-sysmeta-enc-enabled is missing or False perform regular swift core PUT; else:
2. Calculate the en-size and update the object headers size accordingly
3. Remove the user provided etag header if supplied
4. Choose a random key
5. Encrypt the key with container key and set the x-object-sysmeta-key-XXX with the value of the encrypted objkey;
6. For each pre-sized block of data (or less if last block): (1) calculate de-etag, (2) encrypt the data with objkey (3) sign the block (4) send the signed than encrypted block to swift core;
7. After last block, if swift core is successful, compare de-etag to client provided etag and fail the request if not matching.
8. Otherwise, update x-object-sysmeta-key-etag with de-etag and x-object-sysmeta-key-size with de-size using POST.
9. Object server updates the container listing using the data in x-object-sysmeta-key-etag and x-object-sysmeta-key-size

During a HEAD object the middleware:

1. Head the object, decrypt the objkey
2. Replace the etag with the value from x-object-sysmeta-etag
3. Replace the size with the value from x-object-sysmeta-size

During a GET object the middleware:

1. GET the object, decrypt the objkey
2. Replace the etag with the value from x-object-sysmeta-etag
3. Replace the size with the value from x-object-sysmeta-size
4. Per block: (1) check the signature of the block - if failed, fail the response. (2) Decrypt the block and send the decrypted data to the client

During a POST object the middleware:

1. Perform regular Swift core POST

During a COPY the middleware:

1. Decrypt the objkey based on the source container key
2. Encrypt the objkey with the destination container key
3. Store the new encrypted objkey under the appropriate x-object-sysmeta-key-XXX using POST (using the new path of the destination container)
4. Perform regular Swift core COPY

Unresolved issues

1. Each PUT REST request is translated by the middleware to two separate internal REST calls at the middleware: PUT + POST to update the de-etag and the consistency signature. If the cluster is configured with copy on POST enabled, this result in a copy of each encrypted object. If many objects are encrypted, this would effect the cluster performance as it would triple the i/o used during each encrypted PUT operations. Short term solution: do not use both encryption and copy on POST on the same system if the system is required to encrypt many of its objects. Long term solution: remove the need to do copy on post by fixing Swift.

Future

1. Object user metadata keys and values are base64 encrypted using the objkey and sent as user metadata: E.g. 'x-object-meta-confidentiality: Top-Secret' could become 'x-object-meta-Y29uZmlkZW50aWFsaXR5: VG9wLVNlY3JldA==' (I used null encoding in this example).
2. Consider what else needs to be encrypted/MACed - e.g. should we encrypt the x-object-sysmeta-etag ?
3. Encrypt container names
4. Encrypt/MAC User MD on a container