Swift/server-side-enc

Server Side Encryption

Abstract

This wiki page summarizes design aspects and insights into Server Side Encryption for Swift. The general scheme is to create a middleware that will encrypt the object data during PUT and decrypt it during GET. The target is to create two domains - the user domain between the client and the middleware where the data is decrypted and the system domain between the middleware and the data at rest (on the device) where the data is encrypted.

A design goal is to extend swift as necessary but to not change existing swift behaviors.

First level design highlights

The design described here encrypts once at the proxy. To enable encryption, the admin needs to add the encryption middleware to the pipeline. Support is given to non greenfield installations by:

1. Once the middleware is added, the middleware would be able to encrypt new objects and keep old objects non encrypted
2. A PUT object would result in encrypting the object data by the middleware if the container is marked for encryption. If an existing container is marked for encryption, existing objects that area already in the system would remain unencrypted (unless a new version is uploaded with PUT).
3. Encrypted objects would be decrypted even if the container is not marked for encryption
4. If middleware is removed, keys would not be exposed, new objects would be stored unencrypted, encrypted objects would not be decrypted. Adding the middleware would resume the previous state.
5. Each object would have its own key for encrypting the data.
6. The object key of 'AUTH_myaccount/mycontainer/myobject' would be stored as system metadata x-object-sysmeta-key-XXX where XXX is base64(hash('AUTH_myaccount/mycontainer')) with a value of enc(containerkey, objkey) - i.e. objkey encrypted using the container key. This would allow COPY operation without decrypting objects as described below. The hash used would be the Swift system hash (MD5 be default).
7. Non encrypted objects would have no x-object-sysmeta-key-* metadata.
8. New objkey would be randomly chosen during PUT object
9. new containerkey, accountkey would be randomly chosen during the creation of the container/account respectively and will never be changed.
10. The container key is stored as system metadata x-conatiner-sysmeta-key with a value of enc(accountkey, containerkey) - i.e. containerkey encrypted using the account key.
11. The decrypted container key is cached together with the container metadata in memcache (requires enhancing Swift core)
12. The account key is stored as system metadata x-account-sysmeta-key with a value of enc(masterkey, accountkey) - i.e. accountkey encrypted using the master key (master key is stored per account by the key manager).
13. The decrypted account key is cached together with the account metadata in memcache (requires enhancing Swift core)

Server Side Encryption Details

Etag issues

Current Swift behavior:

• During PUT object, at the object server, while the object server writes the chunks to the DiskFile it computes an MD5 checksum of the chunks.
• If the proxy had sent an etag header, the object server will compare the computed etag to the one sent by the proxy. If the etag mismatch, the object server return with HTTPUnprocessableEntity... in which case the etag metadata would not be stored as metadata of the object which presumably would result in the object be discarded at some future time via the auditor.
• else: the etag is stored under the Etag metadata key of the object.

We name here the MD5 of the stored object 'de-etag' (for decrypted etag) and the MD5 of the encrypted object 'en-etag'. The en-etag is stored in the etag-field of the object allowing auditors to continue working unchanged. This is achieved by removing the etag provided by the client (if provided) and thus allowing the object server to calculate the en-etag field and store it.

During a PUT, the middleware is tasked with:

1. Calculating the de-etag while chunks are sent to the object server
2. If the client provided etag, comparing the etag to the calculated de-etag, if not matched responding with HTTPUnprocessableEntity(?) to the client. This would leave an encrypted object without x-object-sysmeta-etag in the system which should be ignored during a GET/HEAD. Eventual consistency should be added to resolve this issue (updating the auditors to discard encrypted objects without x-object-sysmeta-etag)
3. Else: perform a POST and store the de-etag under x-object-sysmeta-etag. If the proxy fails after the PUT and before the POST succeeded, the eventual consistency discussed above would resolve the issue.

During a HEAD, the middleware would send the x-object-sysmeta-etag as the etag (or would indicate that the object does not exist if the x-object-sysmeta-etag is missing).

Keys

Each object is encrypted with its own encryption key ,which is stored encrypted under a container key as part of a private object meta data field. This requires us to add support for such key in the Object Server to avoid the key being deleted during an object POST.

The container key in turn is stored in its a private metadata field after being encrypted under an account key. The encrypted container key is therefore cached. Q: do we want to cache the unencrypted key to improve performance? If we do so we do not need to cache the account master key.

The account key in turn is stored in its a private metadata field after being encrypted under the account master key. The encrypted account key is therefore cached. Q: do we want to cache the unencrypted key to improve performance? If we do so we do not need to cache the account master key. Q: do we want to remove this extra level of indirection and have the container keys encrypted by the account master key?

The account master key is stored by barbican or an alternative key manager. If we do not cache the unencrypted account key or the unencrypted container key we would need to cache the account master key to achieve reasonable performance.

On account master key change, we never re-encrypt the bulk data, but only re-rencrypt the (account/container) keys.

Consistency and Signitures

When decrypting objects, we do consistency check to make sure we are using the right key and internal checksums match; TODO - add a proper encrypt-and-mac to prevent any modifications of the objects.

Blob Encryption

The M2Crypto library is used for the crypto operations.

Unresolved issues

1. Support in COPY- it appears that support in COPY requires either reimplementing COPY in middleware or to modify proxy implementation e.g. by having the proxy call the encryption middleware to decrypt and encrypt the data during COPY
2. ...

Observations

• Activating the optional signature feature results in each PUT REST request being translated by the middleware to two separate internal REST calls at the middleware: PUT + POST. If the cluster is configured with copy on POST enabled, this result in a copy of each encrypted object. If many objects are encrypted, this would effect the cluster performance as it would tripe the i/o used during each encrypted PUT operations. It is recommended to not use encryption and copy on POST on a system that is required to encrypt many of its objects.