• ImageNotes

Nova Image Notes

Image Implementation In a Nutshell

The system currently provides two backends for retrieving images:

• Node-local Store (my term) - A directory on the node where images
  • reside. Images in node-local storage should already in raw format (not AMI)
• S3 Object Store - Currently this is an fake which provides an S3 ReST
  • interface to the filesystem

The Node-local Store appears to be a convenience for testing and perhaps for use by an Ops team when it goes live, whereas, S3 provides a registration mechanism (with hints at Image sharing) that would be exposed to customers.

An Unregistered Image is mapped to a bucket in the S3 object store. Within the bucket is:

• The encrypted AMI
• A JSON manifest (containing the decryption keys)

The Image is Registered by calling endpoints/node.py/register() which in turn issues a PUT request to the object store. Contained within the PUT request is:

• Image Location: account and bucket for the image
• Image ID: unique identifier

The S3-ObjectStore handler (objectstore/handler.py) receives the PUT request and forks of a process to continue the registration process (Image.create found in objectstore/image.py) which includes:

• Fetching the image from Image Location
• Decrypting the Image using the keys provided by the manifest
• Untarring the AMI contents
• Moving the Image to the destination directory for serving using the ReST
  • API

The state of the Image is contained with the JSON manifest which is updated as the image progresses through the unpacking process. Eventually the image reaches the 'available' status and is said to be 'registered'.

Now that the image is registered, the spawn method of compute/node.py can use curl to fetch the image from the objectstore (util.fetchfile) to instantiate a new VM.

It should be noted both the consumer (node.py) and the provider (Image API/objectstore) think of images as having a type:

• Kernel
• Ramdisk
• Image

So, in order spawn a VM Nova you really need to specify *three* images, one of each type.

What the Implementation Gets Right

• The Node-local Store - useful for testing and debugging, plus, could be
  • useful for OPs teams when the system goes live
• Separates the Image API (register/deregister) from Instance API
  • (create/detroy VMS)
• Using fakes for the backends of the object stores.

Where the Implemenation Needs Work

• Registering AMI hits disk at least 3 times (fetch, decrypt, untar). For
  • large images this could be painful.
• The Image registraion API is missing key features (image sharing,
  • metadata, etc)

What the Implmentation Gets Wrong

• Amazon-stack specific: The Image API is coupled to S3 for the object
  • store and AMI's as the image format
• No Caching - each Spawn would require a fetch from Image API
• Does not use OVF container - some of the JSON manifest data would make
  • sense in the manifest.ovf file (others parts would not)
• Conflates two different types of metadata: Image metadata which
  • describes the VM (architecture, etc) versus Image metadata which describes the Image itself (isPublic, isShared, image location). I would argue that these are fundamentally different and make sense stored in two different places. The important question being: does this metadata make sense *outside* of our system.

Ideas for Improvement

* Use Pyfilesystem to store Images. This would provide support for:

• S3

• CloudFiles (once the plugin is written, I'm working on that now)

• Local disk storage
• Memory storage (this could be great for unit-testing)

* Add a caching layer. This could be:

• A separate caching utility
• Part of Pyfilesystems caching layer
• Some combination of both

* Create robust Image registry. This would:

• Backed by Cassandra or Redis
• Support an API for sharing images
• Support metadata on Image Registry entry in addition to metadata stored
  • within OVF container

* There is no need to fetch Images from external storage, if network distributed filesystem (GlusterFS) or over the network block device (NBD or SheepDog) is used. It would be nice to support also those solutions (at least by friendly architecture, so they can be developed by 3rd party).

Style-notes

* Staticmethods everywhere - just make sure staticmethods make sense here

• (most of the time they don't)

* Remove pyc from source tree * multiprocessing + uuid4 may cause issues (dup uuids if not careful)