Latest revision as of 23:29, 17 February 2013

Nova Image Notes

Image Implementation In a Nutshell

The system currently provides two backends for retrieving images:

• Node-local Store (my term) - A directory on the node where images

     reside. Images in node-local storage should already in raw format (not
     AMI)

• S3 Object Store - Currently this is an fake which provides an S3 ReST

     interface to the filesystem

The Node-local Store appears to be a convenience for testing and perhaps for use by an Ops team when it goes live, whereas, S3 provides a registration mechanism (with hints at Image sharing) that would be exposed to customers.

An Unregistered Image is mapped to a bucket in the S3 object store. Within the bucket is:

• The encrypted AMI
• A JSON manifest (containing the decryption keys)

The Image is Registered by calling endpoints/node.py/register() which in turn issues a PUT request to the object store. Contained within the PUT request is:

• Image Location: account and bucket for the image
• Image ID: unique identifier

The S3-ObjectStore handler (objectstore/handler.py) receives the PUT request and forks of a process to continue the registration process (Image.create found in objectstore/image.py) which includes:

• Fetching the image from Image Location
• Decrypting the Image using the keys provided by the manifest
• Untarring the AMI contents
• Moving the Image to the destination directory for serving using the ReST

     API

The state of the Image is contained with the JSON manifest which is updated as the image progresses through the unpacking process. Eventually the image reaches the 'available' status and is said to be 'registered'.

Now that the image is registered, the spawn method of compute/node.py can use curl to fetch the image from the objectstore (util.fetchfile) to instantiate a new VM.

It should be noted both the consumer (node.py) and the provider (Image API/objectstore) think of images as having a type:

• Kernel
• Ramdisk
• Image

So, in order spawn a VM Nova you really need to specify *three* images, one of each type.

What the Implementation Gets Right

• The Node-local Store - useful for testing and debugging, plus, could be

     useful for OPs teams when the system goes live

• Separates the Image API (register/deregister) from Instance API

     (create/detroy VMS)

• Using fakes for the backends of the object stores.

Where the Implemenation Needs Work

• Registering AMI hits disk at least 3 times (fetch, decrypt, untar). For

     large images this could be painful. 

• The Image registraion API is missing key features (image sharing,

     metadata, etc)

What the Implmentation Gets Wrong

• Amazon-stack specific: The Image API is coupled to S3 for the object

     store and AMI's as the image format

• No Caching - each Spawn would require a fetch from Image API
• Does not use OVF container - some of the JSON manifest data would make

     sense in the manifest.ovf file (others parts would not)

• Conflates two different types of metadata: Image metadata which

     describes the VM (architecture, etc) versus Image metadata which
     describes the Image itself (isPublic, isShared, image location). I would
     argue that these are fundamentally different and make sense stored in
     two different places. The important question being: does this metadata
     make sense *outside* of our system.

Ideas for Improvement

• Use Pyfilesystem to store Images. This would provide support for:
  • S3
  • CloudFiles (once the plugin is written, I'm working on that now)
  • Local disk storage
  • Memory storage (this could be great for unit-testing)
• Add a caching layer. This could be:
  • A separate caching utility
  • Part of Pyfilesystems caching layer
  • Some combination of both
• Create robust Image registry. This would:
  • Backed by Cassandra or Redis
  • Support an API for sharing images
  • Support metadata on Image Registry entry in addition to metadata stored

     within OVF container

• There is no need to fetch Images from external storage, if network distributed filesystem (GlusterFS) or over the network block device (NBD or SheepDog) is used. It would be nice to support also those solutions (at least by friendly architecture, so they can be developed by 3rd party).

Style-notes

• Staticmethods everywhere - just make sure staticmethods make sense here

 (most of the time they don't)

• Remove pyc from source tree
• multiprocessing + uuid4 may cause issues (dup uuids if not careful)