Glance-new-upload-workflow

Superceded
This document has been superceded by: https://wiki.openstack.org/wiki/Glance-tasks-import

The related blueprint is:https://blueprints.launchpad.net/glance/+spec/new-upload-workflow

Context

We want to make a Glance API endpoint public to enable end-user (not admin) upload of images.

Problems Facing a Public Glance Endpoint

Making a Glance API endpoint public causes these problems:

• need some kind of quality control on what's being stuffed into glance as "images"
  • this enhances end-user experience (so you don't keep trying to boot an uploaded movie or something -- want to reject bad images before this point if possible)
• don't want the extra bandwidth, processing requirements of these uploads to interfere with normal glance functionality as an image source for nova
  • want to be able to push the upload, validation process out to the cloud periphery

"Import"

We propose to introduce an end-user upload-type operation called "import".

An objection to this proposal is that Glance v1 has "copy-from" and "location" ... why aren't these sufficient?

Difference between "import" and "copy-from":

• copy-from: you've got an Image (capital "I") to upload
• import: you've got some bits that you want turned into an Image

(In other words, we can trust nova but don't know about ordinary glance "users")

Difference between "import" and "location":

• location: you've got an Image stored elsewhere; glance grabs the bits from there when the Image is requested
• import: you've got some bits stored elsewhere, but there's no way to know whether they are an "Image"

So, for these reasons, we will introduce an import operation.

Example Workflow (API View)

Initial Import Request

The initial request would be similar to an image-create, but would be directed to a different path and would return the location of an "image-import" resource.

There are several advantages to using a new resource instead of simply creating an image that could be polled until it turned active:

• if the import is a failure, we don't have to worry about what to do with the image record
• the user's image-list won't be clogged with images that are being imported and can't be used to boot yet, anyway
• the new resource can be structured to allow for helpful error messages; otherwise, we'd have to try to find a way to shoehorn them into the current image resource
• we can create useful status values for these image-import resources without having to modify the current image statuses

Request 1: POST /v2/image-imports
{ "location": "swift://cloud.foo/myaccount/mycontainer/path",
  "image_properties" : {
    "name": "GreatStack 1.22",
    "tags": ["lamp", "custom"]
  } }

Response 1: 201 Created
Location: "http://glance-server/v2/image-imports/12345678-1234-1234-1234-123456789abc"

Polling for Import Status

The image-import resource could be polled for import status information. Ultimately, it would contain either the location of the successfully created image or an informative message indicating why the import failed. The ultimate response (either success or failure) would contain an 'Expires-by' field indicating when the image-import resource would be deleted so we wouldn't have these things accumulating indefinitely.

The image_properties passed in the request are included here so that (a) the user can tell which import this is a record for (in case the user has multiple imports underway), and (b) we'll have them available when the image gets created after a successful import.

Request 2 - N: GET /v2/image-imports/12345678-1234-1234-1234-123456789abc

Response 2 - N-1: 200 OK
{ "status": "waiting", "image_properties" : {... } }
 or
{ "status": "uploading", "image_properties" : {... } }
 or
{ "status": "verifying", "image_properties" : {... } }
 or
{ "status": "what-have-you", "image_properties" : {... } }

Response N: 200 OK
{ "status": "success",
  "image_ref": "/v2/images/deadbeef-dead-dead-dead-beefbeefbeef",
  "expires-by": "2013-05-21T15:19:56+0000" }
or
{ "status": "failure",
  "message": "Could not retrieve image.  Received 404 response for url 'swift://cloud.foo/myaccount/mycontainer/path'.",
  "expires-by": "2013-05-21T15:19:56+0000",
  "image_properties" : {... } }

In the 'success' response, the 'image_properties' element is replaced with an 'image_ref' element. In the 'failure' response, a 'message' element is added to the response.

Image Retrieval

If the import is successful, the imported image is available through Glance in the normal way:

Request N+1: GET /v2/images/deadbeef-dead-dead-dead-beefbeefbeef

Response N+1: 200 OK
{ "status": "active",
   "name": "GreatStack 1.22",
   "tags": ["lamp", "custom"],
    ... }

Example Workflow (Internal View)

1. create image-import object (status: waiting)
2. get the data from the location specified
3. verify the data (status: verifying)
   • not sure what exactly this "validation" will consist of ... probably want to make it pluggable for the cloud provider ?
   • this is also the place where image conversion could be performed if, for example, the user is uploading an image in qcow2 (or whatever is decided to be the openstack standard interchange format) and the cloud being loaded to uses images in a different format
4. if success, create the image in Glance
5. record result in image-import object, set appropriate status, and set expiration date

Summary

• introduce new resource, 'image-import'
  • the resource will need to have an "owner" field (whether or not it's exposed in the response) so that we can make sure that users can't see each other's import requests
  • should implement a GET /v2/image-imports that would respond with a list of all image-imports that exist for this user
  • define JSON schema for 'image-import' request
  • define JSON schema for 'image-import' response
    • enumerate 'status' values
  • define POST for 'image-import'
  • define GET for 'image-import'

Open Questions

1. The current v2 upload workflow allows 3 options: copy_from, location, and direct data upload. The proposed import workflow offers only a "copy_from" functionality. We definitely don't want to allow the current "location" functionality, but what about direct upload of data?
2. What would be appropriate import statuses?
   • waiting : image-import request has been received, but Glance (via async worker or whatever) hasn't begun the actual data transfer yet
   • uploading : data transfer from the specified location is underway
   • verifying : data has been uploaded and is undergoing verification
   • success : data has been verified and Glance image has been created
   • failure : something went wrong, an image will not be created from this import request
   • anything else?
3. So when you put this BP together with the new export BP, we have GET /v2/image-imports and GET /v2/image-exports ... should we instead have something like /v2/image-actions that would list all the non-expired actions you had going? We could then have a single resource type, 'image-action', which would include an "action_type" (either "import" or "export"). This image-actions-list call could allow for filtering by action_type.
   • I'm starting to think image cloning to other regions should create a new resource (image-clone) like image-export or image-import, so I like this idea!