Glance-tasks-import

This is the full specification for blueprint:https://blueprints.launchpad.net/glance/+spec/new-upload-workflow

Context

We want to make a Glance API endpoint public to enable end-user (not admin) upload of images.

Problems Facing a Public Glance Endpoint

Making a Glance API endpoint public causes these problems:

• need some kind of quality control on what's being stuffed into glance as "images"
  • this enhances end-user experience (so you don't keep trying to boot an uploaded movie or something -- want to reject bad images before this point if possible)
• don't want the extra bandwidth, processing requirements of these uploads to interfere with normal glance functionality as an image source for nova
  • want to be able to push the upload, validation process out to the cloud periphery

"Import"

We propose to introduce as a Glance task an end-user upload-type operation called import.

An objection to this proposal is that Glance v1 has "copy-from" and "location" ... why aren't these sufficient?

Difference between "import" and "copy-from":

• copy-from: you've got an Image (capital "I") to upload
• import: you've got some bits that you want turned into an Image

(In other words, we can trust nova but don't know about ordinary glance "users")

Difference between "import" and "location":

• location: you've got an Image stored elsewhere; glance grabs the bits from there when the Image is requested
• import: you've got some bits stored elsewhere, but there's no way to know whether they are an "Image"

So, for these reasons, we will introduce an import operation.

Example Workflow (API View)

Initial Import Request

The initial request would be similar to an image-create, but would be directed to the tasks path and would return the location of a task resource.

There are several advantages to using a task resource instead of simply creating an image that could be polled until it turned active:

• if the import is a failure, we don't have to worry about what to do with the image record
• the user's image-list won't be clogged with images that are being imported and can't be used to boot yet, anyway
• the task resource can be structured to allow for helpful error messages; otherwise, we'd have to try to find a way to shoehorn them into the current image resource
• we can create useful status values for this import task resource without having to modify the current image statuses
• decoupling the task resource from the image resource allows for independent development of each

Import Task Request Properties

(no ordering, just using the numerals to keep count)

1. task_type: import
2. import_from (a URL where glance can get the image data)
3. import_from_format (e.g., 'qcow2')
4. image_properties (basically, this would be the dict of properties you can specify on the current image create call)

Request 1: POST /v2/tasks
{ "task_type": "import",
  "import_from": "swift://cloud.foo/myaccount/mycontainer/path",
  "import_from_format": "qcow2",
  "image_properties" : {
    "name": "GreatStack 1.22",
    "tags": ["lamp", "custom"]
  }
}

The content of the image_properties field is the same content defined for the image create request in the Images v2 "Metadata" API .

Response 1: 201 Created
Location: "http://glance-server/v2/tasks/12345678-1234-1234-1234-123456789abc"

(Note: probably return the task resource in the body) The image_properties passed in the request are included in the task body so that (a) the user can tell which task this is a record for (in case the user has multiple tasks underway), and (b) we'll have them available when the image record finally gets created after a successful import.

Polling for Import Status

The newly-created task resource can now be polled for import status information. Ultimately, it would contain either the location of the successfully created image or an informative message indicating why the import failed. The ultimate response (either success or failure) would contain an expires_at field indicating when this particular task resource may be deleted so we wouldn't have these things accumulating indefinitely.

Request 2 - N: GET /v2/tasks/12345678-1234-1234-1234-123456789abc

Response 2 - N-1: 200 OK
{ "task_status": "waiting", "image_properties" : {... }, "id": "12345678-1234-1234-1234-123456789abc", ... }
 or
{ "task_status": "uploading", "image_properties" : {... }, "id": "12345678-1234-1234-1234-123456789abc", ...  }
 or
{ "task_status": "verifying", "image_properties" : {... }, "id": "12345678-1234-1234-1234-123456789abc", ...  }
 or
{ "task_status": "what-have-you", "image_properties" : {... }, "id": "12345678-1234-1234-1234-123456789abc", ...  }

Response N: 200 OK
{ "task_status": "success",
  "image_ref": "/v2/images/deadbeef-dead-dead-dead-beefbeefbeef",
  "expires_at": "2013-05-21T15:19:56+0000",
  "id": "12345678-1234-1234-1234-123456789abc",
   /* etc */
}
or
{ "task_status": "failure",
  "message": "Could not retrieve image.  Received 404 response for url 'swift://cloud.foo/myaccount/mycontainer/path'.",
  "expires_at": "2013-05-21T15:19:56+0000",
  "image_properties" : {... },
  "id": "12345678-1234-1234-1234-123456789abc",
   /* etc */
}

In the 'success' response, the 'image_properties' element is replaced with an 'image_ref' element. In the 'failure' response, a 'message' element is added to the response.

Import Task Response Properties

(no ordering, just using the numerals to keep count)

1. id
2. owner
3. task_type
4. task_status
5. import_from (so you can keep track of what exactly this task is importing)
6. image_properties
7. expires_at
8. message

Image Retrieval

If the import is successful, the imported image is available through Glance in the normal way:

Request N+1: GET /v2/images/deadbeef-dead-dead-dead-beefbeefbeef

Response N+1: 200 OK
{ "status": "active",
   "name": "GreatStack 1.22",
   "tags": ["lamp", "custom"],
    ... }

Example Workflow (Internal View)

1. create task import object (status: waiting)
2. get the data from the location specified
3. verify the data (status: verifying)
   • not sure what exactly this "validation" will consist of ... probably want to make it pluggable for the cloud provider ?
   • this is also the place where image conversion could be performed if, for example, the user is uploading an image in qcow2 (or whatever is decided to be the openstack standard interchange format) and the cloud being loaded to uses images in a different format
4. if success, create the image in Glance
5. record result in task import object, set appropriate status, and set expiration date

Summary

• introduce new task import resource
  • the resource will need to have an "owner" field (whether or not it's exposed in the response) so that we can make sure that users can't see each other's import requests
  • define JSON schema for task import request
  • define JSON schema for task import response
    • enumerate 'task_status' values for import tasks

Open Questions

1. What would be appropriate import statuses?
   • queued : an import task has been created, but Glance (via async worker or whatever) hasn't begun the actual data transfer yet
   • transferring : data transfer from the specified location is underway
   • processing : data has been uploaded and is undergoing verification (or conversion or whatever)
   • success : data has been verified and a Glance image has been created
   • failure : something went wrong, an image will not be created from this import task
   • anything else?
2. Do we want to allow cancellation of an import task by a DELETE on /v2/tasks/UUID ?
   • only allow 'queued' tasks to be deleted?

Formerly Open Questions (with Answers!)

1. The current v2 upload workflow allows 3 options: copy_from, location, and direct data upload. The proposed import workflow offers only a "copy_from" functionality. We definitely don't want to allow the current "location" functionality, but what about direct upload of data?
   • No direct upload of data for this first implementation

Revison History

This is a revised version of . The revision is mostly to make this part of the tasks API .