Jump to: navigation, search

Difference between revisions of "Glance-tasks-export"

m (Open Questions)
m
 
(10 intermediate revisions by the same user not shown)
Line 16: Line 16:
 
=== Initial Export Request ===
 
=== Initial Export Request ===
 
The initial request would be directed to the <tt>tasks</tt> path and will return the location of  task resource that can be subsequently polled to determine the status of the export.
 
The initial request would be directed to the <tt>tasks</tt> path and will return the location of  task resource that can be subsequently polled to determine the status of the export.
 +
 +
An import request will adhere to the '''task_request''' schema described in <ref>https://wiki.openstack.org/wiki/Glance-tasks-api#Task_Request_Schema</ref>
  
 
<pre>
 
<pre>
 
Request 1: POST /v2/tasks
 
Request 1: POST /v2/tasks
{ "task_type": "export",
+
{ "type": "export",
   "image_uuid": "deadbeef-dead-dead-dead-beefbeefbeef",
+
   "input": {
  "export_to": "swift://cloud.foo/myaccount/mycontainer/path",
+
      "export_uuid": "deadbeef-dead-dead-dead-beefbeefbeef",
  "export_format": "qcow2",
+
      "export_to": "swift://cloud.foo/myaccount/mycontainer/path",
  "export_to_credentials": {
+
      "export_format": "qcow2"
    /* ??? */
 
 
   }
 
   }
 
}
 
}
 
</pre>
 
</pre>
Alternative: could make a POST to <tt>/v2/images/deadbeef-dead-dead-dead-beefbeefbeef/export</tt>
 
 
<pre>
 
<pre>
 
Response 1: 201 created
 
Response 1: 201 created
Line 36: Line 36:
 
(Note: probably return the task resource in the body)
 
(Note: probably return the task resource in the body)
  
==== Export Task Request Properties ====
+
==== Export Task Request '''input''' Content ====
 
(no ordering, just using the numerals to keep count)
 
(no ordering, just using the numerals to keep count)
# task_type: export
+
# export_uuid: the UUID of the image to export
# image_uuid: the image to export
 
 
# export_to: url for glance to write to
 
# export_to: url for glance to write to
 
# export_format: image format to write
 
# export_format: image format to write
# export_to_credentials: credentials allowing glance to write to the export_to url (but see Open Question, below, about this)
+
 
 +
(Note: there are no export credentials supplied; the idea for this first implementation is that a user will use the provider's cloud storage facility as a location to export to; thus, the credentials supplied by the user initiating the request will be suitable for writing to the user's cloud storage.)
  
 
=== Polling for Export Status ===
 
=== Polling for Export Status ===
 +
An export '''task''' response will adhere to the task schema described in <ref>https://wiki.openstack.org/wiki/Glance-tasks-api#Task_Schema</ref>.
 +
 
<pre>
 
<pre>
 
Request 2 - N: GET /v2/tasks/43215678-4321-4321-4321-432156789abc
 
Request 2 - N: GET /v2/tasks/43215678-4321-4321-4321-432156789abc
Line 50: Line 52:
 
<pre>
 
<pre>
 
Response 2 - N-1: 200 OK
 
Response 2 - N-1: 200 OK
{ "task_type": "export",
+
{ "type": "export",
   "task_status" : "waiting",
+
   "status" : "pending",
   "image_uuid" : "deadbeef-dead-dead-dead-beefbeefbeef",
+
   "input": {
  "export_to": "swift://cloud.foo/myaccount/mycontainer/path",
+
      "export_uuid" : "deadbeef-dead-dead-dead-beefbeefbeef",
  "export_format": "qcow2",
+
      "export_to": "swift://cloud.foo/myaccount/mycontainer/path",
 +
      "export_format": "qcow2",
 +
  }
 
   "id": "43215678-4321-4321-4321-432156789abc",
 
   "id": "43215678-4321-4321-4321-432156789abc",
 
   ...,  
 
   ...,  
Line 60: Line 64:
 
</pre>
 
</pre>
  
==== Export Task Response Properties ====
 
(no ordering, just using the numerals to keep count)
 
# id
 
# owner
 
# task_type: export
 
# task_status
 
# image_uuid: the image to export
 
# export_to: url for glance to write to
 
# export_format: image format to write
 
# export_location: URI of the successfully exported image
 
# expires_at
 
# message
 
(Note: we'll have to maintain the credentials, even though they're not included in the response)
 
  
 
=== Final Result of Export ===
 
=== Final Result of Export ===
 
<pre>
 
<pre>
 
Response N: 200 OK
 
Response N: 200 OK
{ "task_status" : "success",
+
{ "type": "export",
   "image_uuid" : "deadbeef-dead-dead-dead-beefbeefbeef",
+
  "status" : "success",
   "export_location": "swift://cloud.foo/myaccount/mycontainer/path/12345678-abcd-abcd-abcd-987654321abc",
+
   "input": {
   "export_format": "qcow2",
+
      "export_uuid" : "deadbeef-dead-dead-dead-beefbeefbeef",
 +
      "export_to": "swift://cloud.foo/myaccount/mycontainer/path",
 +
      "export_format": "qcow2",
 +
  },
 +
   "result": {
 +
      "export_location": "swift://cloud.foo/myaccount/mycontainer/path/12345678-abcd-abcd-abcd-987654321abc"
 +
   },
 
   "expires_at" : "2013-05-21T15:19:56+0000",
 
   "expires_at" : "2013-05-21T15:19:56+0000",
 
   "id": "43215678-4321-4321-4321-432156789abc",
 
   "id": "43215678-4321-4321-4321-432156789abc",
Line 86: Line 83:
 
}
 
}
 
or
 
or
{ "task_status" : "failure",  
+
{ "type": "export",
   "image_uuid" : "deadbeef-dead-dead-dead-beefbeefbeef",
+
  "status" : "failure",  
 +
   "input": {
 +
      "export_uuid" : "deadbeef-dead-dead-dead-beefbeefbeef",
 +
      "export_to": "swift://cloud.foo/myaccount/mycontainer/path",
 +
      "export_format": "qcow2",
 +
  },
 
   "message" : "Some really informative message",
 
   "message" : "Some really informative message",
 
   "expires_at" : "2013-05-21T15:19:56+0000",
 
   "expires_at" : "2013-05-21T15:19:56+0000",
Line 94: Line 96:
 
}
 
}
 
</pre>
 
</pre>
 
+
==== Export Task Response '''result''' Content ====
In this example, the "export_to" field indicated a location where the user had write privileges (how do we handle credentials?).  Alternatively, it could be a temp url where the data could be retrieved before the "expires_at" time; the user would do a GET on the temp url and receive application/octet-stream.  The "expires_at" would also specify the expiration of the task resource containing the field.
+
# export_location: URI of the successfully exported image
 
 
== To Do ==
 
* define JSON schema for export tasks request
 
* define JSON schema for export tasks response
 
** enumerate 'task_status' values for export
 
  
 
== Open Questions ==
 
== Open Questions ==
# Having to maintain credentials for the <tt>export_to</tt> location is a bummer.  Should we instead go with one of these alternatives?
+
# Prohibited exports
#* Use a temp url in the success response; the user has to retrieve the data before the <tt>expires_at</tt> time
+
#* Do we want to provide an image metadatum, e.g., <tt>org_openstack__1__can_haz_image: false</tt> that would cause export to fail immediately, or should we just leave it up to the cloud provider to institute some metadatum of their choosing for this purpose, and handle it themselves as part of whatever export scrubbing process they implement?
#* Only allow export from glance to a user's swift-like container in the same cloud (so we could simply use the user's credentials that were originally passed to glance)
+
== Formerly Open Questions (with Answers!) ==
# When an task resource is created for <tt>{imageId}</tt>, we have to make it the case that the image cannot be deleted until the export task reaches either success or failure status.
+
# Is there any point in exporting to a tempurl and allowing the user to download from there?
#* Do we need an image task_state like instances have?
+
#* No.  (1) We don't want to worry about credentials, etc.  (2) We don't want to tie up glance with downloads to, e.g., slow unreliable connections.  It's better to let this be handled by the cloud storage facility, which is specifically designed to handle lots of downloads of lots of data to connections of varying bandwidth.
# What's the best URI to use for the POST operation?  Here are a few choices:
+
# When an task resource is created for <tt>{imageId}</tt>, should we make it the case that the image cannot be deleted until the export task reaches either success or failure statusDo we need an image task_state like instances have?
#* <tt>/v2/tasks</tt>
+
#* For the first implementation, we will not do any kind of image locking, but the task will be designed to handle failure gracefully and provide a clear, informative message to the user.  (So if you delete an image you are trying to export, the image deletion will succeed with no problemThe task will fail.)
#** Pro: symmetry with import, clone; Con: doesn't make use of the fact that we have an existing image to export
 
#* <tt>/v2/images/UUID/export</tt>
 
#** Pro: makes use of existing image resource; Con: we're not really creating an "export" resource at this URI.
 
#* <tt>/v2/images/UUID/tasks</tt>
 
#** Pro: makes use of existing image resource; Con: we are creating a "task", but not at this URIGuess you ''could'' create a task object at this URI, e.g., <tt>/v2/images/IMAGE-UUID/tasks/TASK-UUID</tt>, but then you lose the ability to do a GET <tt>/v2/tasks</tt> to list all your current tasks.
 
 
# What would be appropriate export statuses?
 
# What would be appropriate export statuses?
#* <code>queued</code> : an export task has been created, but Glance (via async worker or whatever) hasn't begun to act upon the request yet
+
#* Tasks will have very generic states.  A provider's export plugin may do various things, and rather than try to capture that specifically, we'll just cover it under 'processing'.
#* <code>processing</code> : Glance is doing some work (scrubbing or whatever)
+
# What's the best URI to use for the POST operation?
#* <code>transferring</code> : data is being moved
+
##  The consensus is: <tt>/v2/tasks</tt>
#* <code>success</code> : data has been transferred to the location specified in the response
+
##* Pro: symmetry with import, clone; Con: doesn't make use of the fact that we have an existing image to export
#* <code>failure</code> : something went wrong, an exported image will not be created from this export task request
+
## Others considered, recorded for posterity:
#* anything else?
+
### <tt>/v2/images/UUID/export</tt>
 
+
###* Pro: makes use of existing image resource; Con: we're not really creating an "export" resource at this URI.
 +
### <tt>/v2/images/UUID/tasks</tt>
 +
###* Pro: makes use of existing image resource; Con: we are creating a "task", but not at this URI.  Guess you ''could'' create a task object at this URI, e.g., <tt>/v2/images/IMAGE-UUID/tasks/TASK-UUID</tt>, but then you lose the ability to do a GET <tt>/v2/tasks</tt> to list all your current tasks.
 
== Revison History ==
 
== Revison History ==
 
This is a revised version of <ref>https://wiki.openstack.org/wiki/Glance-new-download-workflow</ref>.  The revision is mostly to make this part of the tasks API <ref>https://wiki.openstack.org/wiki/Glance-tasks-api</ref>.
 
This is a revised version of <ref>https://wiki.openstack.org/wiki/Glance-new-download-workflow</ref>.  The revision is mostly to make this part of the tasks API <ref>https://wiki.openstack.org/wiki/Glance-tasks-api</ref>.
 +
 +
Revised again 23 Aug 2013 in light of discussion captured in <ref>https://etherpad.openstack.org/LG39UnQA7z</ref>.
  
 
== References ==
 
== References ==
 
<references />
 
<references />

Latest revision as of 15:31, 23 August 2013

This is the full specification for blueprint:https://blueprints.launchpad.net/glance/+spec/new-download-workflow

Motivation

Here's why we want an alternative to the "normal" Glance download workflow:

  • want to allow a cloud provider the opportunity to "scrub" or otherwise process an image that's going to be pulled out of that provider's cloud
    • they don't have to, but want to make the option available
  • want to push all efforts associated with getting images out of the cloud to the cloud periphery so the bandwidth/compute required doesn't affect normal cloud functionality

"Export"

We propose to introduce as a Glance task an end-user download-type operation called export.

The arguments in favor of an "import" operation also apply to "export", so they're not repeated here. See the Import specification [1] for more information.

Example Workflow (API View)

Initial Export Request

The initial request would be directed to the tasks path and will return the location of task resource that can be subsequently polled to determine the status of the export.

An import request will adhere to the task_request schema described in [2]

Request 1: POST /v2/tasks
{ "type": "export",
  "input": {
      "export_uuid": "deadbeef-dead-dead-dead-beefbeefbeef",
      "export_to": "swift://cloud.foo/myaccount/mycontainer/path",
      "export_format": "qcow2"
  }
}
Response 1: 201 created
Location: "http://glance-server/v2/tasks/43215678-4321-4321-4321-432156789abc"

(Note: probably return the task resource in the body)

Export Task Request input Content

(no ordering, just using the numerals to keep count)

  1. export_uuid: the UUID of the image to export
  2. export_to: url for glance to write to
  3. export_format: image format to write

(Note: there are no export credentials supplied; the idea for this first implementation is that a user will use the provider's cloud storage facility as a location to export to; thus, the credentials supplied by the user initiating the request will be suitable for writing to the user's cloud storage.)

Polling for Export Status

An export task response will adhere to the task schema described in [3].

Request 2 - N: GET /v2/tasks/43215678-4321-4321-4321-432156789abc
Response 2 - N-1: 200 OK
{ "type": "export",
  "status" : "pending",
  "input": {
      "export_uuid" : "deadbeef-dead-dead-dead-beefbeefbeef",
      "export_to": "swift://cloud.foo/myaccount/mycontainer/path",
      "export_format": "qcow2",
  }
  "id": "43215678-4321-4321-4321-432156789abc",
   ..., 
}


Final Result of Export

Response N: 200 OK
{ "type": "export",
  "status" : "success",
  "input": {
      "export_uuid" : "deadbeef-dead-dead-dead-beefbeefbeef",
      "export_to": "swift://cloud.foo/myaccount/mycontainer/path",
      "export_format": "qcow2",
  },
  "result": {
      "export_location": "swift://cloud.foo/myaccount/mycontainer/path/12345678-abcd-abcd-abcd-987654321abc"
  },
  "expires_at" : "2013-05-21T15:19:56+0000",
  "id": "43215678-4321-4321-4321-432156789abc",
   ...,
}
or
{ "type": "export",
  "status" : "failure", 
  "input": {
      "export_uuid" : "deadbeef-dead-dead-dead-beefbeefbeef",
      "export_to": "swift://cloud.foo/myaccount/mycontainer/path",
      "export_format": "qcow2",
  },
  "message" : "Some really informative message",
  "expires_at" : "2013-05-21T15:19:56+0000",
  "id": "43215678-4321-4321-4321-432156789abc",
   ..., 
}

Export Task Response result Content

  1. export_location: URI of the successfully exported image

Open Questions

  1. Prohibited exports
    • Do we want to provide an image metadatum, e.g., org_openstack__1__can_haz_image: false that would cause export to fail immediately, or should we just leave it up to the cloud provider to institute some metadatum of their choosing for this purpose, and handle it themselves as part of whatever export scrubbing process they implement?

Formerly Open Questions (with Answers!)

  1. Is there any point in exporting to a tempurl and allowing the user to download from there?
    • No. (1) We don't want to worry about credentials, etc. (2) We don't want to tie up glance with downloads to, e.g., slow unreliable connections. It's better to let this be handled by the cloud storage facility, which is specifically designed to handle lots of downloads of lots of data to connections of varying bandwidth.
  2. When an task resource is created for {imageId}, should we make it the case that the image cannot be deleted until the export task reaches either success or failure status? Do we need an image task_state like instances have?
    • For the first implementation, we will not do any kind of image locking, but the task will be designed to handle failure gracefully and provide a clear, informative message to the user. (So if you delete an image you are trying to export, the image deletion will succeed with no problem. The task will fail.)
  3. What would be appropriate export statuses?
    • Tasks will have very generic states. A provider's export plugin may do various things, and rather than try to capture that specifically, we'll just cover it under 'processing'.
  4. What's the best URI to use for the POST operation?
    1. The consensus is: /v2/tasks
      • Pro: symmetry with import, clone; Con: doesn't make use of the fact that we have an existing image to export
    2. Others considered, recorded for posterity:
      1. /v2/images/UUID/export
        • Pro: makes use of existing image resource; Con: we're not really creating an "export" resource at this URI.
      2. /v2/images/UUID/tasks
        • Pro: makes use of existing image resource; Con: we are creating a "task", but not at this URI. Guess you could create a task object at this URI, e.g., /v2/images/IMAGE-UUID/tasks/TASK-UUID, but then you lose the ability to do a GET /v2/tasks to list all your current tasks.

Revison History

This is a revised version of [4]. The revision is mostly to make this part of the tasks API [5].

Revised again 23 Aug 2013 in light of discussion captured in [6].

References

  1. https://wiki.openstack.org/wiki/Glance-tasks-import
  2. https://wiki.openstack.org/wiki/Glance-tasks-api#Task_Request_Schema
  3. https://wiki.openstack.org/wiki/Glance-tasks-api#Task_Schema
  4. https://wiki.openstack.org/wiki/Glance-new-download-workflow
  5. https://wiki.openstack.org/wiki/Glance-tasks-api
  6. https://etherpad.openstack.org/LG39UnQA7z