Revision as of 12:27, 27 May 2014

[Draft] MagnetoDB Streaming Bulk Load workflow and API

Workflow

This page describes process of loading large amounts of data into MagnetoDB.

Before uploading the data one should first make sure that destination table exists.

Data is uploaded in one streaming HTTP request.

URL

POST v1/{project_id}/data/tables/{table_name}/bulk_load

Headers

• User-Agent
• Content-Type: application/json
• Accept: application/json
• X-Auth-Token keystone auth token

Request Syntax

Data stream is a plain text than contains '\n' separated sequence of JSON representations of items to be inserted.

{ "attribute_name": { "attribute_type": "attribute_value"}, "attribute_name2": { "attribute_type": "attribute_value"}...}
{ "attribute_name": { "attribute_type": "attribute_value"}, "attribute_name2": { "attribute_type": "attribute_value"}...}

Response Syntax

If all incoming items were written successfully, count of all written items is returned.

{
    "processed": "string"
}

In case of error, incoming data stream stops being read, and response will contain counts of received, processed and failed items, last read item and error messages for failed items.

{
    "read": "string",
    "processed": "string",
    "failed": "string",
    "last_read": { "attribute_name": { "attribute_type": "attribute_value"}, "attribute_name2": { "attribute_type": "attribute_value"}...},
    "errors": [
        {
            "item": { "attribute_name": { "attribute_type": "attribute_value"}, "attribute_name2": { "attribute_type": "attribute_value"}...},
            "error": "string"
        },
        {
            "item": { "attribute_name": { "attribute_type": "attribute_value"}, "attribute_name2": { "attribute_type": "attribute_value"}...},
            "error": "string"
        },
        ...
    ]
}

Due to asynchronous processing of received items, 'PutItem' operations for several items may be enqueued when an error is found. In such case server will wait for all enqueued operations' results. Some of the results may be errors too. So response will contain more than one error.