MagnetoDB/specs/streamingbulkload
Contents
[Draft] MagnetoDB Streaming Bulk Load workflow and API
Workflow
This page describes process of loading large amounts of data into MagnetoDB.
Before uploading the data one should first make sure that destination table exists.
Data is uploaded in one streaming HTTP request.
URL
POST v1/{project_id}/data/tables/{table_name}/bulk_load
Headers
- User-Agent
- Content-Type: application/json
- Accept: application/json
- X-Auth-Token keystone auth token
Request Syntax
Data stream is a plain text than contains '\n' separated sequence of JSON representations of items to be inserted.
{ "attribute_name": { "attribute_type": "attribute_value"}, "attribute_name2": { "attribute_type": "attribute_value"}...}
{ "attribute_name": { "attribute_type": "attribute_value"}, "attribute_name2": { "attribute_type": "attribute_value"}...}
Response Syntax
If all incoming items were written successfully, count of all written items is returned.
{
"processed": "string"
}
In case of error, incoming data stream stops being read, and response will contain counts of received, processed and failed items, last read item and error messages for failed items.
{
"read": "string",
"processed": "string",
"failed": "string",
"last_read": { "attribute_name": { "attribute_type": "attribute_value"}, "attribute_name2": { "attribute_type": "attribute_value"}...},
"errors": [
{
"item": { "attribute_name": { "attribute_type": "attribute_value"}, "attribute_name2": { "attribute_type": "attribute_value"}...},
"error": "string"
},
{
"item": { "attribute_name": { "attribute_type": "attribute_value"}, "attribute_name2": { "attribute_type": "attribute_value"}...},
"error": "string"
},
...
]
}
Due to asynchronous processing of received items, 'PutItem' operations for several items may be enqueued when an error is found. In such case server will wait for all enqueued operations' results. Some of the results may be errors too. So response will contain more than one error.
