Difference between revisions of "Redis Storage Driver Design (Zaqar)"
Prashanthr (talk | contribs) (→Components of the basic driver) |
Prashanthr (talk | contribs) |
||
| (3 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
=== Objective === | === Objective === | ||
| − | Redis is an in memory key-value datastore which can organize data as strings, hashes, lists, sets, sorted sets, bitmaps and hyperloglogs. Being In-memory makes it in ideal candidate for high load, low latency messages. | + | Redis is an in memory key-value datastore which can organize data as strings, hashes, lists, sets, sorted sets, bitmaps and hyperloglogs. Being In-memory makes it in ideal candidate for high load, low latency messages. Redis is distributed under the BSD license and is henceforth unbounded by any licensing constraints as in the case of mongodb. |
| − | + | ==== Use Cases ==== | |
1. High-throughput (2K+ messages/sec per queue)<br /> | 1. High-throughput (2K+ messages/sec per queue)<br /> | ||
| Line 27: | Line 27: | ||
<br /> | <br /> | ||
===== Organization of Data ===== | ===== Organization of Data ===== | ||
| − | Message id's list ( Redis Sorted Set ) | + | ====== Message id's list ( Redis Sorted Set ) ====== |
<br /> | <br /> | ||
Each queue in the system has a set of message ids currently in the queue. The list is sorted based on a ranking which is incremented atomically using the counter(MESSAGE_RANK_COUNTER_SUFFIX) also stored in the database for every queue. | Each queue in the system has a set of message ids currently in the queue. The list is sorted based on a ranking which is incremented atomically using the counter(MESSAGE_RANK_COUNTER_SUFFIX) also stored in the database for every queue. | ||
<br /> | <br /> | ||
| − | Messages(Redis Hash) | + | ====== Messages(Redis Hash) ====== |
<br /> | <br /> | ||
| Line 69: | Line 69: | ||
Queues are scoped by project, which is prefixed to the queue name. | Queues are scoped by project, which is prefixed to the queue name. | ||
| − | Queues ( Redis Sorted Set ) | + | ===== Data Organization ===== |
| + | |||
| + | ====== Queues ( Redis Sorted Set ) ====== | ||
<br /> | <br /> | ||
Id: queues_set | Id: queues_set | ||
<br /> | <br /> | ||
| − | |||
{| class="wikitable" | {| class="wikitable" | ||
|- | |- | ||
| Line 84: | Line 85: | ||
<br /> | <br /> | ||
| − | Queue Information (Redis Hash) | + | ====== Queue Information (Redis Hash) ====== |
<br /> | <br /> | ||
| Line 109: | Line 110: | ||
<br /> | <br /> | ||
| − | Claims list ( Redis Set ) contains claim ids active for a queue. | + | ===== Data Organization ===== |
| + | |||
| + | ======Claims list ( Redis Set ) ====== | ||
| + | |||
| + | contains claim ids active for a queue. | ||
<br /> | <br /> | ||
| Line 115: | Line 120: | ||
<br /> | <br /> | ||
| − | Claim info(Redis Hash) | + | ====== Claim info(Redis Hash) ====== |
<br /> | <br /> | ||
| Line 147: | Line 152: | ||
=== Current Status === | === Current Status === | ||
| − | The basic driver is a "Work in Progress" and is expected to be a part of the code base by Juno-3 deadline (Sept 4th 2014). The clustering and persistence extension is expected to be developed before the end of Kilo. | + | The basic driver is a "Work in Progress" and is expected to be a part of the code base by Juno-3 deadline (Sept 4th 2014). The clustering and persistence extension is expected to be developed before the end of Kilo. It is also one of the requirements for the graduation of Marconi. |
=== Blueprints === | === Blueprints === | ||
| Line 170: | Line 175: | ||
| [https://review.openstack.org/#/c/104058/] || Pools and Catalog Controllers | | [https://review.openstack.org/#/c/104058/] || Pools and Catalog Controllers | ||
|} | |} | ||
| + | |||
| + | === Future Work === | ||
| + | 1. Support Async connections to the redis database. | ||
Latest revision as of 13:52, 11 August 2014
Contents
Objective
Redis is an in memory key-value datastore which can organize data as strings, hashes, lists, sets, sorted sets, bitmaps and hyperloglogs. Being In-memory makes it in ideal candidate for high load, low latency messages. Redis is distributed under the BSD license and is henceforth unbounded by any licensing constraints as in the case of mongodb.
Use Cases
1. High-throughput (2K+ messages/sec per queue)
2. Lots of small messages (<= 1K)
3. Messages are short-lived (minutes, not hours)
4. High durability is optional (2 copies, RAM only)
5. HA is still very important
Components of the basic driver
The basic driver includes support for a single instance redis store as a data backend.
The driver supports 6 main required functions of a marconi storage driver which are:
Message Controller
Helps organize messages scoped by every queue.
Organization of Data
Message id's list ( Redis Sorted Set )
Each queue in the system has a set of message ids currently in the queue. The list is sorted based on a ranking which is incremented atomically using the counter(MESSAGE_RANK_COUNTER_SUFFIX) also stored in the database for every queue.
Messages(Redis Hash)
Each message is stored using the UUID of the message as the redis key. The datastructure has the following information.
NOTE: This datastructure is serialized using msgpack.
| Name | Field |
|---|---|
| id | id |
| created iso | cr_iso |
| ttl | t |
| expires | e |
| body | b |
| claim | c |
| claim expiry time | c.e |
| client uuid | u |
| created time | cr |
Queue Controller
Queues are scoped by project, which is prefixed to the queue name.
Data Organization
Queues ( Redis Sorted Set )
Id: queues_set
| Name | Field |
|---|---|
| name | <project-id.queue-name> |
The set helps faster existence checks, while the list helps paginated retrieval of queues.
Queue Information (Redis Hash)
This Data structure holds all meta information about a queue.
Scope: <project-id_q-name>
| Name | Field |
|---|---|
| count(size) | c |
| claimed | cl |
| metadata | m |
| creation time | t |
Claims Controller
Data Organization
Claims list ( Redis Set )
contains claim ids active for a queue.
Redis ID: <project-id_q-name>
Claim info(Redis Hash)
Contains the metadata information for a particular claim.
| Name | Field |
|---|---|
| ttl | t |
| grace | g |
| id | if |
| claim | c |
| client UUID | u |
Catalogue Controller
TBD
Pools Controller
TBD
Current Status
The basic driver is a "Work in Progress" and is expected to be a part of the code base by Juno-3 deadline (Sept 4th 2014). The clustering and persistence extension is expected to be developed before the end of Kilo. It is also one of the requirements for the graduation of Marconi.
Blueprints
| Blueprint URL | Functionality |
|---|---|
| [1] | Redis as an alternative driver for MongoDB |
| [2] | Redis Storage Driver for Marconi |
Submitted Patches
| Patch URL | Implemented Feature |
|---|---|
| [3] | Messages and Queue Controllers |
| [4] | Claims Controller |
| [5] | Pools and Catalog Controllers |
Future Work
1. Support Async connections to the redis database.
