Difference between revisions of "Redis Storage Driver Design (Zaqar)"
Prashanthr (talk | contribs) (→Organization of Data) |
Prashanthr (talk | contribs) (→Components of the basic driver) |
||
Line 23: | Line 23: | ||
The driver supports 6 main required functions of a marconi storage driver which are: | The driver supports 6 main required functions of a marconi storage driver which are: | ||
− | ==== Message | + | ==== Message Controller ==== |
Helps organize messages scoped by every queue. | Helps organize messages scoped by every queue. | ||
<br /> | <br /> | ||
Line 35: | Line 35: | ||
<br /> | <br /> | ||
− | Each message is stored using the UUID of the message as the redis key. The datastructure has the following information. | + | Each message is stored using the UUID of the message as the redis key. The datastructure has the following information. |
+ | <br /> | ||
+ | NOTE: This datastructure is serialized using msgpack. | ||
<br /> | <br /> | ||
Line 62: | Line 64: | ||
<br /> | <br /> | ||
− | + | ==== Queue Controller ==== | |
+ | <br /> | ||
+ | |||
+ | Queues are scoped by project, which is prefixed to the queue name. | ||
+ | |||
+ | Queues ( Redis Sorted Set ): | ||
+ | <br /> | ||
+ | Id: queues_set | ||
+ | <br /> | ||
+ | |||
+ | Id | ||
+ | {| class="wikitable" | ||
+ | |- | ||
+ | ! Name !! Field | ||
+ | |- | ||
+ | | name || <project-id.queue-name> | ||
+ | |} | ||
+ | The set helps faster existence checks, while the list helps paginated retrieval of queues. | ||
+ | <br /> | ||
+ | |||
+ | Queue Information (Redis Hash): | ||
+ | <br /> | ||
+ | |||
+ | This Data structure holds all meta information about a queue. | ||
+ | <br /> | ||
+ | |||
+ | Scope: <project-id_q-name> | ||
+ | |||
+ | <br /> | ||
+ | {| class="wikitable" | ||
+ | |- | ||
+ | ! Name !! Field | ||
+ | |- | ||
+ | | count(size) || c | ||
+ | |- | ||
+ | | claimed || cl | ||
+ | |- | ||
+ | | metadata || m | ||
+ | |- | ||
+ | | creation time || t | ||
+ | |} | ||
+ | |||
+ | ==== Claims Controller ==== | ||
+ | <br /> | ||
+ | |||
+ | Claims list ( Redis Set ) contains claim ids active for a queue. | ||
+ | <br /> | ||
− | + | Redis ID: <project-id_q-name> | |
+ | <br /> | ||
− | + | Claim info(Redis Hash): | |
+ | <br /> | ||
− | + | Contains the metadata information for a particular claim. | |
+ | <br /> | ||
+ | {| class="wikitable" | ||
+ | |- | ||
+ | ! Name !! Field | ||
+ | |- | ||
+ | | ttl || t | ||
+ | |- | ||
+ | | grace || g | ||
+ | |- | ||
+ | | id || if | ||
+ | |- | ||
+ | | claim || c | ||
+ | |- | ||
+ | | client UUID || u | ||
+ | |} | ||
+ | <br /> | ||
+ | |||
+ | ==== Catalogue Controller ==== | ||
+ | TBD | ||
+ | <br /> | ||
− | + | ||
− | + | ==== Pools Controller ==== | |
− | + | TBD | |
− | + | <br /> | |
=== Current Status === | === Current Status === |
Revision as of 05:03, 9 August 2014
Contents
Objective
Redis is an in memory key-value datastore which can organize data as strings, hashes, lists, sets, sorted sets, bitmaps and hyperloglogs. Being In-memory makes it in ideal candidate for high load, low latency messages.
The main use cases of the driver are listed below:
1. High-throughput (2K+ messages/sec per queue)
2. Lots of small messages (<= 1K)
3. Messages are short-lived (minutes, not hours)
4. High durability is optional (2 copies, RAM only)
5. HA is still very important
Components of the basic driver
The basic driver includes support for a single instance redis store as a data backend.
The driver supports 6 main required functions of a marconi storage driver which are:
Message Controller
Helps organize messages scoped by every queue.
Organization of Data
Message id's list ( Redis Sorted Set )
Each queue in the system has a set of message ids currently in the queue. The list is sorted based on a ranking which is incremented atomically using the counter(MESSAGE_RANK_COUNTER_SUFFIX) also stored in the database for every queue.
Messages(Redis Hash):
Each message is stored using the UUID of the message as the redis key. The datastructure has the following information.
NOTE: This datastructure is serialized using msgpack.
Name | Field |
---|---|
id | id |
created iso | cr_iso |
ttl | t |
expires | e |
body | b |
claim | c |
claim expiry time | c.e |
client uuid | u |
created time | cr |
Queue Controller
Queues are scoped by project, which is prefixed to the queue name.
Queues ( Redis Sorted Set ):
Id: queues_set
Id
Name | Field |
---|---|
name | <project-id.queue-name> |
The set helps faster existence checks, while the list helps paginated retrieval of queues.
Queue Information (Redis Hash):
This Data structure holds all meta information about a queue.
Scope: <project-id_q-name>
Name | Field |
---|---|
count(size) | c |
claimed | cl |
metadata | m |
creation time | t |
Claims Controller
Claims list ( Redis Set ) contains claim ids active for a queue.
Redis ID: <project-id_q-name>
Claim info(Redis Hash):
Contains the metadata information for a particular claim.
Name | Field |
---|---|
ttl | t |
grace | g |
id | if |
claim | c |
client UUID | u |
Catalogue Controller
TBD
Pools Controller
TBD
Current Status
The basic driver is a "Work in Progress" and is expected to be a part of the code base by Juno-3 deadline (Sept 4th 2014). The clustering and persistence extension is expected to be developed before the end of Kilo.
Blueprints
Blueprint URL | Functionality |
---|---|
[1] | Redis as an alternative driver for MongoDB |
[2] | Redis Storage Driver for Marconi |
Submitted Patches
Patch URL | Implemented Feature |
---|---|
[3] | Messages and Queue Controllers |
[4] | Claims Controller |
[5] | Pools and Catalog Controllers |