Difference between revisions of "Redis Storage Driver Design (Zaqar)"
Prashanthr (talk | contribs) (→Organization of Data) |
Prashanthr (talk | contribs) (→Organization of Data) |
||
| Line 34: | Line 34: | ||
Messages(Redis Hash): | Messages(Redis Hash): | ||
<br /> | <br /> | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
Each message is stored using the UUID of the message as the redis key. The datastructure has the following information. | Each message is stored using the UUID of the message as the redis key. The datastructure has the following information. | ||
| + | <br /> | ||
{| class="wikitable" | {| class="wikitable" | ||
| Line 78: | Line 69: | ||
5. Pools Controller<br /> | 5. Pools Controller<br /> | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
2. Queue Controller | 2. Queue Controller | ||
Revision as of 04:50, 9 August 2014
Contents
Objective
Redis is an in memory key-value datastore which can organize data as strings, hashes, lists, sets, sorted sets, bitmaps and hyperloglogs. Being In-memory makes it in ideal candidate for high load, low latency messages.
The main use cases of the driver are listed below:
1. High-throughput (2K+ messages/sec per queue)
2. Lots of small messages (<= 1K)
3. Messages are short-lived (minutes, not hours)
4. High durability is optional (2 copies, RAM only)
5. HA is still very important
Components of the basic driver
The basic driver includes support for a single instance redis store as a data backend.
The driver supports 6 main required functions of a marconi storage driver which are:
Message Controllers
Helps organize messages scoped by every queue.
Organization of Data
Message id's list ( Redis Sorted Set )
Each queue in the system has a set of message ids currently in the queue. The list is sorted based on a ranking which is incremented atomically using the counter(MESSAGE_RANK_COUNTER_SUFFIX) also stored in the database for every queue.
Messages(Redis Hash):
Each message is stored using the UUID of the message as the redis key. The datastructure has the following information.
| Name | Field |
|---|---|
| id | id |
| created iso | cr_iso |
| ttl | t |
| expires | e |
| body | b |
| claim | c |
| claim expiry time | c.e |
| client uuid | u |
| created time | cr |
2. Queue Controller
3. Claims Controller
4. Catalogue Controller
5. Pools Controller
2. Queue Controller 3. Claims Controller 4. Catalogue Controller 5. Pools Controller
Current Status
The basic driver is a "Work in Progress" and is expected to be a part of the code base by Juno-3 deadline (Sept 4th 2014). The clustering and persistence extension is expected to be developed before the end of Kilo.
Blueprints
| Blueprint URL | Functionality |
|---|---|
| [1] | Redis as an alternative driver for MongoDB |
| [2] | Redis Storage Driver for Marconi |
Submitted Patches
| Patch URL | Implemented Feature |
|---|---|
| [3] | Messages and Queue Controllers |
| [4] | Claims Controller |
| [5] | Pools and Catalog Controllers |
