Redis Storage Driver Design (Zaqar)
Redis is an in memory key-value datastore which can organize data as strings, hashes, lists, sets, sorted sets, bitmaps and hyperloglogs. Being In-memory makes it in ideal candidate for high load, low latency messages.
The main use cases of the driver are listed below:
1. High-throughput (2K+ messages/sec per queue)
2. Lots of small messages (<= 1K)
3. Messages are short-lived (minutes, not hours)
4. High durability is optional (2 copies, RAM only)
5. HA is still very important
Components of the basic driver
The basic driver includes support for a single instance redis store as a data backend.
The driver supports 6 main required functions of a marconi storage driver which are:
Helps organize messages scoped by every queue.
Organization of Data
Message id's list ( Redis Sorted Set )
Each queue in the system has a set of message ids currently in the queue. The list is sorted based on a ranking which is incremented atomically using the counter(MESSAGE_RANK_COUNTER_SUFFIX) also stored in the database for every queue.
Each message is stored using the UUID of the message as the redis key. The datastructure has the following information.
NOTE: This datastructure is serialized using msgpack.
|claim expiry time||c.e|
Queues are scoped by project, which is prefixed to the queue name.
Queues ( Redis Sorted Set ):
The set helps faster existence checks, while the list helps paginated retrieval of queues.
Queue Information (Redis Hash):
This Data structure holds all meta information about a queue.
Claims list ( Redis Set ) contains claim ids active for a queue.
Redis ID: <project-id_q-name>
Claim info(Redis Hash):
Contains the metadata information for a particular claim.
The basic driver is a "Work in Progress" and is expected to be a part of the code base by Juno-3 deadline (Sept 4th 2014). The clustering and persistence extension is expected to be developed before the end of Kilo.
|||Redis as an alternative driver for MongoDB|
|||Redis Storage Driver for Marconi|
|Patch URL||Implemented Feature|
|||Messages and Queue Controllers|
|||Pools and Catalog Controllers|