Redis Storage Driver Design (Zaqar)

Objective

Redis is an in memory key-value datastore which can organize data as strings, hashes, lists, sets, sorted sets, bitmaps and hyperloglogs. Being In-memory makes it in ideal candidate for high load, low latency messages.

The main use cases of the driver are listed below:

1. High-throughput (2K+ messages/sec per queue)

2. Lots of small messages (<= 1K)

3. Messages are short-lived (minutes, not hours)

4. High durability is optional (2 copies, RAM only)

5. HA is still very important

Components of the basic driver

The basic driver includes support for a single instance redis store as a data backend.

The driver supports 6 main required functions of a marconi storage driver which are:

Message Controller

Helps organize messages scoped by every queue.

Organization of Data

Message id's list ( Redis Sorted Set )
Each queue in the system has a set of message ids currently in the queue. The list is sorted based on a ranking which is incremented atomically using the counter(MESSAGE_RANK_COUNTER_SUFFIX) also stored in the database for every queue.

Messages(Redis Hash):

Each message is stored using the UUID of the message as the redis key. The datastructure has the following information.
NOTE: This datastructure is serialized using msgpack.

Name	Field
id	id
created iso	cr_iso
ttl	t
expires	e
body	b
claim	c
claim expiry time	c.e
client uuid	u
created time	cr

Queue Controller

Queues are scoped by project, which is prefixed to the queue name.

Queues ( Redis Sorted Set ):
Id: queues_set

Id

Name	Field
name	<project-id.queue-name>

The set helps faster existence checks, while the list helps paginated retrieval of queues.

Queue Information (Redis Hash):

This Data structure holds all meta information about a queue.

Scope: <project-id_q-name>

Name	Field
count(size)	c
claimed	cl
metadata	m
creation time	t

Claims Controller

Claims list ( Redis Set ) contains claim ids active for a queue.

Redis ID: <project-id_q-name>

Claim info(Redis Hash):

Contains the metadata information for a particular claim.

Name	Field
ttl	t
grace	g
id	if
claim	c
client UUID	u

Catalogue Controller

TBD

Pools Controller

TBD

Current Status

The basic driver is a "Work in Progress" and is expected to be a part of the code base by Juno-3 deadline (Sept 4th 2014). The clustering and persistence extension is expected to be developed before the end of Kilo.

Blueprints

Blueprint URL	Functionality
[1]	Redis as an alternative driver for MongoDB
[2]	Redis Storage Driver for Marconi

Submitted Patches

Patch URL	Implemented Feature
[3]	Messages and Queue Controllers
[4]	Claims Controller
[5]	Pools and Catalog Controllers