Jump to: navigation, search

Difference between revisions of "Redis Storage Driver Design (Zaqar)"

(→‎Current Status)
(→‎Components of the basic driver)
Line 18: Line 18:
 
=== Components of the basic driver ===
 
=== Components of the basic driver ===
  
 +
The basic driver includes support for a single instance redis store as a data backend.
 +
<br />
 +
 +
The driver supports 6 main required functions of a marconi storage driver which are:
 +
 +
==== Message Controllers ====
 +
Helps organize messages scoped by every queue.
 +
<br />
 +
===== Organization of Data =====
 +
Message id's list ( Redis Sorted Set )
 +
<br />
 +
Each queue in the system has a set of message ids currently in the queue. The list is sorted based on a ranking which is incremented atomically using the counter(MESSAGE_RANK_COUNTER_SUFFIX) also stored in the database for every queue.
 +
 +
Messages(Redis Hash):
 +
Each message is stored using the UUID of the message as the redis key. The datastructure has the following information.
 +
 +
{| class="wikitable"
 +
|-
 +
! Name !! Field
 +
|-
 +
| id || id
 +
|-
 +
| created iso || cr_iso
 +
|-
 +
| ttl    || t
 +
|-
 +
| expires || e
 +
|-
 +
| body || b
 +
|-
 +
| claim || c
 +
|-
 +
| claim expiry time || c.e
 +
|-
 +
| client uuid || u
 +
|-
 +
| created time || cr
 +
|}
 +
 +
2. Queue Controller
 +
3. Claims Controller
 +
4. Catalogue Controller
 +
5. Pools Controller
  
 
=== Current Status ===
 
=== Current Status ===

Revision as of 04:48, 9 August 2014

Objective

Redis is an in memory key-value datastore which can organize data as strings, hashes, lists, sets, sorted sets, bitmaps and hyperloglogs. Being In-memory makes it in ideal candidate for high load, low latency messages.

The main use cases of the driver are listed below:

1. High-throughput (2K+ messages/sec per queue)

2. Lots of small messages (<= 1K)

3. Messages are short-lived (minutes, not hours)

4. High durability is optional (2 copies, RAM only)

5. HA is still very important


Components of the basic driver

The basic driver includes support for a single instance redis store as a data backend.

The driver supports 6 main required functions of a marconi storage driver which are:

Message Controllers

Helps organize messages scoped by every queue.

Organization of Data

Message id's list ( Redis Sorted Set )
Each queue in the system has a set of message ids currently in the queue. The list is sorted based on a ranking which is incremented atomically using the counter(MESSAGE_RANK_COUNTER_SUFFIX) also stored in the database for every queue.

Messages(Redis Hash): Each message is stored using the UUID of the message as the redis key. The datastructure has the following information.

Name Field
id id
created iso cr_iso
ttl t
expires e
body b
claim c
claim expiry time c.e
client uuid u
created time cr

2. Queue Controller 3. Claims Controller 4. Catalogue Controller 5. Pools Controller

Current Status

The basic driver is a "Work in Progress" and is expected to be a part of the code base by Juno-3 deadline (Sept 4th 2014). The clustering and persistence extension is expected to be developed before the end of Kilo.

Blueprints

Blueprint URL Functionality
[1] Redis as an alternative driver for MongoDB
[2] Redis Storage Driver for Marconi

Submitted Patches

Patch URL Implemented Feature
[3] Messages and Queue Controllers
[4] Claims Controller
[5] Pools and Catalog Controllers
Retrieved from "https://wiki.openstack.org/w/index.php?title=Redis_Storage_Driver_Design_(Zaqar)&oldid=60135"