Jump to: navigation, search

Difference between revisions of "Redis Storage Driver Design (Zaqar)"

(→‎Components of the basic driver)
(→‎Organization of Data)
Line 30: Line 30:
 
<br />
 
<br />
 
Each queue in the system has a set of message ids currently in the queue. The list is sorted based on a ranking which is incremented atomically using the counter(MESSAGE_RANK_COUNTER_SUFFIX) also stored in the database for every queue.
 
Each queue in the system has a set of message ids currently in the queue. The list is sorted based on a ranking which is incremented atomically using the counter(MESSAGE_RANK_COUNTER_SUFFIX) also stored in the database for every queue.
 +
<br />
  
 
Messages(Redis Hash):
 
Messages(Redis Hash):
 +
<br />
 
Each message is stored using the UUID of the message as the redis key. The datastructure has the following information.
 
Each message is stored using the UUID of the message as the redis key. The datastructure has the following information.
  
Line 43: Line 45:
 
|-
 
|-
 
| ttl    || t
 
| ttl    || t
 +
Each message is stored using the UUID of the message as the redis key. The datastructure has the following information.
 +
 +
{| class="wikitable"
 +
|-
 +
! Name !! Field
 +
|-
 +
| id || id
 +
|-
 +
| created iso || cr_iso
 +
|-
 +
| ttl    || t
 +
|-
 +
| expires || e
 +
|-
 +
| body || b
 +
|-
 +
| claim || c
 +
|-
 +
| claim expiry time || c.e
 +
|-
 +
| client uuid || u
 +
|-
 +
| created time || cr
 +
|}
 +
<br />
 +
 +
2. Queue Controller<br />
 +
 +
3. Claims Controller<br />
 +
 +
4. Catalogue Controller<br />
 +
 +
5. Pools Controller<br />
 +
 +
 
|-
 
|-
 
| expires || e
 
| expires || e

Revision as of 04:49, 9 August 2014

Objective

Redis is an in memory key-value datastore which can organize data as strings, hashes, lists, sets, sorted sets, bitmaps and hyperloglogs. Being In-memory makes it in ideal candidate for high load, low latency messages.

The main use cases of the driver are listed below:

1. High-throughput (2K+ messages/sec per queue)

2. Lots of small messages (<= 1K)

3. Messages are short-lived (minutes, not hours)

4. High durability is optional (2 copies, RAM only)

5. HA is still very important


Components of the basic driver

The basic driver includes support for a single instance redis store as a data backend.

The driver supports 6 main required functions of a marconi storage driver which are:

Message Controllers

Helps organize messages scoped by every queue.

Organization of Data

Message id's list ( Redis Sorted Set )
Each queue in the system has a set of message ids currently in the queue. The list is sorted based on a ranking which is incremented atomically using the counter(MESSAGE_RANK_COUNTER_SUFFIX) also stored in the database for every queue.

Messages(Redis Hash):
Each message is stored using the UUID of the message as the redis key. The datastructure has the following information.

Name Field
id id
created iso cr_iso
ttl t

Each message is stored using the UUID of the message as the redis key. The datastructure has the following information.

Name Field
id id
created iso cr_iso
ttl t
expires e
body b
claim c
claim expiry time c.e
client uuid u
created time cr


2. Queue Controller

3. Claims Controller

4. Catalogue Controller

5. Pools Controller


expires e
body b
claim c
claim expiry time c.e
client uuid u
created time cr

2. Queue Controller 3. Claims Controller 4. Catalogue Controller 5. Pools Controller

Current Status

The basic driver is a "Work in Progress" and is expected to be a part of the code base by Juno-3 deadline (Sept 4th 2014). The clustering and persistence extension is expected to be developed before the end of Kilo.

Blueprints

Blueprint URL Functionality
[1] Redis as an alternative driver for MongoDB
[2] Redis Storage Driver for Marconi

Submitted Patches

Patch URL Implemented Feature
[3] Messages and Queue Controllers
[4] Claims Controller
[5] Pools and Catalog Controllers
Retrieved from "https://wiki.openstack.org/w/index.php?title=Redis_Storage_Driver_Design_(Zaqar)&oldid=60136"