GSoC2014: Adding Redis as a Storage Backend to OpenStack Marconi
Student: Chenchong Qin
Marconi is the messaging and notifications service of OpenStack. Using Marconi, tenants can build more complex applications on OpenStack. Marconi provides tenants with different storage backend options to help them keeping balance between latency and persistence. This Project will add a Redis Storage Backend for Marconi. With this new storage backend, tenants can benefit from the excellent latency performance provided by the Redis storage backend when building their applications using Marconi.
Marconi is the messaging and notifications service of OpenStack. It produces an OpenStack messaging API and service that affords a variety of distributed application messaging patterns. With Marconi, tenants can build more complex web applications running on OpenStack.
However, as we all know, requirements of tenants and their applications are highly diverse. They may need Marconi to provide different latency and persistence performances. For example, in busy online retailer systems (amazon.com on Black Friday), persistence performance is more important. It's tolerable for user's orders taking more time to be consumed by order processing system, but orders must not be dropped. On the other hand, for systems performing distributed scientific computing which require real-time interim results, latency performance is crucial. If the real-time interim results can't be passed to next-stage workers within certain time constraint, task will fail.
The different latency and persistence performances of Marconi are closely related to different storage backends. So, though two storage backends (SQLAlchemy and MongoDB) will be provided in Icehouse release of Marconi, new storage backends, like Redis, RethinkDB, etc., still need to be added. Among them, Redis is an open-source in-memory data store with optional durability. When the durability of data is not needed, the in-memory nature of Redis allows it to operate with excellent latency performance.
The Goal of this project is Adding Redis as a Storage Backend to Marconi.
Specifically, it contains flowing 3 sub-goals:
- Provide the implementation of Redis storage backend that passes all core tests. It is the core part of this project dealing with development of the new storage backend.
- Provide additional tests that test the nuances of the Redis storage backend. Though the core test suite is fairly thorough, few additional tests relating to the new backend, if any, should be needed.
- Provide benchmarks comparing Redis backend to MongoDB backend. The 3rd goal is based on the first two. After the new backend is implemented and tested, we can benchmark different backends and compare them. Interesting results may appear!
For additional information, please check the Project Idea Page: Add a New Storage Backend.
Procedures to Reach the Goal
- Learning. Reading documents/wiki, browsing codes, and gaining background knowledge.
- Implementing the Redis driver for Marconi. Implementing the DataDriver and the ControlDriver
- Running unit test for the driver and debugging it. We must make sure that the driver can work properly.
- Implementing the various controllers. Including QueueController, MessageController, ClaimController, ShardController and CatalogueController.
- Testing and benchmarking. Writing additional tests, testing & debugging, benchmarking different backends and comparing them.
Benefits to OpenStack
This Project will benefit OpenStack of a Redis Storage Backend for its Queuing Service Marconi. With this new storage backend, tenants can benefit from the excellent latency performance of queuing service provided by the Redis storage backend when building their applications using Marconi.
- Redis storage backend module (marconi.queues.storage.redis)
- Additional test cases for nuances of Redis backend
- Related docs (setup, usage, etc.)
Milestones & Schedule
- Milestone #1: Redis driver for Marconi implemented
- Milestone #2: Various controllers implemented
- Milestone #3: Additional tests added and tests performed
- Milestone #4: Benchmarking and comparing performed
- Apr 22 - May 18 Community bounding
- May 19 - Jun 1 Implementing the DataDriver and the ControlDriver
- Jun 2 - Jun 15 Testing & Running unit test for the driver and debugging (Milestone #1)
- Jun 16 - Jun 22 Implementing QueueController
- Jun 23 Midterm Evaluation
- Jun 23 - Jun 29 Implementing MessageController
- Jun 30 - Jul 13 Implementing ClaimController, ShardController and CatalogueController (Milestone #2)
- Jul 14 - Jul 27 Writing additional tests, testing & debugging (Milestone #3)
- Jul 28 - Aug 10 Benchmarking different backends and comparing them (Milestone #4)
- Aug 11 - Aug 17 Code cleanup, testing, adding documentation and final commitment
- Aug 18 Final Evaluation