Jump to: navigation, search

Difference between revisions of "GSoC2014/Student/ChenchongQin"

(Possible Mentor)
 
(8 intermediate revisions by the same user not shown)
Line 9: Line 9:
 
= Abstract =
 
= Abstract =
  
Marconi is the messaging and notifications service of OpenStack. Using Marconi, tenants can build more complex applications on OpenStack. Marconi provides tenants with different storage backend options to help them keeping balance between latency and persistence. This Project will add a Redis Storage Backend for Marconi. With this new storage backend, tenants can benefit from the excellent latency performance provided by the Redis storage backend when building their applications using Marconi.
+
Marconi is the messaging and notification service of OpenStack. Using Marconi, tenants can build more complex applications on OpenStack. Marconi provides tenants with different storage backend options to help them keeping balance between latency and persistence. This Project will add a Redis Storage Backend for Marconi. With this new storage backend, tenants can benefit from the excellent latency performance provided by the Redis storage backend when building their applications using Marconi.
  
 
= Project Description =
 
= Project Description =
Line 15: Line 15:
 
== Possible Mentor ==
 
== Possible Mentor ==
  
[[User:Alejandro_Cabrera|Alejandro Cabrera]]
+
* [[User:Alejandro_Cabrera|Alejandro Cabrera]]
  
 
== Project Details ==
 
== Project Details ==
Line 45: Line 45:
  
 
# '''Learning'''. Reading documents/wiki, browsing codes, and gaining background knowledge.
 
# '''Learning'''. Reading documents/wiki, browsing codes, and gaining background knowledge.
# '''Implementing the Redis driver for Marconi'''. Implementing the DataDriver and the ControlDriver  
+
# '''Implementing the Redis driver for Marconi'''. Implementing the Redis DataDriver and ControlDriver.
# '''Running unit test for the driver and debugging it'''. We must make sure that the driver can work properly.
+
# '''Writing additional tests, performing tests and debugging'''. We must make sure that the driver can work properly. Provide additional tests that test the nuances of the Redis backend. Then do the testing & debugging.
# '''Implementing the various controllers'''. Including QueueController, MessageController, ClaimController, ShardController and CatalogueController.
+
# '''Benchmarking and Comparing'''. Develop a benchmarking framework. Run the benchmarking for MongoDB backend and Redis backend respectively. Then compare them.
# '''Testing and benchmarking'''. Writing additional tests, testing & debugging, benchmarking different backends and comparing them.
 
  
 
=== Benefits to OpenStack ===
 
=== Benefits to OpenStack ===
Line 65: Line 64:
  
 
* '''Milestone #1''': Redis driver for Marconi implemented
 
* '''Milestone #1''': Redis driver for Marconi implemented
* '''Milestone #2''': Various controllers implemented
+
* '''Milestone #2''': Additional tests added and tests performed
* '''Milestone #3''': Additional tests added and tests performed
+
* '''Milestone #3''': Benchmarking and comparing performed
* '''Milestone #4''': Benchmarking and comparing performed
 
  
 
=== Schedule ===
 
=== Schedule ===
  
* Apr 22 - May 18 Community bounding
+
* '''Apr 22 - May 18'''
* May 19 - Jun 1 Implementing the DataDriver and the ControlDriver
+
** Community bounding period
* Jun 2 - Jun 15 Testing & Running unit test for the driver and debugging (Milestone #1)
+
** Further contacts with mentor
* Jun 16 - Jun 22  Implementing QueueController
+
** Reading documents, browsing codes, gaining background knowledge
* Jun 23  Midterm Evaluation
+
* '''May 19 - Jun 1'''
* Jun 23 - Jun 29  Implementing MessageController
+
** Implement the DataDriver
* Jun 30 - Jul 13 Implementing ClaimController, ShardController and CatalogueController (Milestone #2)
+
** Including Queues, Messages and Claims Controllers
* Jul 14 - Jul 27 Writing additional tests, testing & debugging (Milestone #3)
+
* '''Jun 2 - Jun 15'''
* Jul 28 - Aug 10 Benchmarking different backends and comparing them (Milestone #4)
+
** Implement the ControlDriver
* Aug 11 - Aug 17 Code cleanup, testing, adding documentation and final commitment
+
** Including Shards and Catalogue Controllers
* Aug 18 Final Evaluation
+
** '''Milestone #1''' reached
 +
* '''Jun 16 - Jun 29'''
 +
** Start performing core tests
 +
** Add additional tests that test the nuances of the Redis backend
 +
** '''Midterm Evaluation''' ('''Jun 23''') in this period
 +
* '''Jun 30 - Jul 13'''
 +
** Perform tests (core tests and additional tests)
 +
** Make sure that the new driver passing all the tests
 +
** '''Milestone #2''' reached
 +
* '''Jul 14 - Jul 27'''
 +
** Develop a simple benchmarking framework for the storage backends of Marconi
 +
* '''Jul 28 - Aug 10'''
 +
** Benchmarking the different backends and comparing them
 +
** '''Milestone #3''' reached
 +
* '''Aug 11 - Aug 17'''
 +
** Code cleanup, adding documentations
 +
** Preparing for the Final Evaluation
 +
* '''Aug 18'''
 +
** We reach the '''Final Evaluation'''
 +
 
 +
== Bugs fixed for OpenStack ==
 +
 
 +
* [https://review.openstack.org/#/c/81689/ Marconi/bug/#1289355]
 +
* [https://review.openstack.org/#/c/81245/ Nova/bug/#1292993]

Latest revision as of 18:05, 21 March 2014

GSoC2014: Adding Redis as a Storage Backend to OpenStack Marconi

Student: Chenchong Qin


Abstract

Marconi is the messaging and notification service of OpenStack. Using Marconi, tenants can build more complex applications on OpenStack. Marconi provides tenants with different storage backend options to help them keeping balance between latency and persistence. This Project will add a Redis Storage Backend for Marconi. With this new storage backend, tenants can benefit from the excellent latency performance provided by the Redis storage backend when building their applications using Marconi.

Project Description

Possible Mentor

Project Details

Background

Marconi is the messaging and notifications service of OpenStack. It produces an OpenStack messaging API and service that affords a variety of distributed application messaging patterns. With Marconi, tenants can build more complex web applications running on OpenStack.

However, as we all know, requirements of tenants and their applications are highly diverse. They may need Marconi to provide different latency and persistence performances. For example, in busy online retailer systems (amazon.com on Black Friday), persistence performance is more important. It's tolerable for user's orders taking more time to be consumed by order processing system, but orders must not be dropped. On the other hand, for systems performing distributed scientific computing which require real-time interim results, latency performance is crucial. If the real-time interim results can't be passed to next-stage workers within certain time constraint, task will fail.

The different latency and persistence performances of Marconi are closely related to different storage backends. So, though two storage backends (SQLAlchemy and MongoDB) will be provided in Icehouse release of Marconi, new storage backends, like Redis, RethinkDB, etc., still need to be added. Among them, Redis is an open-source in-memory data store with optional durability. When the durability of data is not needed, the in-memory nature of Redis allows it to operate with excellent latency performance.

Project Goal

The Goal of this project is Adding Redis as a Storage Backend to Marconi.

Specifically, it contains flowing 3 sub-goals:

  • Provide the implementation of Redis storage backend that passes all core tests. It is the core part of this project dealing with development of the new storage backend.
  • Provide additional tests that test the nuances of the Redis storage backend. Though the core test suite is fairly thorough, few additional tests relating to the new backend, if any, should be needed.
  • Provide benchmarks comparing Redis backend to MongoDB backend. The 3rd goal is based on the first two. After the new backend is implemented and tested, we can benchmark different backends and compare them. Interesting results may appear!


For additional information, please check the Project Idea Page: Add a New Storage Backend.

Procedures to Reach the Goal

  1. Learning. Reading documents/wiki, browsing codes, and gaining background knowledge.
  2. Implementing the Redis driver for Marconi. Implementing the Redis DataDriver and ControlDriver.
  3. Writing additional tests, performing tests and debugging. We must make sure that the driver can work properly. Provide additional tests that test the nuances of the Redis backend. Then do the testing & debugging.
  4. Benchmarking and Comparing. Develop a benchmarking framework. Run the benchmarking for MongoDB backend and Redis backend respectively. Then compare them.

Benefits to OpenStack

This Project will benefit OpenStack of a Redis Storage Backend for its Queuing Service Marconi. With this new storage backend, tenants can benefit from the excellent latency performance of queuing service provided by the Redis storage backend when building their applications using Marconi.

Deliverables

  • Redis storage backend module (marconi.queues.storage.redis)
  • Additional test cases for nuances of Redis backend
  • Related docs (setup, usage, etc.)

Milestones & Schedule

Milestones

  • Milestone #1: Redis driver for Marconi implemented
  • Milestone #2: Additional tests added and tests performed
  • Milestone #3: Benchmarking and comparing performed

Schedule

  • Apr 22 - May 18
    • Community bounding period
    • Further contacts with mentor
    • Reading documents, browsing codes, gaining background knowledge
  • May 19 - Jun 1
    • Implement the DataDriver
    • Including Queues, Messages and Claims Controllers
  • Jun 2 - Jun 15
    • Implement the ControlDriver
    • Including Shards and Catalogue Controllers
    • Milestone #1 reached
  • Jun 16 - Jun 29
    • Start performing core tests
    • Add additional tests that test the nuances of the Redis backend
    • Midterm Evaluation (Jun 23) in this period
  • Jun 30 - Jul 13
    • Perform tests (core tests and additional tests)
    • Make sure that the new driver passing all the tests
    • Milestone #2 reached
  • Jul 14 - Jul 27
    • Develop a simple benchmarking framework for the storage backends of Marconi
  • Jul 28 - Aug 10
    • Benchmarking the different backends and comparing them
    • Milestone #3 reached
  • Aug 11 - Aug 17
    • Code cleanup, adding documentations
    • Preparing for the Final Evaluation
  • Aug 18
    • We reach the Final Evaluation

Bugs fixed for OpenStack