OpenStack Message Bus ("Marconi")

Marconi is a new OpenStack project to create a multi-tenant, light-weight, fast, scale-out alternative to other popular queuing systems. The service will support 100's of thousands (or even millions) of clients and channels in a cost-effective manner, without sacrificing performance, durability, or HA.

This project's objective is not to supplant intra-cluster message buses, such as those based on AMQP and DDS. Rather, Marconi provides a complementary service that is optimized for extra-cluster messaging between many thousands of Internet-connected agents.

Marconi will define a clean, RESTful API, use a modular architecture, and will support both eventing and job-queuing semantics. Users will be able to customize Marconi to achieve a wide range of performance, durability, availability, and efficiency goals.

Design

Marconi's design philosophy is derived from Donald A. Norman's work regarding The Design of Everyday Things:

 The value of a well-designed object is when it has such a rich set of affordances that the people who use it can do things with it that the designer never imagined.

Goals related to the above:

Emergent functionality, utility
Modular, pluggable code base
REST architectural style

Principles to live by:

DRY
YAGNI
KISS

Use Cases

Distribute tasks among multiple workers (transactional job queues)
Forward events to data collectors (transactional event queues)
Publish events to any number of subscribers (pub-sub)
Send commands to one or more agents (point-to-point or pub-sub)
Request information from an agent

Out of Scope

Marconi may be used as the foundation for other services to support the following use cases, but will not support them directly within its code base.

Forwarding notifications to email, SMS, Twitter, etc.
Forwarding notifications to web hooks
Forwarding notifications to APNS, GCM, etc.
Scheduling-as-a-service
Metering usage

Resources

Source code

Bug tracker

Blueprints

Developer Docs

Grizzly Milestones

v0.1

Build on the initial design session held during the Grizzly Design Summit (see also the design session notes).

Set up Marconi wiki, repo, Launchpad, etc.
Define v1.0 functional and non-functional requirements
Define v1.0 API
Create a skeleton implementation
Create test suite

v0.2

What's the simplest thing that could possibly work?

Stabilize v1.0 requirements and API based on community feedback
Flesh out the skeleton implementation with reference modules
Create and publish developer docs
Create a basic CLI client

v0.3

Make it real.

Finalize v1.0 requirements and API
Create production-ready modules
Achieve ~80% code coverage in tests, fix all critical bugs

v0.4

Make it awesome.

Set up continuos integration
Conduct performance and load testing; optimize the code accordingly, and publish results
Conduct integration testing and fix all critical bugs
Clean up all code and documentation
Document the API for end users

v0.5

Go live.

Production trial, publish results
Define next set of features and milestones

Zaqar