Difference between revisions of "Zaqar"
(→Summary) |
(→Mission) |
||
Line 2: | Line 2: | ||
===Mission=== | ===Mission=== | ||
− | To produce a web-friendly OpenStack messaging API and service that affords | + | To produce a web-friendly OpenStack messaging API and service that affords common distributed application messaging patterns in an efficient, scalable and highly-available manner, and to create and maintain associated Python libraries and documentation. |
===Summary=== | ===Summary=== |
Revision as of 16:25, 25 March 2014
Contents
Mission
To produce a web-friendly OpenStack messaging API and service that affords common distributed application messaging patterns in an efficient, scalable and highly-available manner, and to create and maintain associated Python libraries and documentation.
Summary
Marconi provides an HTTP messaging platform for web and mobile application developers, supporting publish-subcribe, producer-consumer, point-to-point, and hyrbrid communication patterns. The project provides a pragmatic, easy-to-use API designed according to direct feedback from users, and allows for a variety of options for backend storage, allowing operators to tailor the performance, durability, and scalability of their offerings to achieve a best-fit messaging solution for both themselves and their users.
Detailed Description
In order to support more complex web applications running on OpenStack, a messaging service was needed. To fill this need, the Marconi project was proposed at the Grizzly design summit. Requirements were discussed with the community and used to form the basis for the project's charter. Implementation began in February 2013, and we have been fortunate to receive regular contributions from Red Hat, Rackspace, IBM, and others since that time.
Marconi's overarching goal is to provide web-scale, highly-available messaging to web applications that run on OpenStack. Marconi runs on Nova servers, behind OpenStack load balancers and uses Keystone authentication middleware. The Marconi implementation makes use of Oslo, and follows the standard OpenStack hacking guidelines.
Marconi provides an interface for posting messages, and later claiming those messages for processing. It also provides an interface to clients for listing messages without needing to claim them (ala RSS and Atom), in order to support pub-sub and passive auditing of producer-consumer workflows. The service guarantees first in, first out (FIFO) order for single producer models, best-effort ordering otherwise.
The Marconi architecture is pluggable in terms of both transport and storage. Reference drivers for HTTP (WSGI), SQLAlchemy, and MongoDB will be provided in the initial release (Icehouse), along with a SQLite driver to facilitate development and testing. Other transport and storage drivers have been proposed, and are currently under discussion. Marconi deployments will support HA, and will be able to scale horizontally in the transport and storage layers to enable large deployments. A routing proxy and migration service is also under development, to provide further horizontal scaling across multiple independent Marconi partitions, for use in extremely large deployments.
The Marconi v1.0 API defines the following operations:
API:
- Get JSON home document
- Check node health
Queues:
- Create a queue
- List queues
- Set queue metadata
- Get queue metadata
- Get queue stats
- Delete a queue
Messages
- Post one or more messages
- List messages
- Get a message
- Delete a message
- Get multiple messages
- Delete multiple messages
Claims
- Claim messages
- Query a claim
- Update a claim
- Release a claim
Core Team
Kurt Griffiths (kgriffs) is the current PTL. Other core team members include the following individuals who have demonstrated solid mentoring skills and good judgment while contributing to the program.
- Flavio Percoco (flaper87)
- Alejandro Cabrera (cpp-cabrera)
- Fei Long Wang (flwang)
- [your name could be here!]
Design
Marconi aims to be pragmatic, building upon the real-world experiences of teams who have solid track records running and supporting web-scale message queueing systems. The project's overarching design philosophy is derived from Donald A. Norman:
The value of a well-designed object is when it has such a rich set of affordances that the people who use it can do things with it that the designer never imagined.
Goals related to the above:
- Emergent functionality, utility
- Modular, pluggable code base
- REST architectural style
Principles to live by:
- DRY
- YAGNI
- KISS
Architecture
Use Cases
- Distribute tasks among multiple workers (transactional job queues)
- Forward events to data collectors (transactional event queues)
- Publish events to any number of subscribers (pub-sub)
- Send commands to one or more agents (point-to-point or pub-sub)
- Request an action or get information from an agent (RPC)
Out of Scope
Marconi may be used as the foundation for other services to support the following use cases, but will not support them directly within its code base.
- Forwarding notifications to email, SMS, Twitter, etc.
- Forwarding notifications to web hooks
- Forwarding notifications to APNS, GCM, etc.
- Scheduling-as-a-service
- Metering usage
Etherpads
- Marconi Client Brainstorm #1
- Common Code Improvements
- ZMQ Brainstorm #1
- FIFO Etc.
- Tests Refactoring
- Placement Service
- Marconi Proxy Issues
Presentations
- Alejandro Cabrera. Rackspace Atlanta. Introducing Openstack Marconi. July 17, 2013. Youtube Speaker Deck
- Flavio Percoco. EuroPython 2013. Marconi: Queuing and Notification Service for Openstack. July 2, 2013. YouTube
- Kurt Griffiths, Allan Metts. Openstack Summit April 2013. Project Overview: OpenStack Queuing and Notification Service ("Marconi""). April 2013. YouTube
- Kurt Griffiths, Flavio Percoco, Allan Metts. Openstack Summit November 2013. Openstack Queuing and Notification Service Marconi. November 2013. YouTube
Articles
- Oz Akan. Rackspace Devops Blog. July 25, 2013. Openstack Marconi API.
FAQ
Will Marconi work with AMQP?
- Planned as a backend for v2 API
- Transport TBD
- Talk to us about use cases
- Need contributors
How does Marconi compare to AWS (SQS/SNS)?
- Targets similar workloads
- Marconi will provide a unified API to handle notifications and queuing
- Marconi is highly customizable
- FIFO and once-and-only-once guaranteed (depending on storage backend)
How mature is the project?
- Marconi is used in production: marconi_users
- Number of contributors is growing
What's next for marconi?
- Releasing API v1.1 for Icehouse
- Additional storage drivers
- Storage sharding: scaling horizontally and heterogeneous storage
- Notifications
- Message signing
- Additional ops features
- Reference client library
See also the Icehouse Roadmap.
How easy is it to contribute/get up and running?
- You don't need devstack
- You decide what you want to work on: storage, transport, client-side
- Very decoupled
- Easy: choose a bug and submit a patch
Resources
Meetings | Meetings/Marconi |
IRC | #openstack-marconi on Freenode |
Trello | https://trello.com/b/7NLODgbr (deprecated in favor of Launchpad) |
Havana Spec | Marconi/specs/havana |
API Blueprint | Marconi/specs/api/v1 |
Other Blueprints | https://blueprints.launchpad.net/marconi |
Milestones | https://launchpad.net/marconi/+milestones |
Developer Docs | Tips for Developers Python Client Bindings Drivers Guarantees |
Incubation | Marconi/Incubation |
Source code | https://github.com/openstack/marconi |
Client source code | https://github.com/openstack/python-marconiclient |
Code Reviews | https://review.openstack.org/#/q/status:open+project:stackforge/marconi,n,z |
Bug tracker | https://bugs.launchpad.net/marconi |