Jump to: navigation, search

Difference between revisions of "Zaqar"

(→‎How easy is it to contribute/get up and running?)
(Move new home page into place)
Line 1: Line 1:
 +
= Messaging Service ("Marconi") =
  
−
===Vision===
+
Marconi is a cloud messaging and notification service for developers building applications on top of OpenStack. The service features a web-friendly HTTP API, which developers can use to send messages between the various components of their SaaS and mobile applications, using a variety of communication patterns. Underlying the API is an efficient messaging engine designed with scalability and security in mind.
  
−
To produce a web-friendly OpenStack messaging API and service for web and mobile application developers that affords common distributed application messaging patterns in an efficient, scalable and highly-available manner, and to create and maintain associated Python libraries and documentation.
+
At first glance, Marconi's RESTful API may seem a bit out of the ordinary. Message brokers typically rely on a custom binary protocol and long-lived connections. However, many of today's developers prefer a more web-friendly HTTP API. They value the simplicity and transparency of the protocol, its firewall-friendly nature, and its huge ecosystem of tools, load balancers and proxies. In addition, cloud operators appreciate the scalability aspects of the REST architectural style.
  
−
===Mission===
+
Several OpenStack programs have expressed interest in integrating with Marconi to surface events to end users and user-agents. We look forward to collaborating with these and other programs on their individual use cases:
 +
* Heat
 +
* Swift
 +
* Horizon
 +
* Guest Agent
 +
* Ceilometer
  
−
Marconi provides an HTTP messaging platform, supporting publish-subcribe, producer-consumer, point-to-point, and hyrbrid communication patterns. The project provides a pragmatic, easy-to-use API designed according to direct feedback from users. Operators may choose from a variety of options for backend storage, allowing them to tailor the performance, durability, and scalability of their offerings to achieve a best-fit messaging solution for both themselves and their users.
+
== Resources ==
−
 
 
−
===Detailed Description===
 
−
 
 
−
In order to support more complex web applications running on OpenStack, a messaging service was needed. To fill this need, the Marconi project was proposed at the Grizzly design summit. Requirements were discussed with the community and used to form the basis for the project's charter. Implementation began in February 2013, and we have been fortunate to receive regular contributions from Red Hat, Rackspace, IBM, and others since that time.
 
−
 
 
−
Marconi's overarching goal is to provide web-scale, highly-available messaging to '''web applications that run on OpenStack'''. Marconi runs on Nova servers, behind OpenStack load balancers and uses Keystone authentication middleware. The Marconi implementation makes use of Oslo, and follows the standard OpenStack hacking guidelines.
 
−
 
 
−
Marconi provides an interface for posting messages, and later claiming those messages for processing. It also provides an interface to clients for listing messages without needing to claim them (ala RSS and Atom), in order to support pub-sub and passive auditing of producer-consumer workflows. The service guarantees first in, first out (FIFO) order for single producer models, best-effort ordering otherwise.
 
−
 
 
−
The Marconi architecture is pluggable in terms of both transport and storage. Reference drivers for HTTP (WSGI), SQLAlchemy, and MongoDB will be provided in the initial release (Icehouse), along with a SQLite driver to facilitate development and testing. Other transport and storage drivers have been proposed, and are currently under discussion. Marconi deployments will support HA, and will be able to scale horizontally in the transport and storage layers to enable large deployments. A routing proxy and migration service is also under development, to provide further horizontal scaling across multiple independent Marconi partitions, for use in extremely large deployments.
 
−
 
 
−
The Marconi v1.0 API defines the following operations:
 
−
 
 
−
'''API''':
 
−
 
 
−
:Get JSON home document
 
−
:Check node health
 
−
 
 
−
'''Queues''':
 
−
 
 
−
:Create a queue
 
−
:List queues
 
−
:Set queue metadata
 
−
:Get queue metadata
 
−
:Get queue stats
 
−
:Delete a queue
 
−
 
 
−
'''Messages'''
 
−
:Post one or more messages
 
−
:List messages
 
−
:Get a message
 
−
:Delete a message
 
−
:Get multiple messages
 
−
:Delete multiple messages
 
−
 
 
−
'''Claims'''
 
−
:Claim messages
 
−
:Query a claim
 
−
:Update a claim
 
−
:Release a claim
 
−
 
 
−
=== Core Reviewers ===
 
−
 
 
−
Kurt Griffiths (kgriffs) is the current PTL, and is joined by the following core team members who have demonstrated solid '''mentoring skills''' and '''good judgment''' while contributing to the program.
 
−
 
 
−
* Flavio Percoco (flaper87)
 
−
* Alejandro Cabrera (cpp-cabrera)
 
−
* Fei Long Wang (flwang)
 
−
* Malini Kamalambal (malini)
 
−
* [your name could be here!]
 
−
 
 
−
=== Champions ===
 
−
 
 
−
* '''Oslo Integration:''' Flavio Percoco (flaper87)
 
−
* '''Application Security:''' Malini Kamalambal (malini), Jamie Painter (painterjd), and Kurt Griffiths (kgriffs)
 
−
 
 
−
== Design ==
 
−
 
 
−
Marconi aims to be pragmatic, building upon the real-world experiences of teams who have solid track records running and supporting web-scale message queueing systems. The project's overarching design philosophy is derived from Donald A. Norman:
 
−
 
 
−
<blockquote>The value of a well-designed object is when it has such a rich set of affordances that the people who use it can do things with it that the designer never imagined.</blockquote>
 
−
 
 
−
Goals related to the above:
 
  
−
# Emergent functionality, utility
+
The Marconi team hangs out in '''#openstack-marconi''' on Freenode. Feel free to stop in anytime to chat with us; the crew is always happy to hear your ideas and answer questions. If you run into a bug, please  report it using our [https://bugs.launchpad.net/marconi issue tracker].
−
# Modular, pluggable code base
 
−
# REST architectural style
 
  
 +
==== Docs ====
  
−
Principles to live by:
+
* [http://docs.openstack.org/developer/marconi/ Marconi User Guide]
 +
* [[Marconi/docs/about|About the Program]]
  
−
# DRY
+
==== Releases ====
−
# YAGNI
 
−
# KISS
 
  
−
== Architecture ==
+
<!-- TODO(kgriffs): Add release notes -->
−
[[File:Marconi-architecture.png|none]]
+
* 2014.1 ([http://tarballs.openstack.org/marconi/marconi-2014.1.tar.gz download] | [[Marconi/releases/icehouse|notes]])
  
−
== Use Cases ==
+
==== Talks ====
−
 
 
−
# Distribute tasks among multiple workers (transactional job queues)
 
−
# Forward events to data collectors  (transactional event queues)
 
−
# Publish events to any number of subscribers (pub-sub)
 
−
# Send commands to one or more agents (point-to-point or pub-sub)
 
−
# Request an action or get information from an agent (RPC)
 
−
 
 
−
== Etherpads ==
 
−
 
 
−
* [https://etherpad.openstack.org/marconi-client Marconi Client Brainstorm #1]
 
−
* [https://etherpad.openstack.org/apiclient-marconi Common Code Improvements]
 
−
* [https://etherpad.openstack.org/marconi-client-zmq ZMQ Brainstorm #1]
 
−
* [https://etherpad.openstack.org/marconi-fifo FIFO Etc.]
 
−
* [https://etherpad.openstack.org/marconi-test-refactoring Tests Refactoring]
 
−
* [https://etherpad.openstack.org/marconi-placement Placement Service]
 
−
* [https://etherpad.openstack.org/marconi-proxy-issues Marconi Proxy Issues]
 
−
 
 
−
== Presentations ==
 
  
 
* Alejandro Cabrera. Rackspace Atlanta. Introducing Openstack Marconi. July 17, 2013. [http://www.youtube.com/watch?v=b-IWDG7kKGQ Youtube] [https://speakerdeck.com/cabrera/introducing-openstack-marconi Speaker Deck]
 
* Alejandro Cabrera. Rackspace Atlanta. Introducing Openstack Marconi. July 17, 2013. [http://www.youtube.com/watch?v=b-IWDG7kKGQ Youtube] [https://speakerdeck.com/cabrera/introducing-openstack-marconi Speaker Deck]
Line 110: Line 33:
 
* Kurt Griffiths, Flavio Percoco, Allan Metts. Openstack Summit November 2013. Openstack Queuing and Notification Service Marconi. November 2013. [http://www.youtube.com/watch?v=IgE3Jskqtek YouTube]
 
* Kurt Griffiths, Flavio Percoco, Allan Metts. Openstack Summit November 2013. Openstack Queuing and Notification Service Marconi. November 2013. [http://www.youtube.com/watch?v=IgE3Jskqtek YouTube]
  
−
== Articles ==
+
==== Articles ====
  
 
* Oz Akan. Rackspace Devops Blog. July 25, 2013. [http://developer.rackspace.com/blog/openstack-marconi-api.html Openstack Marconi API].
 
* Oz Akan. Rackspace Devops Blog. July 25, 2013. [http://developer.rackspace.com/blog/openstack-marconi-api.html Openstack Marconi API].
 +
* Flavio Percoco. Developer's Black Hole. March 2014. [http://blog.flaper87.com/post/531cd585d987d24e83f082a5 People don't like to queue up].
  
−
== FAQ ==
+
== Contributor Guide ==
  
−
=== Will Marconi work with AMQP? ===
+
==== Getting Started ====
  
−
* Planned as a backend for v2 API
+
* [[Marconi/docs/contrib/intro|Introduction]]
−
* Transport TBD
+
* [[Marconi/docs/contrib/tutorials/patch|Your First Patch]]
−
* Talk to us about use cases
+
* [[Marconi/docs/contrib/tutorials/review|Your First Review]]
−
* Need contributors
+
* [[Marconi/docs/contrib/tutorials/wtt|Write the Tests!]]
 +
* [[Marconi/docs/contrib/wtd|Write the Docs!]]
 +
<!-- TODO(kgriffs): screencasts -->
  
−
=== How does Marconi compare to AWS (SQS/SNS)? ===
+
==== API Specs ====
  
−
* Targets similar workloads
+
* [[Marconi/specs/api/v1|API v1.0]] ('''released in 2014.1''')
−
* Marconi will provide a unified API to handle notifications and queuing
+
* [[Marconi/specs/api/v1.1|API v1.1]]
−
* Marconi is highly customizable
+
* [[Marconi/specs/api/v2.0|API v2.0]]
−
* FIFO and once-and-only-once guaranteed (depending on storage backend)
 
  
−
=== How does Marconi compare to oslo.messaging? ===
+
==== Design ====
  
−
* [[Oslo/Messaging | oslo.messsaging]] is a library -- Marconi is a service over a HTTP API
+
* [[Marconi/docs/design/server|Server Architecture]]
−
** You can write a messaging layer with oslo.messaging
+
* [[Marconi/docs/design/client|Client Library]]
−
** You launch Marconi with a WSGI server
 
−
* Marconi is tenant-aware -- oslo.messaging is not
 
−
** Queues created using Marconi are segregated by project-id
 
−
* oslo.messaging delegates to other brokers -- Marconi delegates to backend storage
 
−
** oslo.messaging: kombu, rabbitmq, qpid, zeromq
 
−
** Marconi: mongodb, sqlalchemy
 
  
−
=== How mature is the project? ===
+
==== Quality Engineering ====
  
−
* Marconi is used in production: [[Who Uses Marconi? | marconi_users]]
+
* [[Marconi/docs/contrib/quality/security|Application Security]]
−
* Number of contributors is growing
+
* [[Marconi/docs/contrib/quality/performance|Performance]]
 +
* [[Marconi/docs/contrib/quality/functional|Functional Tests]]
 +
* [[Marconi/docs/contrib/quality/integration|Tempest Tests]]
  
−
=== What's next for marconi? ===
+
==== Resources ====
  
−
* Releasing API v1.1 for Icehouse
+
* [[Marconi/docs/about|About the Program]]
−
* Additional storage drivers
+
* [[Marconi/docs/contrib/style|Programming Style Guide]]
−
* Storage sharding: scaling horizontally and heterogeneous storage
+
* [[Marconi/docs/contrib/process|Development Process]]
−
* Notifications
+
* [[Marconi/docs/contrib/faq|Frequently Asked Questions]]
−
* Message signing
+
* [[Marconi/docs/contrib/help|Getting Help]]
−
* Additional ops features
 
−
* Reference client library
 
  
 +
==== Community ====
  
−
See also the [[Marconi/roadmaps/icehouse|Icehouse Roadmap]].
+
* [https://webchat.freenode.net/?channels=openstack-marconi #openstack-marconi] (IRC)
−
 
+
* [[Meetings/Marconi|Meetings]]
−
=== How easy is it to contribute/get up and running? ===
+
* [https://launchpad.net/marconi/+milestones Milestones]
−
 
+
* [https://blueprints.launchpad.net/marconi Blueprints]
−
* You don't need devstack
+
* [https://bugs.launchpad.net/marconi Bugs]
−
* You decide what you want to work on: storage, transport, client-side, docs, tests
+
* [https://github.com/openstack/marconi Server Code]
−
* Very decoupled
+
* [https://github.com/openstack/python-marconiclient Client Code]
−
* Easy: choose a bug and submit a patch
 
−
 
 
−
== Resources ==
 
  
−
{| border="1" cellpadding="2" cellspacing="0"
+
== Administrivia ==
−
|  Meetings
 
−
| [[Meetings/Marconi]]
 
−
|-
 
−
|  IRC
 
−
| #openstack-marconi on Freenode
 
−
|-
 
−
| Using Marconi in OpenStack
 
−
| [[Marconi/collaborate]]
 
−
|-
 
−
|  API Blueprint 
 
−
| [[Marconi/specs/api/v1]]<br/>
 
−
[[Marconi/specs/zmq/api/v1]]<br/>
 
−
[[Marconi/specs/sharding/v1|Sharding]]
 
−
|-
 
−
|  Other Blueprints   
 
−
| https://blueprints.launchpad.net/marconi
 
−
|-
 
−
|  Milestones 
 
−
| https://launchpad.net/marconi/+milestones
 
−
|-
 
−
|  Developer Docs 
 
−
| [[Marconi/DevTips|Tips for Developers]]<br/>[[Marconi/PythonClient|Python Client Bindings]]<br>[[Marconi/Drivers|Drivers]]<br>[[Marconi/guarantees|Guarantees]]
 
−
|-
 
−
|  Incubation 
 
−
| [[Marconi/Incubation]]
 
−
|-
 
−
| Source code 
 
−
| https://github.com/openstack/marconi
 
−
|-
 
−
| Client source code
 
−
| https://github.com/openstack/python-marconiclient
 
−
|-
 
−
| Code Reviews
 
−
| https://review.openstack.org/#/q/status:open+project:stackforge/marconi,n,z
 
−
|-
 
−
|  Bug tracker 
 
−
| https://bugs.launchpad.net/marconi
 
−
|}
 
  
−
----
+
* [[Marconi/docs/admin/roadmap|Roadmap]]
−
[[Category:Homepage]]
+
* [[Marconi/docs/admin/incubation|Incubation Status]]
 +
* [[Marconi/docs/admin/release/how-to-release|How to Release]]

Revision as of 17:08, 10 May 2014

Messaging Service ("Marconi")

Marconi is a cloud messaging and notification service for developers building applications on top of OpenStack. The service features a web-friendly HTTP API, which developers can use to send messages between the various components of their SaaS and mobile applications, using a variety of communication patterns. Underlying the API is an efficient messaging engine designed with scalability and security in mind.

At first glance, Marconi's RESTful API may seem a bit out of the ordinary. Message brokers typically rely on a custom binary protocol and long-lived connections. However, many of today's developers prefer a more web-friendly HTTP API. They value the simplicity and transparency of the protocol, its firewall-friendly nature, and its huge ecosystem of tools, load balancers and proxies. In addition, cloud operators appreciate the scalability aspects of the REST architectural style.

Several OpenStack programs have expressed interest in integrating with Marconi to surface events to end users and user-agents. We look forward to collaborating with these and other programs on their individual use cases:

  • Heat
  • Swift
  • Horizon
  • Guest Agent
  • Ceilometer

Resources

The Marconi team hangs out in #openstack-marconi on Freenode. Feel free to stop in anytime to chat with us; the crew is always happy to hear your ideas and answer questions. If you run into a bug, please report it using our issue tracker.

Docs

Releases

Talks

  • Alejandro Cabrera. Rackspace Atlanta. Introducing Openstack Marconi. July 17, 2013. Youtube Speaker Deck
  • Flavio Percoco. EuroPython 2013. Marconi: Queuing and Notification Service for Openstack. July 2, 2013. YouTube
  • Kurt Griffiths, Allan Metts. Openstack Summit April 2013. Project Overview: OpenStack Queuing and Notification Service ("Marconi""). April 2013. YouTube
  • Kurt Griffiths, Flavio Percoco, Allan Metts. Openstack Summit November 2013. Openstack Queuing and Notification Service Marconi. November 2013. YouTube

Articles

Contributor Guide

Getting Started

API Specs

Design

Quality Engineering

Resources

Community

Administrivia

Retrieved from "https://wiki.openstack.org/w/index.php?title=Zaqar&oldid=51972"