Jump to: navigation, search

Difference between revisions of "Zaqar"

(Summary)
(Status)
 
(84 intermediate revisions by 8 users not shown)
Line 1: Line 1:
 +
== Projects ==
  
===Mission===
+
=== Zaqar ===
  
To produce a web-friendly OpenStack messaging API and service that affords a variety of distributed application messaging patterns in an efficient, scalable and highly-available manner, and to create and maintain associated Python libraries and documentation.
+
{| border="1" cellpadding="2"
 +
| Official name
 +
| '''OpenStack Messaging'''
 +
|-
 +
| Source code
 +
| https://github.com/openstack/zaqar
 +
|-
 +
| Bug tracker
 +
| https://bugs.launchpad.net/zaqar
 +
|-
 +
| Feature tracker
 +
| https://blueprints.launchpad.net/zaqar
 +
|-
 +
| Contributor documentation
 +
| http://docs.openstack.org/developer/zaqar/
 +
|}
  
===Summary===
+
=== Python Zaqar Client ===
 +
{| border="1" cellpadding="2"
 +
| Source code
 +
| https://github.com/openstack/python-zaqarclient
 +
|-
 +
| Bug tracker
 +
| https://bugs.launchpad.net/python-zaqarclient
 +
|-
 +
| Feature tracker
 +
| https://blueprints.launchpad.net/python-zaqarclient
 +
|}
  
Marconi provides an HTTP messaging platform for web and mobile application developers, supporting publish-subcribe, producer-consumer, point-to-point, and hyrbrid communication patterns. The project provides a pragmatic, easy-to-use API designed according to direct feedback from users, and allows for a variety of options for backend storage, allowing operators to tailor the performance, durability, and scalability of their offerings to achieve a best-fit messaging solution for both themselves and their users.
+
== Overview ==
  
===Detailed Description===
+
Zaqar is a multi-tenant cloud messaging service for web and mobile developers. It combines the ideas pioneered by Amazon's SQS product with additional semantics to support event broadcasting.
  
In order to support more complex web applications running on OpenStack, a messaging service was needed. To fill this need, the Marconi project was proposed at the Grizzly design summit. Requirements were discussed with the community and used to form the basis for the project's charter. Implementation began in February 2013, and we have been fortunate to receive regular contributions from Red Hat, Rackspace, IBM, and others since that time.
+
The service features a fully RESTful API, which developers can use to send messages between various components of their SaaS and mobile applications, by using a variety of communication patterns. Underlying this API is an efficient messaging engine designed with scalability and security in mind.  
  
Marconi's overarching goal is to provide web-scale, highly-available messaging to '''web applications that run on OpenStack'''. Marconi runs on Nova servers, behind OpenStack load balancers and uses Keystone authentication middleware. The Marconi implementation makes use of Oslo, and follows the standard OpenStack hacking guidelines.
+
Other OpenStack components can integrate with Zaqar to surface events to end users and to communicate with guest agents that run in the "over-cloud" layer. Cloud operators can leverage Zaqar to provide equivalents of SQS and SNS to their customers.
  
Marconi provides an interface for posting messages, and later claiming those messages for processing. It also provides an interface to clients for listing messages without needing to claim them (ala RSS and Atom), in order to support pub-sub and passive auditing of producer-consumer workflows. The service guarantees first in, first out (FIFO) order for single producer models, best-effort ordering otherwise.
+
==== Key features ====
  
The Marconi architecture is pluggable in terms of both transport and storage. Reference drivers for HTTP (WSGI), SQLAlchemy, and MongoDB will be provided in the initial release (Icehouse), along with a SQLite driver to facilitate development and testing. Other transport and storage drivers have been proposed, and are currently under discussion. Marconi deployments will support HA, and will be able to scale horizontally in the transport and storage layers to enable large deployments. A routing proxy and migration service is also under development, to provide further horizontal scaling across multiple independent Marconi partitions, for use in extremely large deployments.
+
Zaqar provides the following key features:
  
The Marconi v1.0 API defines the following operations:
+
* Choice between two communication transports. Both with Keystone support:
 +
** Firewall-friendly, HTTP-based RESTful API
 +
*** Many of today's developers prefer a more web-friendly HTTP API. They value the simplicity and transparency of the protocol, its firewall-friendly nature, and its huge ecosystem of tools, load balancers and proxies. In addition, cloud operators appreciate the scalability aspects of the REST architectural style.
 +
** Websocket-based API for persistent connections.
 +
*** Websocket protocol provides communication over persistent connections. Unlike HTTP, where new connections are opened for each request/response pair, Websocket can transfer multiple requests/responses over single TCP connection. It saves much network traffic and minimizes delays.
 +
* Multi-tenant queues based on Keystone project IDs
 +
* Support for several common patterns including event broadcasting, task distribution, and point-to-point messaging (See [[Zaqar/Use_Cases|Zaqar Use Cases]])
 +
* Component-based architecture with support for custom backends and message filters
 +
* Efficient reference implementation with an eye toward low latency and high throughput (dependent on backend)
 +
* Highly-available and horizontally scalable (See [[Server Architecture (Marconi)|Server Architecture]])
 +
* Support for subscriptions to queues. Several notification types are available:
 +
** Email notifications
 +
** Webhook notifications
 +
** Websocket notifications
  
'''API''':
+
==== Project scope ====
  
:Get JSON home document
+
The Zaqar API is data-oriented. That is, it does not provision message brokers and expose those directly to clients. Instead, the API acts as a bridge between the client and one or more backends. A provisioning service for message brokers—however useful—serves a somewhat different market from what Zaqar is targeting today. With that in mind, if users are interested in a broker provisioning service, the community should consider starting a new project to address that need.
:Check node health
 
  
'''Queues''':
+
==== Design principles ====
  
:Create a queue
+
Zaqar, as with all OpenStack projects, is designed with the following guidelines in mind:
:List queues
 
:Set queue metadata
 
:Get queue metadata
 
:Get queue stats
 
:Delete a queue
 
  
'''Messages'''
+
* '''Component-based architecture.''' Quickly add new behaviors
:Post one or more messages
+
* '''Highly available and scalable.''' Scale to very serious workloads
:List messages
+
* '''Fault tolerant.''' Isolated processes avoid cascading failures
:Get a message
+
* '''Recoverable.''' Failures should be easy to diagnose, debug, and rectify
:Delete a message
+
* '''Open standards.''' Be a reference implementation for a community-driven
:Get multiple messages
 
:Delete multiple messages
 
  
'''Claims'''
+
==== Status ====
:Claim messages
 
:Query a claim
 
:Update a claim
 
:Release a claim
 
  
=== Core Team ===
+
Zaqar is production-ready for small and medium-sized workloads. The project is in production at Rackspace and other companies are in the process of deploying it in their own clouds. Support for the v2 APIs is available for Python in [https://github.com/openstack/python-zaqarclient python-zaqarclient], and for other languages through [http://developer.rackspace.com/#home-sdks Rackspace-supported SDKs].
  
Kurt Griffiths (kgriffs) is the current PTL. Other core team members include the following individuals who have demonstrated solid '''mentoring skills''' and '''good judgment''' while contributing to the program.
+
== Documentation ==
  
* Flavio Percoco (flaper87)
+
Currently (during Mitaka release cycle) Zaqar team is reworking Zaqar documentation.
* Alejandro Cabrera (cpp-cabrera)
 
* Fei Long Wang (flwang)
 
* [your name could be here!]
 
  
== Design ==
+
==== Zaqar Configuration Reference ====
  
Marconi aims to be pragmatic, building upon the real-world experiences of teams who have solid track records running and supporting web-scale message queueing systems. The project's overarching design philosophy is derived from Donald A. Norman:
+
[http://docs.openstack.org/newton/config-reference/message.html Message service Configuration Reference]
  
<blockquote>The value of a well-designed object is when it has such a rich set of affordances that the people who use it can do things with it that the designer never imagined.</blockquote>
+
==== Zaqar Contributor Guide ====
  
Goals related to the above:
+
[http://docs.openstack.org/developer/zaqar/ Zaqar Contributor Guide]
  
# Emergent functionality, utility
+
==== Zaqar RESTful API Specs ====
# Modular, pluggable code base
 
# REST architectural style
 
  
 +
* [[Zaqar/specs/api/v1|API v1.0]] ('''Released in 2014.1''', '''Deprecated''')
 +
* [[Zaqar/specs/api/v1.1|API v1.1]] ('''Released in 2014.2''', '''Deprecated''')
 +
* [http://developer.openstack.org/api-ref/messaging/ API v2.0] ('''Current API Version''')
  
Principles to live by:
+
==== Other documentation ====
  
# DRY
+
"Getting Started Guide",  "Zaqar RESTful API v1.0 Guide and Reference" are currently not hosted anywhere, but you can generate these documents from Zaqar source code repository. Here's how to do it:
# YAGNI
+
# Ensure you have '''maven''' and '''git''' programs installed on your system.
# KISS
+
# Clone Zaqar repository in the preferred directory on your machine by executing '''git clone <nowiki>https://github.com/openstack/zaqar.git</nowiki>'''
 +
# Inside cloned repository in ''doc/user-guide'' directory execute '''mvn clean generate-sources''' to generate documentation html files from sources.
 +
# You can view documentation by opening generated html files in your browser:
 +
## Getting Started Guide: ''doc/user-guide/zaqar-get-started/target/docbkx/webhelp/zaqar-get-started/index.html''
 +
## Zaqar Configuration Reference: ''doc/user-guide/zaqar-config-ref/target/docbkx/webhelp/zaqar-config-ref/index.html''
 +
## Zaqar API v1.0 Guide and Reference: ''doc/user-guide/zaqar-api-ref/target/docbkx/webhelp/zaqar-api-ref/index.html''
  
== Architecture ==
+
== Resources ==
[[File:Marconi-architecture.png|none]]
 
 
 
== Use Cases ==
 
  
# Distribute tasks among multiple workers (transactional job queues)
+
The Zaqar team hangs out in '''#openstack-zaqar''' on Freenode. Feel free to stop in anytime to chat with us; the crew is always happy to hear your ideas and answer questions. If you run into a bug, please report it using our [https://bugs.launchpad.net/zaqar issue tracker].
# Forward events to data collectors  (transactional event queues)
 
# Publish events to any number of subscribers (pub-sub)
 
# Send commands to one or more agents (point-to-point or pub-sub)
 
# Request an action or get information from an agent (RPC)
 
  
== Out of Scope ==
+
==== FAQ ====
  
Marconi may be used as the foundation for other services to support the following use cases, but will not support them directly within its code base.
+
* [[Zaqar/Frequently asked questions|Frequently asked questions]]
  
# Forwarding notifications to email, SMS, Twitter, etc.
+
==== Releases ====
# Forwarding notifications to web hooks
 
# Forwarding notifications to APNS, GCM, etc.
 
# Scheduling-as-a-service
 
# Metering usage
 
  
== Etherpads ==
+
<!-- TODO(kgriffs): Add release notes -->
 +
* 2014.1 ([http://tarballs.openstack.org/marconi/marconi-2014.1.tar.gz download] | [[Marconi/releases/icehouse|notes]])
  
* [https://etherpad.openstack.org/marconi-client Marconi Client Brainstorm #1]
+
==== Talks ====
* [https://etherpad.openstack.org/apiclient-marconi Common Code Improvements]
 
* [https://etherpad.openstack.org/marconi-client-zmq ZMQ Brainstorm #1]
 
* [https://etherpad.openstack.org/marconi-fifo FIFO Etc.]
 
* [https://etherpad.openstack.org/marconi-test-refactoring Tests Refactoring]
 
* [https://etherpad.openstack.org/marconi-placement Placement Service]
 
* [https://etherpad.openstack.org/marconi-proxy-issues Marconi Proxy Issues]
 
 
 
== Presentations ==
 
  
 
* Alejandro Cabrera. Rackspace Atlanta. Introducing Openstack Marconi. July 17, 2013. [http://www.youtube.com/watch?v=b-IWDG7kKGQ Youtube] [https://speakerdeck.com/cabrera/introducing-openstack-marconi Speaker Deck]
 
* Alejandro Cabrera. Rackspace Atlanta. Introducing Openstack Marconi. July 17, 2013. [http://www.youtube.com/watch?v=b-IWDG7kKGQ Youtube] [https://speakerdeck.com/cabrera/introducing-openstack-marconi Speaker Deck]
 
* Flavio Percoco. EuroPython 2013. Marconi: Queuing and Notification Service for Openstack. July 2, 2013. [http://www.youtube.com/watch?v=iNOd7-_TdDo YouTube]
 
* Flavio Percoco. EuroPython 2013. Marconi: Queuing and Notification Service for Openstack. July 2, 2013. [http://www.youtube.com/watch?v=iNOd7-_TdDo YouTube]
* Kurt Griffiths, Allan Metts. Openstack Summit April 2013. Project Overview: OpenStack Queuing and Notification Service ("Marconi""). April 2013. [http://www.youtube.com/watch?feature=player_embedded&v=henTTZWtt2k YouTube]
+
* Kurt Griffiths, Allan Metts. Openstack Summit April 2013. Project Overview: OpenStack Queuing and Notification Service ("Marconi"). April 2013. [http://www.youtube.com/watch?feature=player_embedded&v=henTTZWtt2k YouTube]
* Kurt Griffiths, Flavio Percoco, Allan Metts. Openstack Summit November 2013. Openstack Queuing and Notification Service Marconi. November 2013. [http://www.youtube.com/watch?v=IgE3Jskqtek YouTube]
+
* Kurt Griffiths, Flavio Percoco, Allan Metts. Openstack Summit November 2013. Openstack Queuing and Notification Service. November 2013. [http://www.youtube.com/watch?v=IgE3Jskqtek YouTube]
 
+
* Yeela Kaplan. EuroPython 2014. OpenStack Queuing and Notification Service ("Marconi"). July 22, 2014. [https://www.youtube.com/watch?v=d65TtqGp-9Q YouTube]
== Articles ==
 
 
 
* Oz Akan. Rackspace Devops Blog. July 25, 2013. [http://developer.rackspace.com/blog/openstack-marconi-api.html Openstack Marconi API].
 
 
 
== FAQ ==
 
  
=== Will Marconi work with AMQP? ===
+
==== Articles ====
  
* Planned as a backend for v2 API
+
* Oz Akan. Rackspace Devops Blog. July 25, 2013. [http://developer.rackspace.com/blog/openstack-marconi-api.html Openstack Zaqar API v1.0].
* Transport TBD
+
* Flavio Percoco. Developer's Black Hole. March 2014. [http://blog.flaper87.com/post/531cd585d987d24e83f082a5 People don't like to queue up].
* Talk to us about use cases
 
* Need contributors
 
  
=== How does Marconi compare to AWS (SQS/SNS)? ===
+
== Community ==
  
* Targets similar workloads
+
* [https://webchat.freenode.net/?channels=openstack-zaqar #openstack-zaqar] (IRC)
* Marconi will provide a unified API to handle notifications and queuing
+
* [[Zaqar/Feedback|Feedback]]
* Marconi is highly customizable
+
* [https://ask.openstack.org/en/questions/ Ask OpenStack (Community Q&A)]
* FIFO and once-and-only-once guaranteed (depending on storage backend)
 
  
=== How mature is the project? ===
+
== Information for Contributors ==
  
* Marconi is used in production: [[Who Uses Marconi? | marconi_users]]
+
[http://docs.openstack.org/developer/zaqar/ Zaqar Contributor Guide]
* Number of contributors is growing
 
  
=== What's next for marconi? ===
+
''Note: If you can't find the information you are looking for, please drop us a line in IRC (#openstack-zaqar) and we'll be happy to help.''
  
* Releasing API v1.1 for Icehouse
+
==== Design ====
* Additional storage drivers
 
* Storage sharding: scaling horizontally and heterogeneous storage
 
* Notifications
 
* Message signing
 
* Additional ops features
 
* Reference client library
 
  
 +
* [[Zaqar/Use_Cases|Zaqar Use Cases]]
 +
* [[Server Architecture (Marconi)|Server Architecture]]
 +
* Reference Clients
 +
** [[Python Client Library (Marconi)|Python Library & CLI]]
 +
** [https://github.com/abettadapur/marconiclient C# Library]
 +
* Summit Notes
 +
** [[Juno/Notes/Marconi|Juno Easel Pad]]
  
See also the [[Marconi/roadmaps/icehouse|Icehouse Roadmap]].
+
==== Quality Engineering ====
  
=== How easy is it to contribute/get up and running? ===
+
* [[Application Security (Marconi)|Application Security]]
 +
* [[Performance Testing (Marconi)|Performance Testing]]
 +
* [[Functional Testing (Marconi)|Functional Testing]]
 +
* [[Integration Testing (Marconi)|Integration Testing]]
 +
<!-- TODO(kgriffs): What about user experience design/testing? -->
  
* You don't need devstack
+
==== Community ====
* You decide what you want to work on: storage, transport, client-side
 
* Very decoupled
 
* Easy: choose a bug and submit a patch
 
  
== Resources ==
+
* [https://webchat.freenode.net/?channels=openstack-zaqar #openstack-zaqar] (IRC)
 +
* [[Meetings/Zaqar|Meetings]]
 +
* [[Zaqar/Feedback|Feedback]]
 +
* [https://ask.openstack.org/en/questions/ Ask OpenStack (Community Q&A)]
  
{| border="1" cellpadding="2" cellspacing="0"
+
==== Administrivia ====
|  Meetings
 
| [[Meetings/Marconi]]
 
|-
 
|  IRC
 
| #openstack-marconi on Freenode
 
|-
 
|  Trello
 
| https://trello.com/b/7NLODgbr (deprecated in favor of Launchpad)
 
|-
 
|  Havana Spec 
 
| [[Marconi/specs/havana]]
 
|-
 
|  API Blueprint 
 
| [[Marconi/specs/api/v1]]<br/>
 
[[Marconi/specs/zmq/api/v1]]<br/>
 
[[Marconi/specs/sharding/v1|Sharding]]
 
|-
 
|  Other Blueprints   
 
| https://blueprints.launchpad.net/marconi
 
|-
 
|  Milestones 
 
| https://launchpad.net/marconi/+milestones
 
|-
 
|  Developer Docs 
 
| [[Marconi/DevTips|Tips for Developers]]<br/>[[Marconi/PythonClient|Python Client Bindings]]<br>[[Marconi/Drivers|Drivers]]<br>[[Marconi/guarantees|Guarantees]]
 
|-
 
|  Incubation 
 
| [[Marconi/Incubation]]
 
|-
 
| Source code 
 
| https://github.com/openstack/marconi
 
|-
 
| Client source code
 
| https://github.com/openstack/python-marconiclient
 
|-
 
| Code Reviews
 
| https://review.openstack.org/#/q/status:open+project:stackforge/marconi,n,z
 
|-
 
|  Bug tracker 
 
| https://bugs.launchpad.net/marconi
 
|}
 
  
----
+
* [[Roadmap (Marconi)|Roadmap (Juno)]]
[[Category:Homepage]]
+
* [https://etherpad.openstack.org/p/naav-graduation Incubation Status]
 +
* [[How to Release (Marconi)|How to Release]]
 +
* [[Security/Juno/Marconi|OSSG Q&A for Zaqar (Juno)]]

Latest revision as of 02:17, 7 November 2016

Projects

Zaqar

Official name OpenStack Messaging
Source code https://github.com/openstack/zaqar
Bug tracker https://bugs.launchpad.net/zaqar
Feature tracker https://blueprints.launchpad.net/zaqar
Contributor documentation http://docs.openstack.org/developer/zaqar/

Python Zaqar Client

Source code https://github.com/openstack/python-zaqarclient
Bug tracker https://bugs.launchpad.net/python-zaqarclient
Feature tracker https://blueprints.launchpad.net/python-zaqarclient

Overview

Zaqar is a multi-tenant cloud messaging service for web and mobile developers. It combines the ideas pioneered by Amazon's SQS product with additional semantics to support event broadcasting.

The service features a fully RESTful API, which developers can use to send messages between various components of their SaaS and mobile applications, by using a variety of communication patterns. Underlying this API is an efficient messaging engine designed with scalability and security in mind.

Other OpenStack components can integrate with Zaqar to surface events to end users and to communicate with guest agents that run in the "over-cloud" layer. Cloud operators can leverage Zaqar to provide equivalents of SQS and SNS to their customers.

Key features

Zaqar provides the following key features:

  • Choice between two communication transports. Both with Keystone support:
    • Firewall-friendly, HTTP-based RESTful API
      • Many of today's developers prefer a more web-friendly HTTP API. They value the simplicity and transparency of the protocol, its firewall-friendly nature, and its huge ecosystem of tools, load balancers and proxies. In addition, cloud operators appreciate the scalability aspects of the REST architectural style.
    • Websocket-based API for persistent connections.
      • Websocket protocol provides communication over persistent connections. Unlike HTTP, where new connections are opened for each request/response pair, Websocket can transfer multiple requests/responses over single TCP connection. It saves much network traffic and minimizes delays.
  • Multi-tenant queues based on Keystone project IDs
  • Support for several common patterns including event broadcasting, task distribution, and point-to-point messaging (See Zaqar Use Cases)
  • Component-based architecture with support for custom backends and message filters
  • Efficient reference implementation with an eye toward low latency and high throughput (dependent on backend)
  • Highly-available and horizontally scalable (See Server Architecture)
  • Support for subscriptions to queues. Several notification types are available:
    • Email notifications
    • Webhook notifications
    • Websocket notifications

Project scope

The Zaqar API is data-oriented. That is, it does not provision message brokers and expose those directly to clients. Instead, the API acts as a bridge between the client and one or more backends. A provisioning service for message brokers—however useful—serves a somewhat different market from what Zaqar is targeting today. With that in mind, if users are interested in a broker provisioning service, the community should consider starting a new project to address that need.

Design principles

Zaqar, as with all OpenStack projects, is designed with the following guidelines in mind:

  • Component-based architecture. Quickly add new behaviors
  • Highly available and scalable. Scale to very serious workloads
  • Fault tolerant. Isolated processes avoid cascading failures
  • Recoverable. Failures should be easy to diagnose, debug, and rectify
  • Open standards. Be a reference implementation for a community-driven

Status

Zaqar is production-ready for small and medium-sized workloads. The project is in production at Rackspace and other companies are in the process of deploying it in their own clouds. Support for the v2 APIs is available for Python in python-zaqarclient, and for other languages through Rackspace-supported SDKs.

Documentation

Currently (during Mitaka release cycle) Zaqar team is reworking Zaqar documentation.

Zaqar Configuration Reference

Message service Configuration Reference

Zaqar Contributor Guide

Zaqar Contributor Guide

Zaqar RESTful API Specs

Other documentation

"Getting Started Guide", "Zaqar RESTful API v1.0 Guide and Reference" are currently not hosted anywhere, but you can generate these documents from Zaqar source code repository. Here's how to do it:

  1. Ensure you have maven and git programs installed on your system.
  2. Clone Zaqar repository in the preferred directory on your machine by executing git clone https://github.com/openstack/zaqar.git
  3. Inside cloned repository in doc/user-guide directory execute mvn clean generate-sources to generate documentation html files from sources.
  4. You can view documentation by opening generated html files in your browser:
    1. Getting Started Guide: doc/user-guide/zaqar-get-started/target/docbkx/webhelp/zaqar-get-started/index.html
    2. Zaqar Configuration Reference: doc/user-guide/zaqar-config-ref/target/docbkx/webhelp/zaqar-config-ref/index.html
    3. Zaqar API v1.0 Guide and Reference: doc/user-guide/zaqar-api-ref/target/docbkx/webhelp/zaqar-api-ref/index.html

Resources

The Zaqar team hangs out in #openstack-zaqar on Freenode. Feel free to stop in anytime to chat with us; the crew is always happy to hear your ideas and answer questions. If you run into a bug, please report it using our issue tracker.

FAQ

Releases

Talks

  • Alejandro Cabrera. Rackspace Atlanta. Introducing Openstack Marconi. July 17, 2013. Youtube Speaker Deck
  • Flavio Percoco. EuroPython 2013. Marconi: Queuing and Notification Service for Openstack. July 2, 2013. YouTube
  • Kurt Griffiths, Allan Metts. Openstack Summit April 2013. Project Overview: OpenStack Queuing and Notification Service ("Marconi"). April 2013. YouTube
  • Kurt Griffiths, Flavio Percoco, Allan Metts. Openstack Summit November 2013. Openstack Queuing and Notification Service. November 2013. YouTube
  • Yeela Kaplan. EuroPython 2014. OpenStack Queuing and Notification Service ("Marconi"). July 22, 2014. YouTube

Articles

Community

Information for Contributors

Zaqar Contributor Guide

Note: If you can't find the information you are looking for, please drop us a line in IRC (#openstack-zaqar) and we'll be happy to help.

Design

Quality Engineering

Community

Administrivia