Zaqar
Contents
Overview
Zaqar is a multi-tenant cloud messaging service for web developers. It combines the ideas pioneered by Amazon's SQS product with additional semantics to support event broadcasting.
The service features a fully RESTful API, which developers can use to send messages between various components of their SaaS and mobile applications, by using a variety of communication patterns. Underlying this API is an efficient messaging engine designed with scalability and security in mind.
Other OpenStack components can integrate with Zaqar to surface events to end users and to communicate with guest agents that run in the "over-cloud" layer. Cloud operators can leverage Zaqar to provide equivalents of SQS and SNS to their customers.
Key features
Zaqar provides the following key features:
- Firewall-friendly, HTTP-based API with Keystone support
- Multi-tenant queues based on Keystone project IDs
- Support for several common patterns including event broadcasting, task distribution, and point-to-point messaging
- Component-based architecture with support for custom backends and message filters
- Efficient reference implementation with an eye toward low latency and high throughput (dependent on backend)
- Highly-available and horizontally scalable
Project scope
The Zaqar API is data-oriented. That is, it does not provision message brokers and expose those directly to clients. Instead, the API acts as a bridge between the client and one or more backends. A provisioning service for message brokers—however useful—serves a somewhat different market from what Zaqar is targeting today. With that in mind, if users are interested in a broker provisioning service, the community should consider starting a new project to address that need.
Design principles
Zaqar, as with all OpenStack projects, is designed with the following guidelines in mind:
- Component-based architecture. Quickly add new behaviors
- Highly available and scalable. Scale to very serious workloads
- Fault tolerant. Isolated processes avoid cascading failures
- Recoverable. Failures should be easy to diagnose, debug, and rectify
- Open standards. Be a reference implementation for a community-driven
The API
At first glance, Zaqar's RESTful API may seem a bit out of the ordinary. Message brokers typically rely on a custom binary protocol and long-lived connections. However, many of today's developers prefer a more web-friendly HTTP API. They value the simplicity and transparency of the protocol, its firewall-friendly nature, and its huge ecosystem of tools, load balancers and proxies. In addition, cloud operators appreciate the scalability aspects of the REST architectural style.
Version 1.0 of the API has been stable for some time. We are working on a 1.1 version with minor changes to remove some of the rough edges in 1.0, and we welcome community feedback. Note that some proposals by the community, such as removing random access to individual messages by ID, will be taken care of in version 2.0 when we will have more freedom to make major changes.
Status
With the close of the Icehouse cycle, the program has achieved a number of important milestones:
- Zaqar's first official, production-ready "1.0" release is now available for download. This first release includes a battle-tested MongoDB driver, and production-ready drivers for additional backends are in the works.
- Zaqar's v1.0 API is stable and ready to code against.
- Basic user and operator docs are now available, and we will be adding tons of new content during Juno.
- A reference client library (written in Python) is now available on PyPI which supports the entire v1.0 API. Support for other languages is available through Rackspace-supported SDKs.
The program has attracted a growing community of contributors from all over the world. We are particularly proud of our interns:
- 10+ organizations represented
- 5 core reviewers
- 6 interns for Juno (representing GSoC, GNOME OPW, Rackspace and Red Hat)
Resources
The Zaqar team hangs out in #openstack-zaqar on Freenode. Feel free to stop in anytime to chat with us; the crew is always happy to hear your ideas and answer questions. If you run into a bug, please report it using our issue tracker.
Docs
Releases
Talks
- Alejandro Cabrera. Rackspace Atlanta. Introducing Openstack Marconi. July 17, 2013. Youtube Speaker Deck
- Flavio Percoco. EuroPython 2013. Marconi: Queuing and Notification Service for Openstack. July 2, 2013. YouTube
- Kurt Griffiths, Allan Metts. Openstack Summit April 2013. Project Overview: OpenStack Queuing and Notification Service ("Marconi"). April 2013. YouTube
- Kurt Griffiths, Flavio Percoco, Allan Metts. Openstack Summit November 2013. Openstack Queuing and Notification Service. November 2013. YouTube
- Yeela Kaplan. EuroPython 2014. OpenStack Queuing and Notification Service ("Marconi"). July 22, 2014. YouTube
Articles
- Oz Akan. Rackspace Devops Blog. July 25, 2013. Openstack Marconi API.
- Flavio Percoco. Developer's Black Hole. March 2014. People don't like to queue up.
Contributor Guide
Note: The Marconi team is reworking this section of the wiki with tons of updated content, so some pages are missing or incomplete at the moment. If you can't find the information you are looking for, please drop us a line in IRC (#openstack-zaqar) and we'll be happy to help.
Getting Started
- Welcome new contributors
- Give Zaqar a try
- Your first patch
- Your first review
- Write the tests!
- Write the docs!
API Specs
Design
- Zaqar Use Cases
- Server Architecture
- Reference Clients
- Summit Notes
Quality Engineering
Resources
- About the program
- Frequently asked questions
- Development process
- Blueprinting process
- Programming style guide
- Reviewer guide
- Getting help