Difference between revisions of "Frequently asked questions (Zaqar)"

Revision as of 06:52, 11 May 2014

Is Marconi a provisioning service, or a data API?

Marconi's API is data-oriented. That is, it does not provision message brokers. Instead, it acts as a bridge between the client and one or more backends. In fact, Marconi's API *is* the product; it provides common messaging semantics on top of existing messaging and storage systems. Marconi's data API aims to be simpler, more flexible and more oriented to the needs of web and mobile application developers than the underlying API.

Regardless of the backend in use, Marconi provides a web-friendly API that is service-oriented and multi-tenant, and is as lightweight, pragmatic and useful as possible. Marconi can run standalone, but works best when leveraging other OpenStack programs such as Keystone and Barbican. In turn, other programs can use Marconi to surface events to end users.

Marconi's API provides:

• A first-class, idiomatic HTTP transport that is familiar to web developers and works well across firewalls and poor network connections.
• A design that scales to hundreds of thousands or even millions of message topics AKA queues, allowing web developers to communicate with an "Internet of things".
• A JSON-based message schema that is easy to understand, efficient to transmit, and quick to parse
• An architecture that allows for future transports such as WebSockets, tcp etc.
• An architecture that allows multiple messaging and storage backends to run side-by-side under the same API.
• A (relatively) easy-to-scale, HA, multi-tenant messaging service

A provisioning service for message brokers, however useful, serves a somewhat different market from what Marconi is targeting today. If users are interested in a queue provisioning service, the community should consider starting a new program to address that need.

What is the purpose of the SQLAlchemy driver?

It is well known that a classic RDBMS, such as MySQL, performs poorly at scale as a messaging backend. Therefore, bundling a SQLAlchemy driver with Marconi may seem like an odd thing to do. Nevertheless, the driver does have its uses:

• It provides support for SQLite, which is very useful for development. Among other benefits, supporting SQLite means you can go from zero to a working local Marconi server in just 3 steps.
• It provides a reference implementation of the storage driver interface. The same functional tests that pass when running against the SQLAlchemy driver must also pass for all other drivers, helping ensure consistent behavior.
• It helps the team be more pragmatic in the design of Marconi's API and in the service's architecture. The SQLAlchemy driver is part of a broader effort to create drivers to represent each of the most common data store families, including NoSQL, SQL, and AMQP.

Why did you start with MongoDB?

Some NoSQL databases work better for implementing messaging systems than do others. For example, Redis and MongoDB are known to work well in this role, while Cassandra performs rather poorly.

The Marconi team decided to start with MongoDB because:

• MongoDB's flexible query system was able to support all of the operations required by the 1.0 API
• It was able to manage a large number of short-lived records without hitting a wall
• The system offered a well-rounded mixture of durability, performance, HA, and scalability
• The core team was already very familiar with MongoDB and had experience running it in production
• MongoDB's schemaless design allowed the team to iterate rapidly

With this in mind, it's important to mention that despite the fact MongoDB was the first production-ready storage driver implemented for Marconi, it's not the team's intention to make it the one-and-only driver for all use cases. In fact, more drivers are already in the works. Stay tuned!

What messaging patterns does Marconi support?

Marconi's API provides a basic set of semantics that, when combined, afford a variety of messaging patterns, such as pub-sub, task distribution, and point-to-point. Messages can be consumed as feeds, queues, or a combination of the two. This can be a little confusing at first, since the API uses the term "queues" to represent a hybrid feeds-queues resource.

When interacting with the API, a client can choose to read a queue in a similar manner to an Atom feed, where any client can read any message, and is responsible for keeping track of its own marker (its position in the feed). This provides for messaging patterns such as pub-sub and point-to-point.

Alternatively, an application can implement task distribution by creating a pool of workers that simply claim messages off the front of the queue. Once a message is claimed, it becomes invisible to other workers in the pool to prevent messages from being processed more than once.

Finally, a client can use both feed and claim semantics simultaneously to create hybrid messaging patterns. For example, while workers are busy claiming messages from a task queue, an auditor can passively sample those same messages as they flow by.

Will Marconi work with AMQP?

• Planned as a backend for v2 API
• Transport TBD
• Talk to us about use cases
• Need contributors

How does Marconi compare to AWS (SQS/SNS)?

• Targets similar workloads
• Marconi will provide a unified API to handle notifications and queuing
• Marconi is highly customizable
• FIFO and once-and-only-once guaranteed (depending on storage backend)

How does Marconi compare to oslo.messaging?

oslo.messsaging is a client library that provides a standard mechanism for various OpenStack services to communicate with each other. As such, the library is single-tenant (the single tenant being the cloud operator). Oslo Messaging was originally developed as an abstraction over AMQP, but has since added support for ZeroMQ.

By contrast, Marconi is a hosted, multi-tenant cloud messaging and notifications service. It was created to serve the needs of application developers building on top of OpenStack, and to provide a way for other OpenStack services to surface events to end users.

How mature is the project?

Marconi is used in production today: marconi_users

• Marconi's first official, production-ready "1.0" release is now available for download. This first release includes a battle-tested MongoDB driver, and production-ready drivers for additional backends are in the works.
• Marconi's v1.0 API is stable and ready to code against.
• Basic user and operator docs are now available, and we will be adding tons of new content during Juno.
• A reference client library (written in Python) is now available on PyPI which supports the entire v1.0 API. Support for other languages is available through Rackspace-supported SDKs.

Latest team stats:

• 10+ organizations represented
• 5 core reviewers
• 6 interns (representing GSoC, GNOME OPW, Rackspace and Red Hat)

What's next for marconi?

• API v1.1
• Additional backend drivers
• Queue Flavors
• Notifications
• Message signing
• Automated performance testing
• Automated security testing
• Additional Dev/Ops features
• Expanded documentation