Jump to: navigation, search

Difference between revisions of "Zaqar/FAQ"

(First draft. Pretty rough still!)
(Undo revision 52034 by Kgriffs (talk))
Line 1: Line 1:
−
== Is Marconi a provisioning service, or a data API? ==
+
== Is Marconi a Provisioning or a data API? ==
  
−
Marconi's API is data-oriented. That is, it does not provision message brokers. Instead, it acts as a bridge between the client and one or more backends. In fact, Marconi's API *is* the product; it provides common messaging semantics on top of existing messaging and storage systems. Marconi's data API aims to be simpler, more flexible and more oriented to the needs of web and mobile application developers than the underlying API.
+
Marconi's API is data oriented. That is, it does not provision messaging services, instead, it acts as a bridge between the client and the messaging system. In fact, Marconi's API *is* the product, it provides common messaging semantics on top of the existing messaging systems. Marconi's data API aims to be simpler, more flexible and more oriented to the user needs than the underlying API. Regardless of the messaging system in use, Marconi will provide a web-friendly API that is service oriented - with support for multiple projects -, fully integrated with other openstack modules - but also capable of running as a stand-alone service - and as lightweight, pragmatic and useful as possible.
 +
More specifically, Marconi's API tries to solve the following issues (please, note this is not a exahustive list):
  
−
Regardless of the backend in use, Marconi provides a web-friendly API that is service-oriented and multi-tenant, and is as lightweight, pragmatic and useful as possible. Marconi can run standalone, but works best when leveraging other OpenStack programs such as Keystone and Barbican. In turn, other programs can use Marconi to surface events to end users.
+
* First-class, idiomatic HTTP transport that is familiar to web developers and works  well across poor connections, such as those found on mobile devices.
 +
* Scales  to hundreds of thousands or even millions of message topics AKA queues,  allowing web developers to communicate with an "internet of things".
 +
* Simple, light JSON request payloads that are easy to grok, efficient to transmit, and quick to parse
 +
* Architecture that allows for future transports such as WebSockets, tcp etc.
 +
* Architecture that allows for multiple messaging technologies to run alongside each other under the same API.
 +
* Easy to scale messaging service. (Refer to Marconi's sharding capabilities)
  
−
Marconi's API provides:
+
A provision API, although useful, belongs to a complete different project and aims to provide a different kind of service from what Marconi is today.  This is currently out of scope of the Marconi program, but if users are interested in a queue provisioning service the community should consider starting a new project to address that need.
  
−
* A first-class, idiomatic HTTP transport that is familiar to web developers and works well across firewalls and poor network connections.
+
== What is the SQLAlchemy driver's purpose? ==
−
* A design that scales to hundreds of thousands or even millions of message topics AKA queues, allowing web developers to communicate with an "Internet of things".
 
−
* A JSON-based message schema that is easy to understand, efficient to transmit, and quick to parse
 
−
* An architecture that allows for future transports such as WebSockets, tcp etc.
 
−
* An architecture that allows multiple messaging and storage backends to run side-by-side under the same API.
 
−
* A (relatively) easy-to-scale, HA, multi-tenant messaging service
 
  
 +
'''NOTE: This section is for "old"SQL databases not for NewSQL ( http://en.wikipedia.org/wiki/NewSQL )'''
  
−
A provisioning service for message brokers, however useful, serves a somewhat different market from what Marconi is targeting today. If users are interested in a queue provisioning service, the community should consider starting a new program to address that need.
+
Neither the project or the team consider using SQL databases as a good store for queuing systems. However, the presence of a `sqlalchemy` storage driver can be somehow misleading in this sense. The sqlalchemy driver was created for 3 main reasons:
  
−
== What is the purpose of the SQLAlchemy driver? ==
+
* The support for sqlite is very useful for development. It makes it really simple to spawn up a marconi server in 3 steps and run tests for it.
 +
* The sqlalchemy driver brings enough support for marconi to run in OpenStack's gates.
 +
* Marconi is a non-opinionated project, hence the team wanted to provide 1 driver for each of the most common families of stores / technologies out there (nosql, in-memory [also considered NoSQL], SQL, AMQP).
  
−
It is well known that a classic RDBMS, such as MySQL, performs poorly at scale as a messaging backend. Therefore, bundling a SQLAlchemy driver with Marconi may seem like an odd thing to do. Nevertheless, the driver does have its uses:
+
== Why is it an anti-pattern to use an SQL database for messaging? ==
  
−
* It provides support for SQLite, which is very useful for development. Among other benefits, supporting SQLite means you can go from zero to a working local Marconi server in just 3 steps.
+
There are many references about this all over the internet. Here's a short list:
−
* It provides a reference implementation of the storage driver interface. The same functional tests that pass when running against the SQLAlchemy driver must also pass for all other drivers, helping ensure consistent behavior.
 
−
* It helps the team be more pragmatic in the design of Marconi's API and in the service's architecture. The SQLAlchemy driver is part of a broader effort to create drivers to represent each of the most common data store families, including NoSQL, SQL, and AMQP.
 
  
−
<!-- == Why is it an anti-pattern to use a SQL database for messaging? ==
+
* http://mikehadlow.blogspot.it/2012/04/database-as-queue-anti-pattern.html
  
−
Many RDBMSes simply aren't optimized to cope with the constant churn of a large number of ephemeral records. If you'd like to learn more, a few online searches should bring some the writings on the topic.
+
== Why isn't it an anti-pattern to use NoSQL databases for messaging? ==
−
-->
 
−
== Why did you start with MongoDB? ==
 
  
−
Some NoSQL databases work better for implementing messaging systems than do others. For example, Redis and MongoDB are known to work well in this role, while Cassandra performs rather poorly.
+
Not all NoSQL databases are good for messaging systems. Although they all share a set of base features, they're implemented using different algorithms, technologies and different. higher-level, set of features.  MongoDB has demostrated over the time to be a good database to handle simple  messaging systems. Without adding lot of complexity to what it does,  it's well capable of supporting all Marconi's API requirements.  Mongodb's driver has been implemented in a way that almost all cases are  covered by the index. However, we're well aware that this doesn't make  it the *perfect* driver for every use-case and that the database itself  makes parts of the implementation harder and sometimes even slower.
 +
MongoDB is not the only NoSQL database that is good for this job. However, there are a couple of important motivations that made the team choose it instead of other databases:
  
−
The Marconi team decided to start with MongoDB because:
+
* Core team was most familiar with it (practical reasons)
 +
* Lots of operators/developers are already familiar with MongoDB and want to use it.
 +
* Schemaless, etc. - good for rapid iteration in the early days.
  
−
* MongoDB's flexible query system was able to support all of the operations required by the 1.0 API
+
With all that said, it's important to mention that despite the fact MongoDB was the first production-ready storage driver implemented for Marconi, it's not the team's intention to make it the one-and-only driver for all use cases. The team works hard to keep Marconi up-to-date with existing technologies and more importantly, cover the user needs.
−
* It was able to manage a large number of short-lived records without hitting a wall
 
−
* The system offered a well-rounded mixture of durability, performance, HA, and scalability
 
−
* The core team was already very familiar with MongoDB and had experience running it in production
 
−
* MongoDB's schemaless design allowed the team to iterate rapidly
 
  
 +
== What Message Exchange Patterns (MEP) does Marconi support? ==
  
−
With this in mind, it's important to mention that despite the fact MongoDB was the first production-ready storage driver implemented for Marconi, it's not the team's intention to make it the one-and-only driver for all use cases. In fact, more drivers are already in the works. Stay tuned!
+
- [http://en.wikipedia.org/wiki/Producer%E2%80%93consumer_problem Producer and Consumer]
−
 
 
−
== What messaging patterns does Marconi support? ==
 
−
 
 
−
Marconi's API provides a basic set of semantics that, when combined, afford a variety of messaging patterns, such as pub-sub, task distribution, and point-to-point. Messages can be consumed as feeds, queues, or a combination of the two. This can be a little confusing at first, since the API uses the term "queues" to represent a hybrid feeds-queues resource.
 
−
 
 
−
When interacting with the API, a client can choose to read a queue in a similar manner to an Atom feed, where any client can read any message, and is responsible for keeping track of its own marker (its position in the feed). This provides for messaging patterns such as pub-sub and point-to-point.
 
−
 
 
−
Alternatively, an application can implement task distribution by creating a pool of workers that simply claim messages off the front of the queue. Once a message is claimed, it becomes invisible to other workers in the pool to prevent messages from being processed more than once.
 
−
 
 
−
Finally, a client can use both feed and claim semantics simultaneously to create hybrid messaging patterns. For example, while workers are busy claiming messages from a task queue, an auditor can passively sample those same messages as they flow by.
 
−
 
 
−
== Will Marconi work with AMQP? ==
 
−
 
 
−
* Planned as a backend for v2 API
 
−
* Transport TBD
 
−
* Talk to us about use cases
 
−
* Need contributors
 
−
 
 
−
== How does Marconi compare to AWS (SQS/SNS)? ==
 
−
 
 
−
* Targets similar workloads
 
−
* Marconi will provide a unified API to handle notifications and queuing
 
−
* Marconi is highly customizable
 
−
* FIFO and once-and-only-once guaranteed (depending on storage backend)
 
−
 
 
−
== How does Marconi compare to oslo.messaging? ==
 
−
 
 
−
[[Oslo/Messaging | oslo.messsaging]] is a client library that provides a standard mechanism for various OpenStack services to communicate with each other. As such, the library is single-tenant (the single tenant being the cloud operator). Oslo Messaging was originally developed as an abstraction over AMQP, but has since added support for ZeroMQ.
 
−
 
 
−
By contrast, Marconi is a hosted, multi-tenant cloud messaging and notifications service. It was created to serve the needs of application developers building on top of OpenStack, and to provide a way for other OpenStack services to surface events to end users.
 
−
 
 
−
== How mature is the project? ==
 
−
 
 
−
Marconi is used in production today: [[Who Uses Marconi? | marconi_users]]
 
−
 
 
−
* Marconi's first official, production-ready "1.0" release is now [http://tarballs.openstack.org/marconi/ available for download]. This first release includes a battle-tested MongoDB driver, and production-ready drivers for additional backends are in the works.
 
−
* Marconi's v1.0 API is stable and ready to code against.
 
−
* Basic user and operator docs are [http://docs.openstack.org/developer/marconi/ now available], and we will be adding tons of new content during Juno.
 
−
* A reference client library (written in Python) is now [https://pypi.python.org/pypi/python-marconiclient/ available on PyPI] which supports the entire v1.0 API. Support for other languages is available through [http://developer.rackspace.com/#home-sdks Rackspace-supported SDKs].
 
−
 
 
−
 
 
−
Latest team stats:
 
−
 
 
−
* 10+ organizations represented
 
−
* 5 core reviewers
 
−
* 6 interns (representing GSoC, GNOME OPW, Rackspace and Red Hat)
 
−
 
 
−
== What's next for marconi? ==
 
−
 
 
−
* API v1.1
 
−
* Additional backend drivers
 
−
* Queue Flavors
 
−
* Notifications
 
−
* Message signing
 
−
* Automated performance testing
 
−
* Automated security testing
 
−
* Additional Dev/Ops features
 
−
* Expanded documentation
 

Revision as of 06:51, 11 May 2014

Is Marconi a Provisioning or a data API?

Marconi's API is data oriented. That is, it does not provision messaging services, instead, it acts as a bridge between the client and the messaging system. In fact, Marconi's API *is* the product, it provides common messaging semantics on top of the existing messaging systems. Marconi's data API aims to be simpler, more flexible and more oriented to the user needs than the underlying API. Regardless of the messaging system in use, Marconi will provide a web-friendly API that is service oriented - with support for multiple projects -, fully integrated with other openstack modules - but also capable of running as a stand-alone service - and as lightweight, pragmatic and useful as possible. More specifically, Marconi's API tries to solve the following issues (please, note this is not a exahustive list):

  • First-class, idiomatic HTTP transport that is familiar to web developers and works well across poor connections, such as those found on mobile devices.
  • Scales to hundreds of thousands or even millions of message topics AKA queues, allowing web developers to communicate with an "internet of things".
  • Simple, light JSON request payloads that are easy to grok, efficient to transmit, and quick to parse
  • Architecture that allows for future transports such as WebSockets, tcp etc.
  • Architecture that allows for multiple messaging technologies to run alongside each other under the same API.
  • Easy to scale messaging service. (Refer to Marconi's sharding capabilities)

A provision API, although useful, belongs to a complete different project and aims to provide a different kind of service from what Marconi is today. This is currently out of scope of the Marconi program, but if users are interested in a queue provisioning service the community should consider starting a new project to address that need.

What is the SQLAlchemy driver's purpose?

NOTE: This section is for "old"SQL databases not for NewSQL ( http://en.wikipedia.org/wiki/NewSQL )

Neither the project or the team consider using SQL databases as a good store for queuing systems. However, the presence of a `sqlalchemy` storage driver can be somehow misleading in this sense. The sqlalchemy driver was created for 3 main reasons:

  • The support for sqlite is very useful for development. It makes it really simple to spawn up a marconi server in 3 steps and run tests for it.
  • The sqlalchemy driver brings enough support for marconi to run in OpenStack's gates.
  • Marconi is a non-opinionated project, hence the team wanted to provide 1 driver for each of the most common families of stores / technologies out there (nosql, in-memory [also considered NoSQL], SQL, AMQP).

Why is it an anti-pattern to use an SQL database for messaging?

There are many references about this all over the internet. Here's a short list:

Why isn't it an anti-pattern to use NoSQL databases for messaging?

Not all NoSQL databases are good for messaging systems. Although they all share a set of base features, they're implemented using different algorithms, technologies and different. higher-level, set of features. MongoDB has demostrated over the time to be a good database to handle simple messaging systems. Without adding lot of complexity to what it does, it's well capable of supporting all Marconi's API requirements. Mongodb's driver has been implemented in a way that almost all cases are covered by the index. However, we're well aware that this doesn't make it the *perfect* driver for every use-case and that the database itself makes parts of the implementation harder and sometimes even slower. MongoDB is not the only NoSQL database that is good for this job. However, there are a couple of important motivations that made the team choose it instead of other databases:

  • Core team was most familiar with it (practical reasons)
  • Lots of operators/developers are already familiar with MongoDB and want to use it.
  • Schemaless, etc. - good for rapid iteration in the early days.

With all that said, it's important to mention that despite the fact MongoDB was the first production-ready storage driver implemented for Marconi, it's not the team's intention to make it the one-and-only driver for all use cases. The team works hard to keep Marconi up-to-date with existing technologies and more importantly, cover the user needs.

What Message Exchange Patterns (MEP) does Marconi support?

- Producer and Consumer

Retrieved from "https://wiki.openstack.org/w/index.php?title=Zaqar/FAQ&oldid=52036"