Is Marconi a Provisioning or a data API?
Marconi's API is data oriented. That is, it does not provision messaging services, instead, it acts as a bridge between the client and the messaging system. In fact, Marconi's API *is* the product, it provides common messaging semantics on top of the existing messaging systems. Marconi's data API aims to be simpler, more flexible and more oriented to the user needs than the underlying API. Regardless of the messaging system in use, Marconi will provide a web-friendly API that is service oriented - with support for multiple projects -, fully integrated with other openstack modules - but also capable of running as a stand-alone service - and as lightweight, pragmatic and useful as possible. More specifically, Marconi's API tries to solve the following issues (please, note this is not a exahustive list):
- First-class, idiomatic HTTP transport that is familiar to web developers and works well across poor connections, such as those found on mobile devices.
- Scales to hundreds of thousands or even millions of message topics AKA queues, allowing web developers to communicate with an "internet of things".
- Simple, light JSON request payloads that are easy to grok, efficient to transmit, and quick to parse
- Architecture that allows for future transports such as WebSockets, tcp etc.
- Architecture that allows for multiple messaging technologies to run alongside each other under the same API.
- Easy to scale messaging service. (Refer to Marconi's sharding capabilities)
A provision API, although useful, belongs to a complete different project and aims to provide a different kind of service from what Marconi is today. This is currently out of scope of the Marconi program, but if users are interested in a queue provisioning service the community should consider starting a new project to address that need.
What is the SQLAlchemy driver's purpose?
NOTE: This section is for "old"SQL databases not for NewSQL ( http://en.wikipedia.org/wiki/NewSQL )
Neither the project or the team consider using SQL databases as a good store for queuing systems. However, the presence of a `sqlalchemy` storage driver can be somehow misleading in this sense. The sqlalchemy driver was created for 3 main reasons:
- The support for sqlite is very useful for development. It makes it really simple to spawn up a marconi server in 3 steps and run tests for it.
- The sqlalchemy driver brings enough support for marconi to run in OpenStack's gates.
- Marconi is a non-opinionated project, hence the team wanted to provide 1 driver for each of the most common families of stores / technologies out there (nosql, in-memory [also considered NoSQL], SQL, AMQP).
Why is it an anti-pattern to use an SQL database for messaging?
There are many references about this all over the internet. Here's a short list:
Why isn't it an anti-pattern to use NoSQL databases for messaging?
Not all NoSQL databases are good for messaging systems. Although they all share a set of base features, they're implemented using different algorithms, technologies and different. higher-level, set of features. MongoDB has demostrated over the time to be a good database to handle simple messaging systems. Without adding lot of complexity to what it does, it's well capable of supporting all Marconi's API requirements. Mongodb's driver has been implemented in a way that almost all cases are covered by the index. However, we're well aware that this doesn't make it the *perfect* driver for every use-case and that the database itself makes parts of the implementation harder and sometimes even slower. MongoDB is not the only NoSQL database that is good for this job. However, there are a couple of important motivations that made the team choose it instead of other databases:
- Core team was most familiar with it (practical reasons)
- Lots of operators/developers are already familiar with MongoDB and want to use it.
- Schemaless, etc. - good for rapid iteration in the early days.
With all that said, it's important to mention that despite the fact MongoDB was the first production-ready storage driver implemented for Marconi, it's not the team's intention to make it the one-and-only driver for all use cases. The team works hard to keep Marconi up-to-date with existing technologies and more importantly, cover the user needs.