Jump to: navigation, search

Difference between revisions of "Zaqar/Incubation-Old"

m (Current Code Contributors)
m (Malini moved page Marconi/Incubation-Old to Zaqar/Incubation-Old: Project Rename)
 
(47 intermediate revisions by 4 users not shown)
Line 2: Line 2:
  
 
Marconi
 
Marconi
 +
==Mission==
 +
 +
To produce an OpenStack message queueing API and service that affords a variety of distributed application messaging patterns in an efficient, scalable and highly-available manner, and to create and maintain associated Python libraries and documentation.
 +
 
==Summary==
 
==Summary==
  
Marconi is a messaging and notifications queue for the OpenStack product portfolio, supporting both producer-consumer and publish-subscribe modes.  Marconi is designed to perform and scale in a multi-tenant environment.  
+
Marconi is a messaging and notifications service for the OpenStack product portfolio, supporting both producer-consumer and publish-subscribe modes.  Marconi is designed to perform and scale in a multi-tenant environment.
 +
 
 
==Detailed Description==
 
==Detailed Description==
  
As the OpenStack product line grows, the need for a messaging and notification service is growing in order to support more complex, distributed application architectures. Marconi is designed to run on Nova servers, behind OpenStack load balancers, and in coordination with Keystone auth.  
+
In order to support more complex web applications running on OpenStack, a messaging service is needed. To fill this need, the Marconi project was proposed at the Grizzly design summit. Requirements were discussed with the community and used to form the basis for the project's charter. Implementation began in February 2013, and we have been fortunate to receive regular, major contributions from Red Hat and Rackspace since that time.
  
Marconi provides an interface for users to create queues, producers add messages to those queues, workers to claims messages from the queues, and listeners to get feeds from queue activity. Marconi is not designed to provide workflow management, but will guarantee first in, first out (FIFO) order for single producer models.  
+
Marconi's overarching goal is to provide web-scale, highly-available message queues to web applications that run on OpenStack. Marconi runs on Nova servers, behind OpenStack load balancers and Keystone authentication middleware. The Marconi implementation makes use of Oslo, and follows the standard OpenStack hacking guidelines.
  
Backed by a clean, simple REST API, the Marconi API has been developed in a pluggable fashion, with a goal of allowing users to switch storage backends as desired. Today, Marconi supports MongoDB in production, with additional storage plugins in testing and active development.
+
Marconi provides an interface for creating queues, enqueuing messages, and later claiming those messages for processing. It also provides an interface to clients for listing messages without needing to claim them (ala RSS and Atom), in order to support pub-sub and passive auditing of producer-consumer workflows. The service guarantees first in, first out (FIFO) order for single producer models, best-effort ordering otherwise.  
  
The Marconi API exposes the following calls:
+
The Marconi architecture is pluggable in terms of both transport and storage. Reference drivers for HTTP (WSGI), SQLAlchemy, and MongoDB will be provided in the initial release, along with a SQLite driver to facilitate development and testing. Other transport and storage drivers have been proposed, and are currently under discussion. Marconi deployments will support HA, and will be able to scale horizontally in the transport and storage layers to enable large deployments. A routing proxy and migration service is also under development, to provide further horizontal scaling across multiple independent Marconi partitions, for use in extremely large deployments.
 +
 
 +
 
 +
The Marconi v1.0 API defines the following operations:
 +
 
 +
'''API''':
 +
 
 +
:Get JSON home document
 +
:Check node health
  
 
'''Queues''':
 
'''Queues''':
  
:Create Queue
+
:Create a queue
:List Queues
+
:List queues
:Set Queue Metadata
+
:Set queue metadata
:Get Queue Metadata
+
:Get queue metadata
:Get Queue Stats
+
:Get queue stats
:Delete Queue
+
:Delete a queue
  
 
'''Messages'''
 
'''Messages'''
 
+
:Post one or more messages
:Post Message(s)
+
:List messages
:Get Messages
+
:Get a message
:Get a Specific Message
+
:Delete a message
:Claim Messages
+
:Get multiple messages
 +
:Delete multiple messages
  
 
'''Claims'''
 
'''Claims'''
:
+
:Claim messages
:Query Claim
+
:Query a claim
:Update Claim
+
:Update a claim
:Release Claim
+
:Release a claim
  
 
==Basic roadmap for the project==
 
==Basic roadmap for the project==
Line 41: Line 55:
  
 
* REST API for queue/messages/claim management
 
* REST API for queue/messages/claim management
 +
* FIFO guaranteed for single producer, best-effort otherwise
 +
* Guaranteed once-and-only-once delivery (with some caveats, see also [[Marconi/guarantees]])
 +
* (Relatively) simple to achieve HA and durability
 
* Multi-tenant
 
* Multi-tenant
* Integrated with Keystone for authentication
+
* Keystone-based authentication
* Command Line Interface
+
* Python bindings
* Python Bindings
 
 
* Error handling, edge cases
 
* Error handling, edge cases
 +
* Oslo-based logging, configuration, etc.
 
* Ruggedization/packaging for production deployments
 
* Ruggedization/packaging for production deployments
 
* Operator Documentation
 
* Operator Documentation
* Performance Tuning
+
* Performance Tests & Tuning
* Bulk message deletion
+
* Unit & Functional Tests
  
 
===Future Release Plans===
 
===Future Release Plans===
  
 
* Non-Python language bindings
 
* Non-Python language bindings
* CI/CD, Complete System Tests
+
* Add Functional, Load & Security tests to a CI pipeline
 
* Message Compression
 
* Message Compression
* MySQL Support
+
* WebSocket and ZMQ transport drivers
 +
* SQLAlchemy storage driver
 
* Kombu Integration
 
* Kombu Integration
 
* Message tags
 
* Message tags
 +
* Exploration of a domain-specific storage backend
  
 
==Location of project source code==
 
==Location of project source code==
  
Marconi on Launchpad
+
Server: https://github.com/stackforge/marconi<br />
Marconi at StackForge
+
Client: https://github.com/stackforge/python-marconiclient
python-marconiclient at StackForge
 
  
 
==Programming language, required technology dependencies==
 
==Programming language, required technology dependencies==
Line 76: Line 94:
 
==Level of maturity of software and team==
 
==Level of maturity of software and team==
  
The Marconi project has been under active development at both Rackspace and Red Hat since January of 2013 and recently reached a level of maturity that will support production work loads. The current code base contains numerous unit and functional tests. A comprehensive load and security test suite is currently under development. The Marconi team is ramping up on client library development, and will be incubating the common OpenStack API library as part of the project.
+
The Marconi project has been under active development at both Rackspace and Red Hat since January of 2013 and recently reached a level of maturity that will support production work loads. The current code base contains numerous unit and functional tests. A comprehensive load and security test suite is currently under development. The Marconi team is ramping up on client library development, and plans to have Python bindings ready to coincide with v1.0 of the service.
  
 
==Proposed project technical lead and qualifications==
 
==Proposed project technical lead and qualifications==
  
Kurt Griffiths is a Principal Architect at Rackspace Hosting, currently working with other technical leaders inside and outside the company on cloud architecture, strategy and evangelism. Kurt focuses on web service APIs, agents, cryptography, and performance. He works with other senior technologists to provide day-to-day and long-term technical guidance for products developed at Rackspace.
+
Kurt Griffiths is a Principal Architect at Rackspace Hosting, currently working with other technical leaders inside and outside the company on cloud architecture, strategy and evangelism. Kurt focuses on web service APIs, agents, cryptography, and performance. He works with other senior technologists to provide day-to-day and long-term technical guidance for products developed at Rackspace. Over the years Kurt has had the opportunity to work in a variety of roles at companies including Rackspace, Novell, ExxonMobil, Microsoft, and several startups.
 +
 
 +
Kurt is passionate about building open, collaborative software communities, believing they provide the best future for our profession.
 +
 
 
==Other project developers and qualifications==
 
==Other project developers and qualifications==
===Current Code Contributors===
+
===Active Code Contributors===
  
 +
In addition to Kurt Griffiths:
 +
 +
* ''Alejandro Cabrera'' is a software developer at Rackspace. His expertise include: Python, Linux, open-source, community building, distributed systems, and automation.
 +
* ''Allan Metts'' is the Director of Engineering for the Rackspace Atlanta office, which is the site responsible for the company's Cloud Backup, Jungle Disk, and Cloud Queuing products.  Prior to Rackspace, Allan led the development of several large-scale systems in the telecommunications, transportation, and financial sectors. He specializes in distributed systems that crunch large amounts of data at high rates of speed.
 
* ''Flavio Percoco'' is a Software Engineer at Red Hat.  Active in the OpenStack community, Flavio has contributed and reviewed code on OpenStack’s Block Storage, Image, Oslo, and Stable Maintainers teams.  
 
* ''Flavio Percoco'' is a Software Engineer at Red Hat.  Active in the OpenStack community, Flavio has contributed and reviewed code on OpenStack’s Block Storage, Image, Oslo, and Stable Maintainers teams.  
* ''Malini Kamalambal''is a Quality Engineer at Rackspace, with over 11 years of industry experience.  In addition to test planning and functional execution, Malini has history in software development.  
+
* ''Malini Kamalambal'' is a Quality Engineer at Rackspace, with over 11 years of industry experience.  In addition to test planning and functional execution, Malini has history in software development.  
 
* ''Oz Akan'' is a Cloud Engineering Manager at Rackspace and has been working in technology since 1997.  Oz is an expert in database management, having spent time working with MySQL, Oracle, and Mongo DB.  
 
* ''Oz Akan'' is a Cloud Engineering Manager at Rackspace and has been working in technology since 1997.  Oz is an expert in database management, having spent time working with MySQL, Oracle, and Mongo DB.  
* ''Allan Metts'' is the Director of Engineering for the Rackspace Atlanta office, which is the site responsible for the company's Cloud Backup, Jungle Disk, and Cloud Queuing products.  Prior to Rackspace, Allan led the development of several large-scale systems in the telecommunications, transportation, and financial sectors. He specializes in distributed systems that crunch large amounts of data at high rates of speed.
 
* ''Victoria Martínez de la Cruz'' is a Licenciate in Computer Science student and one of the interns of GNOME OPW for OpenStack.
 
 
* ''Zhihao Yuan'' is a Software Developer at Rackspace, and a member of the Technical Leadership Team.  Zihihao has experience with FreeBSD, C++, Python, and more.
 
* ''Zhihao Yuan'' is a Software Developer at Rackspace, and a member of the Technical Leadership Team.  Zihihao has experience with FreeBSD, C++, Python, and more.
* "Alejandro Cabrera" is a software developer at Rackspace. His expertise include: Python, Linux, open-source, community building, distributed systems, and automation.
 
  
 
===Previous Code Contributors===
 
===Previous Code Contributors===
  
* ''Jamie Painter'' is a Software Manager at Rackspace, as well as a member of Rackspace's Technical Leadership Team. He specializes in architecture/design, C/C++, Python, Linux, shell scripting, encryption and security.  
+
* ''Bryan Davidson'' is a Software Developer at Rackspace. He has over 5 years of industry experience building web-scale applications.
* ''Bryan Davidson'' is a Software Developer at Rackspace. He has over 5 years of industry experience building web-scale applications.  
+
* ''Jamie Painter'' is a Software Manager at Rackspace, as well as a member of Rackspace's Technical Leadership Team. He specializes in architecture/design, C/C++, Python, Linux, shell scripting, encryption and security.  
 
+
* ''Victoria Martínez de la Cruz'' is a Licenciate in Computer Science student and one of the interns of GNOME OPW for OpenStack.
  
 
===API Design Contributors (no code submitted)===
 
===API Design Contributors (no code submitted)===
 +
* ''Hong Yuan'' is a Senior Software Engineer at HP, working on projects such as Messaging as a Service and Database as a Service.  Hong specializes in software architecture and development with high scalability, high availability, and low latency.
 
* ''John Hopper'' is a Software Develop at Rackspace and has over 11 years of industry experience.  John specializes in tackling large, complex models and prototyping solutions.
 
* ''John Hopper'' is a Software Develop at Rackspace and has over 11 years of industry experience.  John specializes in tackling large, complex models and prototyping solutions.
* ''Hong Yuan'' is a Senior Software Engineer at HP, working on projects such as Messaging as a Service and Database as a Service.  Hong specializes in software architecture and development with high scalability, high availability, and low latency.
 
 
* ''Travis Reeder'' is the a Co-Founder and CTO at Iron.io.  Travis specializes in building highly scalable applications, Go (golang), Ruby on Rails, and Java.
 
* ''Travis Reeder'' is the a Co-Founder and CTO at Iron.io.  Travis specializes in building highly scalable applications, Go (golang), Ruby on Rails, and Java.
  
Line 112: Line 134:
 
==Related Links==
 
==Related Links==
  
Documentation
+
Project Home: https://launchpad.net/marconi <br />
[https://github.com/stackforge/marconi Marconi on OpenStack Wiki]
+
API Blueprint: https://wiki.openstack.org/wiki/Marconi/specs/api/v1
 +
 
 +
==Status==
 +
[[Marconi/Incubation/Graduation|Pending Graduation]]
 +
 
 +
==Raised Questions + Answers==
 +
 
 +
''Q. Is Marconi a provisioning service?''<br />
 +
A. No, Marconi is a data service, providing a web-friendly API for sending, observing, and claiming arbitrary messages.
 +
 
 +
''Q. What was wrong with qpid, rabbitmq, activemq, zeromq, ${your favorite queue here} that required marconi?''<br />
 +
A, The features supported by AMQP brokers, ZMQ, and Marconi certainly do overlap in some areas. At the same time, however, each of these options offer distinct features that may or may not align with what a web developer is trying to accomplish.
 +
 
 +
Here are a few of Marconi's unique features, relative to the other options mentioned:
 +
 
 +
*  Multi-tenant
 +
*  Keystone integration
 +
*  100% Python
 +
*  First-class, stateless, firewall-friendly HTTP(S) transport driver
 +
*  Simple protocol, easy for clients to implement
 +
*  Scales to an unlimited number of queues and clients
 +
*  Per-queue stats, useful for monitoring and autoscale
 +
*  Tag-based message filtering (planned)
 +
 
 +
 
 +
Relative to SQS, Marconi:
 +
 
 +
*  Is open-source and community-driven
 +
*  Supports private and hybrid deployments
 +
*  Offers hybrid pub-sub and producer-consumer semantics
 +
*  Provides a clean, modern HTTP API
 +
*  Can route messages to multiple queues (planned)
 +
*  Can perform custom message transformations (planned)
  
[https://launchpad.net/marconi Project Home]
 
  
==Status==
+
''Q. Will Marconi support delayed queues ala SQS?''<br />
Pending Incubation
+
A. Generally speaking, we are always open to feature requests based on the needs of the community. Our feeling is that advanced features, such as [https://blueprints.launchpad.net/marconi/+spec/delayed-queues delayed queues], are certainly within the scope of the program.
 +
 
 +
''Q. Does Marconi include a custom storage backend or does it build on an existing solution?''<br />
 +
A. Marconi provides a modular backend framework which allows different storage drivers to be loaded via stevedore. We already have drivers for MongoDB (production) and SQLite (development/testing), and plan to also implement a production driver for MySQL. We are also experimenting with Redis as a potential backend.
 +
 
 +
''Q. Will Marconi support AMQP?''<br />
 +
A. The v1.0 API semantics can be mapped to AMQP as a transport. However, for Marconi's initial release, it will not be possible to use use an AMQP broker directly as a storage backend. We plan to revisit the question of an AMQP backend while designing the v2.0 API, pending community interest.
 +
 
 +
''Q. Will Marconi support frontends other than HTTP?''<br />
 +
A. Marconi service's transport layer (frontends) is modular and supports arbitrary drivers, assuming they support the API's defined semantics. HTTP is implemented currently as a WSGI driver, and we are planning a ZMQ module as well. We have discussed direct support for other transports such as WebSockets and AMQP, pending community interest.
 +
 
 +
''Q. How will Marconi handle 3rd-party drivers?''<br />
 +
A. The Marconi program will maintain a list of compatible drivers, as defined by a standard set of functional tests that a given transport or storage driver must pass (which will also be used to vet official drivers).  In addition, requirements will be documented on the OpenStack wiki ([[Marconi/docs/drivers|work in progress]]). 3rd-parties will be responsible for implementing and maintaining their own drivers.
 +
 
 +
''Q. What WSGI framework does Marconi use, and has the team considered Pecan?''<br />
 +
A. Marconi's WSGI driver currently uses Falcon, a micro-framework designed for high-throughput web services. During incubation (if accepted), we are planning an [https://blueprints.launchpad.net/marconi/+spec/pecan-framework in-depth evaluation] of Pecan to see if it meets our needs.
 +
 
 +
==Incubation Period Notes==
 +
 
 +
[[Marconi/Incubation/Graduation|Graduation To Do]]

Latest revision as of 18:42, 7 August 2014

Project Codename

Marconi

Mission

To produce an OpenStack message queueing API and service that affords a variety of distributed application messaging patterns in an efficient, scalable and highly-available manner, and to create and maintain associated Python libraries and documentation.

Summary

Marconi is a messaging and notifications service for the OpenStack product portfolio, supporting both producer-consumer and publish-subscribe modes. Marconi is designed to perform and scale in a multi-tenant environment.

Detailed Description

In order to support more complex web applications running on OpenStack, a messaging service is needed. To fill this need, the Marconi project was proposed at the Grizzly design summit. Requirements were discussed with the community and used to form the basis for the project's charter. Implementation began in February 2013, and we have been fortunate to receive regular, major contributions from Red Hat and Rackspace since that time.

Marconi's overarching goal is to provide web-scale, highly-available message queues to web applications that run on OpenStack. Marconi runs on Nova servers, behind OpenStack load balancers and Keystone authentication middleware. The Marconi implementation makes use of Oslo, and follows the standard OpenStack hacking guidelines.

Marconi provides an interface for creating queues, enqueuing messages, and later claiming those messages for processing. It also provides an interface to clients for listing messages without needing to claim them (ala RSS and Atom), in order to support pub-sub and passive auditing of producer-consumer workflows. The service guarantees first in, first out (FIFO) order for single producer models, best-effort ordering otherwise.

The Marconi architecture is pluggable in terms of both transport and storage. Reference drivers for HTTP (WSGI), SQLAlchemy, and MongoDB will be provided in the initial release, along with a SQLite driver to facilitate development and testing. Other transport and storage drivers have been proposed, and are currently under discussion. Marconi deployments will support HA, and will be able to scale horizontally in the transport and storage layers to enable large deployments. A routing proxy and migration service is also under development, to provide further horizontal scaling across multiple independent Marconi partitions, for use in extremely large deployments.


The Marconi v1.0 API defines the following operations:

API:

Get JSON home document
Check node health

Queues:

Create a queue
List queues
Set queue metadata
Get queue metadata
Get queue stats
Delete a queue

Messages

Post one or more messages
List messages
Get a message
Delete a message
Get multiple messages
Delete multiple messages

Claims

Claim messages
Query a claim
Update a claim
Release a claim

Basic roadmap for the project

Version 1.0

  • REST API for queue/messages/claim management
  • FIFO guaranteed for single producer, best-effort otherwise
  • Guaranteed once-and-only-once delivery (with some caveats, see also Marconi/guarantees)
  • (Relatively) simple to achieve HA and durability
  • Multi-tenant
  • Keystone-based authentication
  • Python bindings
  • Error handling, edge cases
  • Oslo-based logging, configuration, etc.
  • Ruggedization/packaging for production deployments
  • Operator Documentation
  • Performance Tests & Tuning
  • Unit & Functional Tests

Future Release Plans

  • Non-Python language bindings
  • Add Functional, Load & Security tests to a CI pipeline
  • Message Compression
  • WebSocket and ZMQ transport drivers
  • SQLAlchemy storage driver
  • Kombu Integration
  • Message tags
  • Exploration of a domain-specific storage backend

Location of project source code

Server: https://github.com/stackforge/marconi
Client: https://github.com/stackforge/python-marconiclient

Programming language, required technology dependencies

Language: Python

Dependencies: Falcon, MongoDB, Oslo, Stevedore

Is project currently open sourced? What license?

Yes, under the Apache 2.0 license.

Level of maturity of software and team

The Marconi project has been under active development at both Rackspace and Red Hat since January of 2013 and recently reached a level of maturity that will support production work loads. The current code base contains numerous unit and functional tests. A comprehensive load and security test suite is currently under development. The Marconi team is ramping up on client library development, and plans to have Python bindings ready to coincide with v1.0 of the service.

Proposed project technical lead and qualifications

Kurt Griffiths is a Principal Architect at Rackspace Hosting, currently working with other technical leaders inside and outside the company on cloud architecture, strategy and evangelism. Kurt focuses on web service APIs, agents, cryptography, and performance. He works with other senior technologists to provide day-to-day and long-term technical guidance for products developed at Rackspace. Over the years Kurt has had the opportunity to work in a variety of roles at companies including Rackspace, Novell, ExxonMobil, Microsoft, and several startups.

Kurt is passionate about building open, collaborative software communities, believing they provide the best future for our profession.

Other project developers and qualifications

Active Code Contributors

In addition to Kurt Griffiths:

  • Alejandro Cabrera is a software developer at Rackspace. His expertise include: Python, Linux, open-source, community building, distributed systems, and automation.
  • Allan Metts is the Director of Engineering for the Rackspace Atlanta office, which is the site responsible for the company's Cloud Backup, Jungle Disk, and Cloud Queuing products. Prior to Rackspace, Allan led the development of several large-scale systems in the telecommunications, transportation, and financial sectors. He specializes in distributed systems that crunch large amounts of data at high rates of speed.
  • Flavio Percoco is a Software Engineer at Red Hat. Active in the OpenStack community, Flavio has contributed and reviewed code on OpenStack’s Block Storage, Image, Oslo, and Stable Maintainers teams.
  • Malini Kamalambal is a Quality Engineer at Rackspace, with over 11 years of industry experience. In addition to test planning and functional execution, Malini has history in software development.
  • Oz Akan is a Cloud Engineering Manager at Rackspace and has been working in technology since 1997. Oz is an expert in database management, having spent time working with MySQL, Oracle, and Mongo DB.
  • Zhihao Yuan is a Software Developer at Rackspace, and a member of the Technical Leadership Team. Zihihao has experience with FreeBSD, C++, Python, and more.

Previous Code Contributors

  • Bryan Davidson is a Software Developer at Rackspace. He has over 5 years of industry experience building web-scale applications.
  • Jamie Painter is a Software Manager at Rackspace, as well as a member of Rackspace's Technical Leadership Team. He specializes in architecture/design, C/C++, Python, Linux, shell scripting, encryption and security.
  • Victoria Martínez de la Cruz is a Licenciate in Computer Science student and one of the interns of GNOME OPW for OpenStack.

API Design Contributors (no code submitted)

  • Hong Yuan is a Senior Software Engineer at HP, working on projects such as Messaging as a Service and Database as a Service. Hong specializes in software architecture and development with high scalability, high availability, and low latency.
  • John Hopper is a Software Develop at Rackspace and has over 11 years of industry experience. John specializes in tackling large, complex models and prototyping solutions.
  • Travis Reeder is the a Co-Founder and CTO at Iron.io. Travis specializes in building highly scalable applications, Go (golang), Ruby on Rails, and Java.

Infrastructure requirements (testing, etc)

Currently on StackForge using Gerrit and Jenkins to run the unit test suite and flake8. A QA cluster is nearing completion which will facilitate automated load and security testing, with infrastructure donated by Rackspace. The QA cluster will serve as a reference configuration for operators wishing to deploy a Marconi cluster. No additional infrastructure requirements are expected.

Have all current contributors agreed to the OpenStack CLA?

Yes.

Related Links

Project Home: https://launchpad.net/marconi
API Blueprint: https://wiki.openstack.org/wiki/Marconi/specs/api/v1

Status

Pending Graduation

Raised Questions + Answers

Q. Is Marconi a provisioning service?
A. No, Marconi is a data service, providing a web-friendly API for sending, observing, and claiming arbitrary messages.

Q. What was wrong with qpid, rabbitmq, activemq, zeromq, ${your favorite queue here} that required marconi?
A, The features supported by AMQP brokers, ZMQ, and Marconi certainly do overlap in some areas. At the same time, however, each of these options offer distinct features that may or may not align with what a web developer is trying to accomplish.

Here are a few of Marconi's unique features, relative to the other options mentioned:

  • Multi-tenant
  • Keystone integration
  • 100% Python
  • First-class, stateless, firewall-friendly HTTP(S) transport driver
  • Simple protocol, easy for clients to implement
  • Scales to an unlimited number of queues and clients
  • Per-queue stats, useful for monitoring and autoscale
  • Tag-based message filtering (planned)


Relative to SQS, Marconi:

  • Is open-source and community-driven
  • Supports private and hybrid deployments
  • Offers hybrid pub-sub and producer-consumer semantics
  • Provides a clean, modern HTTP API
  • Can route messages to multiple queues (planned)
  • Can perform custom message transformations (planned)


Q. Will Marconi support delayed queues ala SQS?
A. Generally speaking, we are always open to feature requests based on the needs of the community. Our feeling is that advanced features, such as delayed queues, are certainly within the scope of the program.

Q. Does Marconi include a custom storage backend or does it build on an existing solution?
A. Marconi provides a modular backend framework which allows different storage drivers to be loaded via stevedore. We already have drivers for MongoDB (production) and SQLite (development/testing), and plan to also implement a production driver for MySQL. We are also experimenting with Redis as a potential backend.

Q. Will Marconi support AMQP?
A. The v1.0 API semantics can be mapped to AMQP as a transport. However, for Marconi's initial release, it will not be possible to use use an AMQP broker directly as a storage backend. We plan to revisit the question of an AMQP backend while designing the v2.0 API, pending community interest.

Q. Will Marconi support frontends other than HTTP?
A. Marconi service's transport layer (frontends) is modular and supports arbitrary drivers, assuming they support the API's defined semantics. HTTP is implemented currently as a WSGI driver, and we are planning a ZMQ module as well. We have discussed direct support for other transports such as WebSockets and AMQP, pending community interest.

Q. How will Marconi handle 3rd-party drivers?
A. The Marconi program will maintain a list of compatible drivers, as defined by a standard set of functional tests that a given transport or storage driver must pass (which will also be used to vet official drivers). In addition, requirements will be documented on the OpenStack wiki (work in progress). 3rd-parties will be responsible for implementing and maintaining their own drivers.

Q. What WSGI framework does Marconi use, and has the team considered Pecan?
A. Marconi's WSGI driver currently uses Falcon, a micro-framework designed for high-throughput web services. During incubation (if accepted), we are planning an in-depth evaluation of Pecan to see if it meets our needs.

Incubation Period Notes

Graduation To Do