Obsolete:QueueService - Revision history

Fifieldt: Fifieldt moved page QueueService to Obsolete:QueueService

2013-07-25T19:09:37Z

Fifieldt moved page QueueService to Obsolete:QueueService

Ryan Lane: Text replace - "NOTOC" to ""

2013-02-17T23:30:55Z

Text replace - "__NOTOC__" to ""

Tom at 13:19, 8 January 2013

2013-01-08T13:19:03Z

EricDay at 22:32, 19 May 2011

2011-05-19T22:32:52Z

EricDay at 18:34, 12 May 2011

2011-05-12T18:34:52Z

Show changes

JonathanBryce: linking into related projects category

2011-03-04T20:05:21Z

linking into related projects category

EricDay at 19:23, 2 March 2011

2011-03-02T19:23:38Z

EricDay at 18:54, 24 February 2011

2011-02-24T18:54:42Z

EricDay at 18:36, 24 February 2011

2011-02-24T18:36:47Z

EricDay at 02:21, 23 February 2011

2011-02-23T02:21:05Z

← Older revision		Revision as of 23:30, 17 February 2013
Line 1:		Line 1:
−	~~__NOTOC__~~	+

	<pre><nowiki>#!wiki caution		<pre><nowiki>#!wiki caution

← Older revision		Revision as of 13:19, 8 January 2013
Line 1:		Line 1:
	__NOTOC__		__NOTOC__
		+
		+	<pre><nowiki>#!wiki caution
		+	'''This page is outdated'''
		+
		+	The content of this page has not been updated for a very long time. The Burrow project has not been updated for the current release.
		+	</nowiki></pre>
		+
		+
	= Queue Service =		= Queue Service =

@@ Line 4: / Line 4: @@
 This document is a proposal and plan for a queue service, it is not an official [[OpenStack]] project yet. Once there is an implementation to evaluate, it will be submitted to the [[OpenStack]] [[Governance/PPB|Project Policy Board]] for approval.
-This project has been started on Launchpad under the name "Burrow". You can find it at: https://launchpad.net/burrow
+This project has been started on Launchpad under the name "Burrow". You can find the source code, current tasks, and other information at: https://launchpad.net/burrow
-Most of the docs that were once here are now located on the new burrow documentation site at: http://burrow.openstack.org/
+The docs that were once here are now located on the new burrow documentation site at: http://burrow.openstack.org/
 == Possible Features in the Future ==
@@ Line 13: / Line 13: @@
 * Allow secondary indexes on user-defined metadata.
 * Support in-process router "workers" for advanced routing and replication configurations. This will require API additions to manage the routing rules.
 ----
 [[Category:RelatedProjects]]

← Older revision		Revision as of 20:05, 4 March 2011
Line 195:		Line 195:

	When the proxy is handling worker requests it will use the same hashing algorithms. When a worker GETs a queue name to read messages, the account portion is parsed and a connection is made to all queue servers. It will then aggregate messages from all queue servers handling that account into one view for the worker to consume. The proxy and queue servers may need to use a more efficient multiplexing protocol that can keep state for multiple accounts and requests rather than simple REST based calls to keep the number of connections reasonable.		When the proxy is handling worker requests it will use the same hashing algorithms. When a worker GETs a queue name to read messages, the account portion is parsed and a connection is made to all queue servers. It will then aggregate messages from all queue servers handling that account into one view for the worker to consume. The proxy and queue servers may need to use a more efficient multiplexing protocol that can keep state for multiple accounts and requests rather than simple REST based calls to keep the number of connections reasonable.
		+
		+	----
		+	[[Category:RelatedProjects]]

@@ Line 83: / Line 83: @@
 * limit=COUNT - Limit the number of matched messages. This allows a worker to grab as few or as many messages it can handle at once. For example, a worker resizing images may only grab one at a time, but a worker processing log messages may grab many for efficient batch processing.
-* last=ID - Only match IDs after the given ID. This allows for multi-cast messages by always having a 'hide' value of 0 and not deleting messages (let the TTL delete them automatically).
+* marker=ID - Only match IDs after the given ID. This allows for multi-cast messages by always having a 'hide' value of 0 and not deleting messages (let the TTL delete them automatically).
 * hidden=true|false - Whether to match messages that are currently hidden. The default should be false for queues, but true for message-specific operations.
 * wait=SECONDS - Wait the given time period for a new message if no messages currently match. This allows workers to long-poll if needed.
@@ Line 165: / Line 165: @@
 Worker1: Return from blocking GET request with message id1
 Worker2: Return from blocking GET request with message id1
-Worker1: GET /account/queue?wait=60&last=id1
+Worker1: GET /account/queue?wait=60&marker=id1
-Worker2: GET /account/queue?wait=60&last=id1
+Worker2: GET /account/queue?wait=60&marker=id1
 </nowiki></pre>

← Older revision	Revision as of 19:09, 25 July 2013
(No difference)

@@ Line 90: / Line 90: @@
 == Behavior and Constraints ==
-The default queue behavior is FIFO, although LIFO should be a trivial addition. Message insert and retrieval order is only guaranteed for a single queue server. When used along with a proxy, it is possible that the proxy server will return newer messages from one backing queue server before older messages in another. The proxy server will make a best effort to drain the queue servers evenly, but strict message ordering (for messages that hash to different queue servers) is not guaranteed. Applications can usually be design to accommodate this in some way.
+The default queue behavior is FIFO, although LIFO should be a trivial addition (this will be revisited later). Message insert and retrieval order is only guaranteed for a single queue server. When used along with a proxy, or when a client uses multiple queue servers, it is possible that newer messages from one queue server will be returned before older messages in another. The proxy server will make a best effort to drain the queue servers evenly, but strict message ordering is not guaranteed. Applications that require strict ordering will need to use some external mechanism to guarantee this.
 In certain failure scenarios such as a queue server crash, it is possible, depending on deployment configuration, for messages to either be lost or duplicated. Lost messages should only happen if they were inserted with no persistence (such as an in-memory queue or some other non-persistent backing store). It is up to the deployment to offer guarantees according to what message persistence modules they choose to offer. For example, a queue server backed by MySQL will only be as durable as the MySQL configuration, which can vary from in-memory only to extremely durable (commit to disk and replicated).
 Duplicate messages can happen when a worker pulls a message to work on, the server crashes, the worker finishes, and the message delete call fails. When the server comes back up, the message will appear in the queue again (assuming it was in a persistent message store). Applications can handle this in a variety of ways, from simply running the job twice, to rolling back changes on the delete failure, to tracking unique message IDs to ensure they are processed only once.
 == Other Features ==

@@ Line 4: / Line 4: @@
 <<[[TableOfContents]]()>>
-This document is a proposal and plan for a queue service, it is not an official [[OpenStack]] project yet. Once there is a simple implementation to evaluate, it will be submitted the project to the [[OpenStack]] Project Oversight Committee for approval.
+This project has been started on Launchpad under the name "Burrow". You can find it at: https://launchpad.net/burrow
-The reason for initiating a new project vs using an existing one is due to simplicity, modularity, and scale. Very few (if any) existing queue systems out there were built with multi-tenant cloud use cases in mind. Very few also have a simple and extensible REST API. There are possible solutions to build an AMQP based service, but AMQP brings complexity and a protocol not optimized for high latency and intermittent connectivity.
+This document is a proposal and plan for a queue service, it is not an official [[OpenStack]] project yet. Once there is an implementation to evaluate, it will be submitted to the [[OpenStack]] Project Oversight Committee for approval.
-This project has been started on Launchpad under the name "Burrow". You can find it at: https://launchpad.net/burrow
+The reason for initiating a new project vs using an existing one is due to simplicity, modularity, and scale. Very few (if any) existing queue systems out there were built with multi-tenant cloud use cases in mind. Very few also have a simple and extensible REST API. There are possible solutions to build an AMQP based service, but AMQP brings complexity and a protocol not optimized for high latency and intermittent connectivity. The benefits and constraints of a distributed, multi-tenant environment (cloud services) requires a different approach than single tenant and/or clustered services.
 == Prototype ==
@@ Line 34: / Line 34: @@
 The queue service will consist of the following components:
 * Server - This is where messages are routed through. Clients will insert messages and workers will receive messages. Multiple servers may be run for high availability, and servers do not need to be aware of one another.
 * Proxy - This is an optional service that sits between the server and clients/workers. This allows clients and workers to connect to a single address and proxy to multiple servers. This is only required for extremely large deployments (such as a public cloud service) and will not be implemented until at least the second major milestone.
-* Client - An external application that inserts or updates messages in the queue.
-* Worker - An external application that reads, updates, or deletes messages in the queue. Workers need to be idempotent, as the queue service may deliver the same message more than once in some failover situations. Note that the same application can be both a client and worker, the differentiation is only made based on typical behaviors, not on what they are capable of doing.
+=== Data ===
 * Accounts - This represents a user, application, or some other account context. Every message is associated with an account, and accounts can have ACLs to control the CRUD operations for other accounts and queues. ACL entries may have some simple pattern matching capabilities.
 * Queue - A queue is simply a namespace, it doesn't have metadata associated with it. Queue names are unique within an account context. Queues cannot be created or destroyed, they automatically appear to clients and workers when a message exists for the queue name, and is automatically removed when all messages for a given queue name are deleted.
 * Messages - A message consists of a unique ID, metadata, and a body. The unique ID is composed of the account, queue name, and a message ID. A server will only contain a single message for a given unique ID. If two messages are inserted with the same unique ID, the last one overwrites the first.
-===  Message Metadata ===
+====  Message Metadata ====
 * ttl=SECONDS - How long the message will exist in the queue.
@@ Line 81: / Line 87: @@
 * wait=SECONDS - Wait the given time period for a new message if no messages currently match. This allows workers to long-poll if needed.
 * detail=none|id|metadata|all - What details to return in the response body.
 == Other Features ==

← Older revision		Revision as of 02:21, 23 February 2011
Line 7:		Line 7:

	The reason for initiating a new project vs using an existing one is due to simplicity, modularity, and scale. Very few (if any) existing queue systems out there were built with multi-tenant cloud use cases in mind. Very few also have a simple and extensible REST API. There are possible solutions to build an AMQP based service, but AMQP brings complexity and a protocol not optimized for high latency and intermittent connectivity.		The reason for initiating a new project vs using an existing one is due to simplicity, modularity, and scale. Very few (if any) existing queue systems out there were built with multi-tenant cloud use cases in mind. Very few also have a simple and extensible REST API. There are possible solutions to build an AMQP based service, but AMQP brings complexity and a protocol not optimized for high latency and intermittent connectivity.
		+
		+	This project has been started on Launchpad under the name "Burrow". You can find it at: https://launchpad.net/burrow

	== Prototype ==		== Prototype ==