Jump to: navigation, search

Difference between revisions of "Cue"

(Use Cases)
(Code)
 
(11 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 
= Cue =
 
= Cue =
 
== Mission Statement ==
 
== Mission Statement ==
The OpenStack Message Broker Provisioning as a Service Mission: To provide scalable and reliable Message Broker provisioning functionality for off the shelf messaging technologies, as well as to provide advanced management, tuning, and administration of underlying message brokers.
+
To provide scalable and reliable Message Broker provisioning functionality for off the shelf messaging technologies, as well as to provide advanced management, tuning, and administration of underlying message brokers.
  
 
== Description ==
 
== Description ==
Line 8: Line 8:
  
 
== Why ==
 
== Why ==
Messaging is a common development pattern for building loosely coupled distributed systems.  Messaging systems also act as glue between independent applications.  Several off-the-shelf products exist that implement messaging and queuing semantics.
+
Messaging is a common development pattern for building loosely coupled distributed systems.  Messaging systems also act as glue between independent applications.  Several off-the-shelf products exist that implement messaging and queuing semantics, many of which implement open protocols such as AMQP 0.9 and 1.0.  
 
    
 
    
Provisioning and supporting Messaging systems for an individual application can be a time consuming and painful experience. This product aims to simplify the provisioning and management of messaging systems, providing High Availability and auto-healing capabilities to the end user, while providing tenant-level isolation.
+
Provisioning and supporting messaging systems for an individual application can be a time consuming and painful experience. Cue aims to simplify the provisioning and management of messaging systems, providing high availability and auto-healing capabilities for both the cloud operator and end user, while providing secure tenant-level isolation.
 
    
 
    
The main goal of this service is to simplify the end user Application development lifecycle and allow the developer to focus on their application, instead of the backend middleware services.
+
The main goal of this service is to simplify the end user application development lifecycle for both legacy and "cloud native" applications, allowing the developer to focus on their application, instead of the underlying middleware services.
  
== Use Cases (in-work) ==
+
== Use Cases ==
'''Macro'''
+
'''Micro services - distributing work between services deployed on top of OpenStack via PaaS offerings like Cloud Foundry'''
# IT teams providing managed, highly available message brokers to their app development teams, on private clouds they own and operate.
 
# OpenStack service providers seeking to provide a managed, highly available messaging service with multiple backends such as RabbitMQ and Kafka to their end users.
 
  
'''Micro'''
+
John works in the IT Operations department at a telco company and has been tasked with providing a highly available messaging service to internal development teams. Prior to this effort, development teams were free to stand up their own messaging components to support their applications, causing significant headaches for the Ops team. The company has standardized on AMQP as the standard protocol for sending messages between applications and will use RabbitMQ as the default broker for most projects. For portability and scalability reasons, the company is also in the process of standardizing on Cloud Foundry as the default platform for building cloud native applications on top of both OpenStack and non-OpenStack clouds.
# Distributing work between micro-services deployed on top of OpenStack via PaaS offerings like Cloud Foundry.
+
 
# Sending messages between legacy systems and cloud native applications.
+
To start, John downloads and installs Cue in an existing OpenStack environment. As the cloud administrator installing Cue, John has a couple of options at install time. He can specify which messaging brokers he wants to enable within the environment. He can also control default quotas and create flavors that are attached to specific brokers, such as RabbitMQ or Kafka. A flavor can be configured to be shared across multiple tenants in a secure manner, depending on the broker enabled (not all brokers may support this feature). The installation results in the Cue service appearing in the Keystone catalog as "Messaging Service". It also creates a new top-level service in Horizon called "Messaging".
 +
 
 +
Now that Cue is installed and operational, development teams can freely provision instances/clusters of message brokers using Cue's REST API, CLI, or Horizon. Because John's company has chosen Cloud Foundry as the platform standard, he has one more step to take. John must integrate Cue with a [http://docs.cloudfoundry.org/services/api.html#api-overview Cloud Foundry service broker] so that Cue is exposed to development teams at the Cloud Foundry layer. This is important because most development teams building cloud native applications at the company do not access the OpenStack layer directly, meaning they do not have Keystone credentials (or access by way of Keystone<->AD/LDAP integration). By integrating Cue with a Cloud Foundry service broker, development teams can freely provision topics/queues as needed for the applications from the Cloud Foundry CLI or UI. The Cue service broker for Cloud Foundry exposes the same flavors, known as "plans" in Cloud Foundry terminology, that John configured during install time. Once the service broker is implemented, Cue can be exposed directly to any Cloud Foundry environment or enabled in an existing one.
 +
 
 +
Sally, a developer at the telco company is building a new prototype which uses a micro services architecture leveraging Cloud Foundry. She authenticates to a Cloud Foundry cluster already running on top of OpenStack. The Cloud Foundry UI shows that a new service is available to her called "Messaging Service". She sees several plans for the service and selects the small RabbitMQ plan, which results in a single instance of RabbitMQ (1 small virtual machine) being provisioned and bound to the micro service. Provisioning and binding happens in seconds. Connection info/credentials to the RabbitMQ instance are subsequently presented. Sally then selects the appropriate [http://www.rabbitmq.com/clients.html client library for RabbitMQ] and continues writing her prototype. On the backend, the RabbitMQ instance is part of the Cue pool of resources which is monitored and managed by the Ops team.
  
 
== Goals ==
 
== Goals ==
Line 45: Line 47:
  
 
== API ==  
 
== API ==  
API documentation is located here: [https://wiki.openstack.org/wiki/Cue/api Cue API Documentation]
+
API documentation is located here:
 +
 
 +
[https://wiki.openstack.org/wiki/Cue/api Cue API Approach 1]
 +
 
 +
[https://wiki.openstack.org/wiki/Cue/api_2 Cue API Approach 2]
 +
 
 +
[https://wiki.openstack.org/wiki/Cue/api1vsapi2 Comparison]
  
 
== Code ==
 
== Code ==
We've started building the project, it now lives on Stackforge.
+
Source code:
 +
 
 +
https://github.com/openstack/cue
 +
 
 +
Code Reviews:
 +
 
 +
https://review.openstack.org/#/q/project:openstack/cue,n,z
  
https://github.com/stackforge/cue
+
== Find Us ==
 +
Come talk to us on #openstack-cue on Freenode.

Latest revision as of 04:42, 1 October 2015

Cue

Mission Statement

To provide scalable and reliable Message Broker provisioning functionality for off the shelf messaging technologies, as well as to provide advanced management, tuning, and administration of underlying message brokers.

Description

Cue is a Message Broker provisioning service for Openstack. Its goal is to simplify and automate the complex tasks of provisioning, management, and administration of message brokers. Users of the service can create and manage multiple clusters of Message brokers, of various types as needed. The service will provide resource isolation at the VM level, and does not attempt to implement multi-tenancy at the message broker layer.

Why

Messaging is a common development pattern for building loosely coupled distributed systems. Messaging systems also act as glue between independent applications. Several off-the-shelf products exist that implement messaging and queuing semantics, many of which implement open protocols such as AMQP 0.9 and 1.0.

Provisioning and supporting messaging systems for an individual application can be a time consuming and painful experience. Cue aims to simplify the provisioning and management of messaging systems, providing high availability and auto-healing capabilities for both the cloud operator and end user, while providing secure tenant-level isolation.

The main goal of this service is to simplify the end user application development lifecycle for both legacy and "cloud native" applications, allowing the developer to focus on their application, instead of the underlying middleware services.

Use Cases

Micro services - distributing work between services deployed on top of OpenStack via PaaS offerings like Cloud Foundry

John works in the IT Operations department at a telco company and has been tasked with providing a highly available messaging service to internal development teams. Prior to this effort, development teams were free to stand up their own messaging components to support their applications, causing significant headaches for the Ops team. The company has standardized on AMQP as the standard protocol for sending messages between applications and will use RabbitMQ as the default broker for most projects. For portability and scalability reasons, the company is also in the process of standardizing on Cloud Foundry as the default platform for building cloud native applications on top of both OpenStack and non-OpenStack clouds.

To start, John downloads and installs Cue in an existing OpenStack environment. As the cloud administrator installing Cue, John has a couple of options at install time. He can specify which messaging brokers he wants to enable within the environment. He can also control default quotas and create flavors that are attached to specific brokers, such as RabbitMQ or Kafka. A flavor can be configured to be shared across multiple tenants in a secure manner, depending on the broker enabled (not all brokers may support this feature). The installation results in the Cue service appearing in the Keystone catalog as "Messaging Service". It also creates a new top-level service in Horizon called "Messaging".

Now that Cue is installed and operational, development teams can freely provision instances/clusters of message brokers using Cue's REST API, CLI, or Horizon. Because John's company has chosen Cloud Foundry as the platform standard, he has one more step to take. John must integrate Cue with a Cloud Foundry service broker so that Cue is exposed to development teams at the Cloud Foundry layer. This is important because most development teams building cloud native applications at the company do not access the OpenStack layer directly, meaning they do not have Keystone credentials (or access by way of Keystone<->AD/LDAP integration). By integrating Cue with a Cloud Foundry service broker, development teams can freely provision topics/queues as needed for the applications from the Cloud Foundry CLI or UI. The Cue service broker for Cloud Foundry exposes the same flavors, known as "plans" in Cloud Foundry terminology, that John configured during install time. Once the service broker is implemented, Cue can be exposed directly to any Cloud Foundry environment or enabled in an existing one.

Sally, a developer at the telco company is building a new prototype which uses a micro services architecture leveraging Cloud Foundry. She authenticates to a Cloud Foundry cluster already running on top of OpenStack. The Cloud Foundry UI shows that a new service is available to her called "Messaging Service". She sees several plans for the service and selects the small RabbitMQ plan, which results in a single instance of RabbitMQ (1 small virtual machine) being provisioned and bound to the micro service. Provisioning and binding happens in seconds. Connection info/credentials to the RabbitMQ instance are subsequently presented. Sally then selects the appropriate client library for RabbitMQ and continues writing her prototype. On the backend, the RabbitMQ instance is part of the Cue pool of resources which is monitored and managed by the Ops team.

Goals

  1. Produce a Message Broker Provisioning Service for Openstack
  2. Protocol Agnostic
  3. Support multiple Broker backends
  4. Provide HA and Fault Tolerance by leveraging underlying Broker Cluster capabilities
  5. Provide tuned and properly configured Message Brokers on-demand
  6. Support Broker management operations such as Backup of Configuration, Scale Up, Scale Down

Non-Goals

  1. Data plane API
  2. Multi-tenancy at the Message Broker layer

Brokers

Cue will support the following brokers (in order of priority):

  1. RabbitMQ
  2. qpid
  3. Kafka
  4. .. others?

API

API documentation is located here:

Cue API Approach 1

Cue API Approach 2

Comparison

Code

Source code:

https://github.com/openstack/cue

Code Reviews:

https://review.openstack.org/#/q/project:openstack/cue,n,z

Find Us

Come talk to us on #openstack-cue on Freenode.