Cue

= Cue =

Mission Statement
To provide scalable and reliable Message Broker provisioning functionality for off the shelf messaging technologies, as well as to provide advanced management, tuning, and administration of underlying message brokers.

Description
Cue is a Message Broker provisioning service for Openstack. Its goal is to simplify and automate the complex tasks of provisioning, management, and administration of message brokers. Users of the service can create and manage multiple clusters of Message brokers, of various types as needed. The service will provide resource isolation at the VM level, and does not attempt to implement multi-tenancy at the message broker layer.

Why
Messaging is a common development pattern for building loosely coupled distributed systems. Messaging systems also act as glue between independent applications. Several off-the-shelf products exist that implement messaging and queuing semantics, many of which implement open protocols such as AMQP 0.9 and 1.0. Provisioning and supporting messaging systems for an individual application can be a time consuming and painful experience. Cue aims to simplify the provisioning and management of messaging systems, providing high availability and auto-healing capabilities for both the cloud operator and end user, while providing secure tenant-level isolation. The main goal of this service is to simplify the end user application development lifecycle for both legacy and "cloud native" applications, allowing the developer to focus on their application, instead of the underlying middleware services.

Use Cases
Micro services - distributing work between services deployed on top of OpenStack via PaaS offerings like Cloud Foundry

John works in the IT Operations department at a telco company and has been tasked with providing a highly available messaging service to internal development teams. Prior to this effort, development teams were free to stand up their own messaging components to support their applications, causing significant headaches for the Ops team. The company has standardized on AMQP as the standard protocol for sending messages between applications and will use RabbitMQ as the default broker for most projects. For portability and scalability reasons, the company is also in the process of standardizing on Cloud Foundry as the default platform for building cloud native applications on top of both OpenStack and non-OpenStack clouds.

To start, John downloads and installs Cue in an existing OpenStack environment. As the cloud administrator installing Cue, John has a couple of options at install time. He can specify which messaging brokers he wants to enable within the environment. He can also control default quotas and create flavors that are attached to specific brokers, such as RabbitMQ or Kafka. A flavor can be configured to be shared across multiple tenants in a secure manner, depending on the broker enabled (not all brokers may support this feature). The installation results in the Cue service appearing in the Keystone catalog as "Messaging Service". It also creates a new top-level service in Horizon called "Messaging".

Now that Cue is installed and operational, development teams can freely provision instances/clusters of message brokers using Cue's REST API, CLI, or Horizon. Because John's company has chosen Cloud Foundry as the platform standard, he has one more step to take. John must integrate Cue with a Cloud Foundry service broker so that Cue is exposed to development teams at the Cloud Foundry layer. This is important because most development teams building cloud native applications at the company do not access the OpenStack layer directly, meaning they do not have Keystone credentials (or access by way of Keystone<->AD/LDAP integration). By integrating Cue with a Cloud Foundry service broker, development teams can freely provision topics/queues as needed for the applications from the Cloud Foundry CLI or UI. The Cue service broker for Cloud Foundry exposes the same flavors, known as "plans" in Cloud Foundry terminology, that John configured during install time. Once the service broker is implemented, Cue can be exposed directly to any Cloud Foundry environment or enabled in an existing one.

Sally, a developer at the telco company is building a new prototype which uses a micro services architecture leveraging Cloud Foundry. She authenticates to a Cloud Foundry cluster already running on top of OpenStack. The Cloud Foundry UI shows that a new service is available to her called "Messaging Service". She sees several plans for the service and selects the small RabbitMQ plan, which results in a single instance of RabbitMQ (1 small virtual machine) being provisioned and bound to the micro service. Provisioning and binding happens in seconds. Connection info/credentials to the RabbitMQ instance are subsequently presented. Sally then selects the appropriate client library for RabbitMQ and continues writing her prototype. On the backend, the RabbitMQ instance is part of the Cue pool of resources which is monitored and managed by the Ops team.

Goals

 * 1) Produce a Message Broker Provisioning Service for Openstack
 * 2) Protocol Agnostic
 * 3) Support multiple Broker backends
 * 4) Provide HA and Fault Tolerance by leveraging underlying Broker Cluster capabilities
 * 5) Provide tuned and properly configured Message Brokers on-demand
 * 6) Support Broker management operations such as Backup of Configuration, Scale Up, Scale Down

Non-Goals

 * 1) Data plane API
 * 2) Multi-tenancy at the Message Broker layer

Brokers
Cue will support the following brokers (in order of priority):
 * 1) RabbitMQ
 * 2) qpid
 * 3) Kafka
 * 4) .. others?

API
API documentation is located here:

Cue API Approach 1

Cue API Approach 2

Comparison

Code
Source code:

https://github.com/openstack/cue

Code Reviews:

https://review.openstack.org/#/q/project:openstack/cue,n,z

Find Us
Come talk to us on #openstack-cue on Freenode.