Murano/ApplicationCatalog

Project Mission
The mission for this project is to provide a way to make third-party applications and services running on VMs or even external services available as self-service for OpenStack. These applications may be a simple, a single VM or complex, multi tier applications with autoscaling and self healing.

From the third-party tool developer’s perspective, the application catalog will provide a way to publish applications and services, including deployment rules and requirements, suggested configuration, output parameters and billing rules. It will also provide a way to track billing and usage information.

From the user’s perspective, the application catalog will be a place to find and self-provision third-party applications and services, integrate them into their environment, and track usage information and costs.

Rationale
The Application Catalog project aims to make it easier for, external developers to create application and services for use by OpenStack end users. In this way, these third party developers can enrich the OpenStack ecosystem to make it more attractive for users, and users can get more out of their OpenStack clusters more easily, fostering adoption of OpenStack itself.

Installing third party services and applications can be difficult in any environment, but the dynamic nature of an OpenStack environment can make this problem worse. The project is designed to solve this problem by providing an additional integration layer between third party components an OpenStack infrastructure. This integration layer makes it possible to provide both Infrastructure-as-a-Service and Platform-as-a-Service from a single control plane. For users, this control plan then becomes a single interface from which they can provision an entire fully-functional cloud-based application environment.

Accessed by self-service portals, application catalogs contain a list of applications and services from which cloud consumers select for self-provisioning. Because each application and service definition includes all of the information the system needs for deployment, users will not have to work through various IT departments in order to provision a cloud service, nor are users required to provide detailed IT specifications. They are only required to provide business and organization requirements.

In this proposal, we will look at the Application Catalog design, define the various components, and look at the various use cases involved.

Application Catalog Design
The application catalog makes use of existing OpenStack projects and services, as you can see in this diagram:



The Application Catalog project integrates all OpenStack components directly and indirectly via Heat. The Ceilometer service will collect usage information, which the Murano-API will use during billing rules processing to calculate billing information.

The Murano API will expose API calls to manage (CRUD) services available for deployment. This API will be used by the Service admin user interface to simplify Service management.

Application Catalog components and definitions
The key components of the application catalog are:


 * Metadata service - Storage for application descriptions and service metadata, including application-related Heat templates, software configurations and provision scripts, as well as UI forms definitions
 * API service - Used by the UI, the API enables administrators to manipulate application metadata, developers to publish and manage application, and end-users to perform application configuration and self-provisioning. The API provides RBAC capabilities to control access to various components and applications.
 * Environment control - Integration with OpenStack deployment engines such as Heat, Trove, and Savanna to perform the actual deployment of the application.
 * Billing - Works with Ceilometer usage data to prepare billing information for the deployed application. Each application can have its own billing rules, set by the developer who created it.

In evaluating this proposal, consider these definitions:


 * Environment - A logical entity for grouping applications into a single deployment.
 * Application - A software component that has its own configuration and deployment scripts. An application may use multiple VMs for deployment, such as a Galera cluster or a SQL Cluster, for example.
 * Application definition - A description of an application, which includes metadata describing how the application is to be deployed, application requirements, the application UI and so on.
 * RBAC - Role Based Access Control
 * Murano - a group of components which provides a control on application management and deployment. It is accessible via a defined endpoint, and may refer to one more more application catalog instances.  For example, it may refer to a single local instance, a department tree of instances, a remote instance, or any combination of those.
 * Application catalog instance - An instance of the Murano service that hosts service catalog components and manages one or more application definitions. An application catalog instance may be local and associated with a single cloud, or remote, providing services to multiple OpenStack environments.
 * Application Publisher - An individual or company that publishes an application to an application catalog
 * End User - An individual or company attempting to self-provision an application from an application catalog
 * Catalog Administrator - An individual or company that maintains an application catalog and determines any relevant policies regarding its use.

Application Publisher
The process begins when an Application Publisher creates a new application description and publishes it to a Murano endpoint. It will then be available within any application catalog instances defined by that Murano endpoint, depending on the policies for that instance.

Application Publishers should be able to create new application by defining service metadata, describing properties and specifying all the steps necessary for deploying the service and its dependencies. The developer can create this definition from scratch or use an existing definition by extending it, similar to inheritance in the object-oriented paradigm. The Application Publisher can define the external dependencies of the service. This list of dependencies defines the other services (specified by their type) that must be present in the environment when the given service is being deployed.

Consider this example. An Application Publisher creates a service that provides a web application. The developer provides the name and other service properties, and specifies that the external dependencies are a web server and a database. When users want to deploy this service in an environment, they need to have a web-server service and a database service in that environment, and must be able to specify how they want to fulfill those requirements. (See the End User use cases for more information.)

The Application Publisher may define additional terms of use for their Service. For example, the developer may limit its usage and extensibility (via inheritance or referencing from another service) or specify billing rules.

Another important set of parameters that the Application Publisher may specify in the Service Definition are the usage metrics. These usage metrics define which aspects of the service should be monitored by Ceilometer or other monitoring tools supported by Murano when its instances are running. The Application Publisher can then specify the billing rules used with those metrics, essentially defining how much usage of a service will cost the user.

[Note that this proposal is meant to define a project that provides billing information, but because different organizations have different needs, it doesn’t define actual payment methods; payment may be handled by an external component, or it may be addressed in future versions of Murano.]

A service definition is not bound to any particular OpenStack deployment or instance of Murano. The developer may create a service definition and then publish that definition in several service catalog instances, (as long as publishing is permitted by the administrator of that catalog (see below)).

Catalog Administrator
A published service definition is managed by the catalog administrator.

Catalog administrators are the maintainers of the application service catalog. They have the ability to manually add or remove service definitions in a catalog, or act as moderators allowing or disallowing other Application Publishers to publish their service definitions. This control can be granular or not, as the administrator chooses. For example, the administrator may specify that any new submissions must be approved before being available to any end users, or the administrator may instead choose to make services available only to the OpenStack tenant associated with the application publisher until a service is approved. The administrator can also decide to make all services available to all upon submission, as in the case of a test cloud, or a small cloud in which all developers are “trusted”.

Administrators may also define their own billing rules, which will be in addition to the billing rules specified by the application publisher (if they were defined). This enables catalog administrators to cover the costs involved in running and maintaining the cloud. For example, a service that requires Microsoft Windows may incur a licensing cost for the operating system; this mechanism enables the catalog administrator to recoup that cost.

Catalog administrators also configure Role-Based Access Control rules (RBAC), which define which end users (which are associated with tenants) of the cloud have access to which services in the catalog, and whether they may be directly deployed or must be approved before deployment (see End User use cases). The billing rules for a particular service may also be defined specifically for a given tenant or a given user.

End User
Finally the service is ready for the end user.

A user should be able to create environments composed of one or more available services. The process is as follows:

The user browses a list of available services and selects one or more for deployment. If a selected service has dependencies that require other services to be deployed in the same environment, the user may either select an instance of the necessary service from instances of that type that are already present in the environment, or add a new instance of that type instead. Dependencies may include other services, or they may include resources such as a floating IP address or license key. Each service added to the environment must be properly configured; the user is prompted to provide all required properties, and the input is validated according to the rules defined in each service definition. When the user has finished configuring the environment, he or she can deploy the environment -- if he or she has the appropriate permissions. (See below.) Deployment of the environment means that instances are created, services are deployed, and all required configuration actions take place.

In some environments, it will be more appropriate for end users to submit their deployments to IT as a ticket. The IT department can then sanity-check the definitions, determine whether they are appropriate, and approve, modify, or deny the deployment. If the request is approved or modified, the IT department can then initiate the deployment, rather than the user.

Users can browse any deployed environments for which they have permissions, and inspect their state. Inspection includes the ability to determine which services are running on which nodes, how the services are configured, and so on. Users can modify service settings, add new services or remove existing ones, validate the changes (i.e. check that all the required properties are set to valid values, all the service dependencies exist and so on), and redeploy the environment by propagating these changes into the Cloud. The user can also inspect the usage metrics of the services running in his or her environments, and see billable activities and the total amount of money spent for a particular service.

Applications Examples

 * RDS and NoSQL databases provided by Trove or deployed by custom deployment scripts.
 * Hadoop Cluster provided by Sahara.
 * OpenShift PaaS Cluster provisioned through Heat
 * MS SQL Cluster deployed by Murano workflows
 * IIS Server Farm deployed by Murano workflows
 * File Share (SMB or NFS) deployed by Murano
 * Chef Server or Puppet Master node installed my Murano workflows
 * Nagios or Zabbix monitoring managed by Murano workflows

Examples
1. Enterprise IT department created private OpenStack cloud. There is a hardware storage which is used by Cinder service. By default this hardware storage exposes simple volumes without additional features and options turned on. At the same time this hardware storage has features like on-line encryption and anti-virus scanner which may be enabled for each volume. IT department decided to expose these features in OpenStack and created a new service in application catalog with name ISCSI disk with encryption and anti-virus scanner. Now end-user can create secured FileStorage solution by adding corresponding applications (NFS or CIFS server) and ISCSI disk with encryption and anti-virus scanner to the environment. During environment deployment Murano application catalog generates Heat template to provision required resources in OpenStack and generates all necessary workflows in Mistral service to configure a new volume on storage device and attach it to VM.

2. IT department want to provide OpenStack cloud users with a set of applications with limited supported configurations. IT department publish these services in application catalog and configure access to published applications for specific users. Now end users can use Horizon dashboard to configure their own environments with using published applications as a building blocks. As end-user can't modify underlying application definition IT service can offer support for published services as it fully controls workflows and Heat templates for published applications.