Murano/MultiCloudSupport

Summary
This is a high level proposal for adding to Murano capability to deploy applications across different clouds, choice of the target cloud should be controllable through configuration and through policies by cloud operator and/or by end user through user interface. Murano should include support for the following clouds: OpenStack (natively), VMWare, Amazon. We consider that Murano is a part of the OpenStack and can deploy applications not only to the same cloud where it’s installed but also to the several other clouds (hybrid-cloud applications / disaster recovery).

Proposed solution includes implementation of the following features/pieces:
 * plugins for each cloud, implementing lifecycle for the low level cloud resources
 * core library classes describing hierarchy of the abstract cloud resources and actual implementations with cloud-specific features
 * configuration-based identity/credentials management for clouds
 * policy-based dependency injection mechanism for classes
 * Changes required in the user interface (OpenStack Dashboard)

High-Level Description
Applications in Murano are defined using object-oriented language called MuranoPL. Murano does not allow to execute any IO operations directly from MuranoPL, all the calls to the needed HTTP APIs should be written in Python and then consumed from the MuranoPL. Murano has set of predefined MuranoPL classes built-in called core library which provide access to cloud resources and which essentially are SDK for developing apps.

We need to design new hierarchy of classes in core library for the abstract cloud resources (e.g. vm, image) and actual implementations for each cloud with cloud-specific features backed by Python-based plugins for each cloud with actual lifecycle for this resources.

Identity management is rudimentary in Murano, because currently Murano operates under auth-token received from the user, and does not store any credentials by itself. At the beginning storing configuration for each cloud in simple configuration file may be sufficient, while spec how to implement properly identity management going to be drafted. Policies are set of rules and settings that are used by the cloud operator to control the behavior of applications. This is another area which is not yet covered in Murano. Among others policy can be used to configure what cloud platform application should be deployed on. The goal is to create mechanism that will be able to choose best possible interface implementation for particular cloud or policy requirements.

Existing Implementation
Murano uses Object Oriented Modeling approach to define all cloud entities starting from low level infrastructure entities like servers, networks, volumes etc and up to application components and application themselves. In this approach everything that can be used or deployed or consumed in any way is an object. As usual objects expose data fields (properties) and methods - actions that can be performed on the given object.

As commonly found in most OOP systems objects are instances of classes that define their structure and behavior. Murano uses self-developed programming language called MuranoPL for that purpose.

MuranoPL is a full-fledged dynamic object oriented language like Python is and has most of concepts typical to that kind of languages: class inheritance (including multi-inheritance), method overloading, polymorphism, exception handling as well as some basic things like variables, loops and so on. On the other hand it is also a domain specific language (DSL) that was specially crafted for Murano use cases.

One of the main use case is application composition. It is common that different classes are developed independently by different vendors. In fact Murano application catalog is a repository for classes and when a user constructs application environment in UI he is implicitly constructing a new program from the classes in repository. In order for different classes to coexist together in that program and be able to talk to each other MuranoPL provides rich abilities for classes to express their requirements on each other. Another property that MuranoPL shares with many other DSLs is that it is expressed in YAML format.

Because anyone can contribute to the class repository MuranoPL was designed to be sandboxed and secure in order to be run on a shared server without a need for code to be trusted. As a result it cannot do any I/O on its own. Instead it relies on a lower-level pieces of code written in Python that provide MuranoPL with APIs to interact with other cloud services, perform operations on virtual machines etc. Some of those APIs are part of Murano source code while others can be implemented as plugins.

Murano is shipped with number of classes that represent common infrastructure entities like Instance, Network etc. Those classes together are known as Murano Core Library and are used by most applications to interact with the underlying cloud. Currently those classes were written with only OpenStack in mind and their implementation mostly relies on OpenStack Heat API to create OpenStack resources. That makes it not sufficient for other clouds where there is no such API available and entities not always have the same properties as they have in OpenStack.

Proposed Solution
There are several pieces of technology that are missing in Murano in order for it to operate on several clouds:
 * Cloud Python Bindings to talk to various cloud management platform APIs
 * Multi-Cloud Core Library
 * Multi-Cloud Identity Management
 * Cloud Credentials
 * Instance Credentials (Credentials to log into the orchestrated virtual machines)
 * Policy System
 * Openstack Dashboard Enhancements



Cloud Python Bindings
To make Murano Multi Cloud Capable we need python bindings to talk to the API of these various cloud management platforms. As of today murano can orchestrate low level cloud resources only on an Openstack Cloud Management Platform using Heat Orchestration Engine APIs (HOT Templates). This functionality is baked into the Murano Core Library instead of as plugin making it hard to provide a flexibility of choosing various python bindings. As part of this release low level cloud python bindings will be implemented as plugins which the Core MuranoPL library can invoke.

Plugins for the following clouds will be implemented
 * Openstack, VMWare vCenter and Amazon AWS

Lifecycle Implementation will be provided for the following low level cloud resources
 * Compute (Virtual Machines).
 * Security Groups for the following clouds:
 * Openstack and AWS
 * Concept of Security Groups needs to be explored in vCenter.
 * Life cycle implementation of other Cloud Resources such as Networks and Storage Blocks will be addressed in future releases. This release addresses the issue of attaching the compute to already existing network and storage resources.

These python bindings will be bundled with the murano code but will not be installed by default as part of the murano installation. To reduce the dependency of going through openstack approval process for packages required to support these cloud python bindings, these bindings will have to be installed separately by the cloud administrators via pip install or by running the setup.py scripts provided with the plugins.

Multi-Cloud Core Library
We need more complex class hierarchy, properties and methods that are common to all (or most) clouds need to be extracted into top-level interfaces and let cloud-specific classes to implement those interfaces. Those classes may expose additional properties and methods that are specific for particular cloud to give applications to use full cloud capabilities for cases where there are no requirements for cross-cloud portability and yet.

In order to build such class hierarchy we need to do a careful analysis on a wide range of cloud platforms in order to find out what is common and what is specific to each implementation. Not only the difference between attributes or similar entities in different clouds but also different network topologies, security management need to be taken into account. Also there may be other factors that affect the design like distinction between Linux and Windows instances, different ways to interact with the instance, alternate network implementations in the same cloud and so on.

New classes will require new native APIs to talk to different cloud platforms - Python classes or functions that will expose low-level cloud functions to MuranoPL. API methods should provide ability to perform basic CRUD operations on target cloud. Those APIs should be as small and flexible as possible while the most of orchestration and logic should be put into MuranoPL YAML classes.

Identity Management
Murano is a multi-tenant solution that is well integrated with the OpenStack. In particular Murano neither deals with authentication and authorization nor asks user for a password or other credentials. Instead it relies on OpenStack dashboard to provide Murano with a Keystone token that can be used to further authenticate to other OpenStack services during deployment time.

However Keystone token is no help for deployment on other platforms. So for cases where the target cloud is not the one Murano runs in authentication must become a part of the Murano workflow and part of the user input. This also implies that the policies must affect user input as well because different cloud platforms may require different input fields and different processing.

The configurations of target clouds (including the access credentials) should become first-class Murano objects, accessible in MuranoPL as properties of cloud-affiliated classes (applications, VMs etc). In future, such configuration objects should be creatable using regular generic Murano tooling, but for now we may have a set of custom DB tables to store them, API methods to access and modify and UI forms to interact with them.

The following are the two basic identities that need to be managed by Murano as part of this project. etc..
 * Cloud Credentials
 * Cloud Credentials are the credentials needed to access the user’s cloud account
 * Typically the Cloud Credentials consist of the following fields:
 * User
 * Key: For certain clouds such as Azure, the key is not a string but is a PEM file
 * Cloud Endpoints
 * IP Address of the Cloud Management Platform.
 * Environment variables specific to each cloud
 * Datacenter (GCE, vCenter)
 * Project (GCE)

Hence need to design the Cloud Credentials object with an eye on supporting different key types and other properties that need to be included.

Execute the plan using SSH as a transport mechanism ala the Ansible Way, agentless approach to implementing the execution plan.
 * Credentials to access the Virtual Machines that are launched.
 * This would be required to support SSH as a transport mechanism to get the execution plan to the Murano Agent or

Implementation details and in depth details on these will be captured as part of a separate spec. In addition to this, the spec will also capture on how this might be integrated with other openstack projects such as Barbican and Keystone.

Dependency Injection Mechanism
There needs to be a mechanism in MuranoPL to choose the best possible interface implementation of low level cloud resources based on the cloud the operator wants to deploy a given environment (set of applications) on. Without this mechanism it would be hard for the cloud administrators or users to define cloud agnostic packages.

For example, the application need only specify that it needs any implementation of “Instance” interface but the cloud operator wishes to deploy the application on AWS then the dependency injection mechanism will need to find that particular Instance implementation that can work on AWS. This would be the same case for other low level resources such as Network, Security Groups, Storage Volumes.

Also to express class capabilities (like on which cloud platform it can be used) additional metadata need to be present in MuranoPL. This is need to be designed and developed as well. To summarize the above the policy system can be split into the following parts:
 * Dependency injection mechanism need to be developed that will make Murano automatically inject best possible implementation for any requested interface and policy rules.
 * Improve MuranoPL so that it will be possible to put additional metadata on classes (methods, properties) to express their capabilities. This mechanism can be used to express other useful metadata as well like UI hints, property groups, ACL hints etc. so it need to be extensible enough.

Initial release of Murano Multi Cloud will support a simple policy that provides the users the ability to deploy applications/environments on the cloud of their choice at any given time. Future releases of this will target more complex policies such as Cloud Bursting, Cloud Balancing and Cloud Migration giving the users the ability to fully realize their cloud strategies.

Openstack Dashboard Enhancements
To make this release a minimum viable release without the dependency on the user portal strategy the following bare minimal enhancements need to be made to the Openstack Murano Dashboard.
 * Panel for configuring the Cloud Properties such as
 * Cloud Credentials
 * Instance Credentials
 * Enhancement that provides the ability to choose which Cloud to configure the environment for.
 * Design needs to take into account the capabilities that need to be provided in future releases such as Cloud Bursting, Cloud Balancing and Migration as well as use cases such as High Availability Scenarios (Primary/Secondary Cloud Concepts).
 * Enhancements to chose the networks (VPCs, Distributed Port Groups) etc.
 * Enhancements to choose the security port groups.
 * Enhancements to choose the various cloud specific flavors
 * vCenter does not have a concept of flavor, need to provide a way to input number of CPUs and amount of RAM.

These are at a high level some of the enhancements that need to be made to Openstack Murano Dashboard to make this release a minimum viable release till the strategy for integration with the user cloud portal gets hashed out. More details on this aspect will be captured in a separate spec.

Alternative 1

 * Extend heat to support native multi cloud instances and other objects
 * Define new resources in heat template
 * Extend nova to support native instance extensions
 * Networking needs to be looked at holistically
 * Horizon extensions to support multi cloud



Blueprints

 * Multi Cloud Support in Murano
 * Primary Assignee: Satish Grandhi
 * Add Multi Cloud Capability to the Murano Core Library Classes
 * Primary Assignee: Nagraj Shenoy
 * Multi Cloud Capable Murano Amazon EC2 Python Binding
 * Primary Assignee: Ruchika Luthra
 * Multi Cloud Capable Murano - Identity Management
 * Primary Assignee: Govardhan Sakanna

Specifiations

 * Multi Cloud Capable Murano Amazon EC2 Python Binding Spec
 * Primary Assignee: Ruchika Luthra
 * Add Multi Cloud Capability to the Murano Core Library Classes
 * Primary Assignee: Nagraj Shenoy
 * Multi Cloud Capable Murano - Identity Management
 * Primary Assignee: Govardhan Sakanna

Test Cases
This section gives rough guidelines on the test cases that need to be executed for Murano extension for Multi Cloud support.

System Level TestCase 1
User is able to deploy an environment consisting of a multi tiered application across multiple Openstack installation
 * Linux based application
 * Windows based application

System Level TestCase 2
User is able to deploy an environment consisting of a multi tiered application on AWS from an Openstack Cloud
 * Linux based application
 * Windows based application

System Level TestCase 3
User is able to deploy an environment consisting of a multi tiered application on another Openstack cloud
 * Linux based application
 * Windows based application

System Level TestCase 4
User is able to deploy an environment consisting of a multi tiered application on VMware based cloud (vSphere vCenter)
 * Linux based application
 * Windows based application

System Level TestCase 5
User is able to deploy an environment consisting of several applications with each individual app on a cloud of his choice
 * Linux based application
 * Windows based application

System Level TestCase 6
User is able to deploy an environment consisting of a multi tiered application on two clouds at a given time for high availability purposes (Primary/Secondary Cloud concept for a given application)

Functional TestCase 1
Target Funtional Area Cloud Identity Management

Description
 * User logs into Murano workspace and clicks on Manage -> Cloud Credentials
 * User should be able to create, read, update & delete Cloud Credentials.
 * User should be able to validate the cloud credentials before creating them in the Murano Database.
 * Verify that credentials are stored encrypted in the database.
 * Credentials are scoped per OpenStack Project
 * Update and Delete is disallowed in case the credentials are used by existing application deployments.

Functional TestCase 2
Target Funtional Area Security Group Management

Description Developer through App should be able to create, read, update & delete Cloud Security Groups I.e. Security Groups modelled in cloud agnostic manner

Functional TestCase 3
Target Funtional Area Environment Management

Description
 * User logs into Murano workspace and clicks on Application Catalog -> Environments
 * User creates a new environment on an Openstack cloud.
 * User sees a subset of applications tied to Openstack show up to be selected.
 * User should be able to key in information needed for i.e. flavor, glance image name, keypair, ssh username/password.

Note: In case of multi cloud we’ll have to model cloud instances separately which will maintain a loose coupling to instances present in different clouds.

Functional TestCase 4
Target Funtional Area Environment Management

Description
 * User logs into Murano workspace and clicks on Application Catalog -> Environments
 * User creates a new environment on a EC2 cloud.
 * User sees a subset of applications tied to EC2 show up to be selected.
 * User should be able to key in information needed for i.e. flavor, ami name, security groups,keypair,ssh username/ password.
 * User schedules the deployment of the environment now.

Functional TestCase 5
Target Funtional Area Environment Management

Description
 * User logs into Murano workspace and clicks on Application Catalog -> Environments
 * User creates a new environment picking up EC2 & Openstack cloud.
 * User sees a relevant applications tied to both clouds.
 * User key in information needed for i.e. flavor, image names, security groups,keypair,ssh username/ password.
 * User schedules the deployment of the environment now.