Murano/SimplifiedMetadataRepository

Scope
This document defines the functional requirements and gives some design and implementation proposals for the simplified (initial) version of Murano Metadata repository.

Purpose
The simplified (initial) version of Murano Metadata Repository will be used to operate (store, display, group and use for deployment) the metadata objects in their current format (so called “Murano vCurrent”) - without service interdependencies, inheritance, aggregation and most of other service-specific metadata. The intent is to create simplest possible storage for metadata objects, without any significant modifications in the existing Murano components. However, this simple Repository will serve as a starting point for future modification and refactoring - which, eventually, will lead to an advanced, metadata-driven catalog of interoperable Services (so called “Murano vNext”)

Structure of metadata objects
Currently, there are 5 types of metadata objects used by Murano Conductor:
 * Workflow
 * Heat templates
 * Agent templates
 * Agent config templates
 * Powershell scripts

Also, there are UI definitions, which are used by Murano Dashboard. It is proposed to add one more type of metadata (“Service Manifests”) which will bind all other types of metadata objects with the given Service and will be used by Metadata Service to build a Service Catalog. All the metadata objects should be stored at Metadata Repository grouped by type, maintaining a nested, directory-like structure (all the files of each type should go to the same directory). The directory structure should be defined according to the following table:

Service Manifest
This is a new metadata object proposed to support Service Catalog feature. At the first release of Metadata Repository it will define just several basic properties of the services, but will dramatically evolve in future, eventually defining the service and its place in the Murano eco-system. It also binds together all other types of metadata objects by defining the names of the files which are required for a given service, thus allowing to easily list and retrieve all the dependencies of the single service. The manifest should be defined as a yaml file, containing the following properties: In future revisions this property will contain an ordered list of forms (in some format like filename:formname), so UI of single service may be loaded from several shared files. Scripts - a list of filenames of the powershell scripts for this Service.
 * Manifest version - a version of metadata format, indicating how to interpret all other fields. This is the only field The first implementation of Metadata Repository (implemented according to the current document) should have it equal to 0.1. All subsequent format modifications should increment this value. It is expected that when we consider Murano vNext implemented, the manifest should have version 1.0
 * Service Display Name - for now, this duplicates the “name” property of the Dynamic UI. In future, we will remove it from there. In the first version let’s have the duplication: the one in UI files will be used for displaying name of the service in “New Service” dialog, the one in Manifest will be used to list Service in the UI of Metadata Repository.
 * Description - for now, this duplicates the “description” property of the Dynamic UI. In future, we will remove it from there. In the first version let’s have the duplication: the one in UI files will be used for displaying description of the service in “New Service” dialog, the one in Manifest will be used to list Service in the UI of Metadata Repository.
 * Fully-qualified service name - the complete service name, intended to be globally unique. For now, this should duplicate the “type” property of the Dynamic UI. In future, we will remove it from there, and in manifest it will be in an inverted domain-based form, like com.mirantis.murano.windows.activedirectory
 * Author - the name of the author of the Service Definition. Just for the information to be displayed in the Metadata Repository UI
 * Version - the version of the Service Definition. Just for the information to be displayed in the Metadata Repository UI
 * Enabled - boolean field allowing cloud administrator to disallow usage of this service definition. The service which has this property equal to False will be present in Metadata Repository, but the service will not be available to select in the “New Service” dialog of the Murano Dashboard.
 * UI forms - a list of filenames of UI definition files defining User Interface for this Service. In the first version it will always contain a single file per Service due to the restrictions of the current DynamicUI design. All the forms (steps) of Create Service dialog will be loaded from this file.
 * Workflows - a list of filenames of workflow definitions files for this Service.
 * Heat Templates - a list of filenames of heat template files for this Service.
 * Agent Templates - a list of filenames of agent template files for this Service.

All the files which are referenced at Workflows, Heat Templates, Agent Templates and Scripts should have filepath which is relative to the appropriate directory (see table above). For example, a workflow from file named “AD.xml” will be looked at “data/workflows/AD.xml”, and an agent template named “SqlServerCluster/FailoverCluster.template” will be looked at “data/templates/agent/SqlServerCluster/FailoverCluster.template” “..” is forbidden in the filename, so the search should not be done in the directories outside of the root directory for the given metadata object type. If any of the files mentioned in these section is not found in the Metadata Repository, the Service definition in considered invalid and cannot be used neither in Dashboard nor in Conductor.

Metadata-querying API
Metadata Repository will provide a RESTful API endpoint for Murano Components (in the first version - for Conductor and Dashboard only), so they can act as a client for Metadata Repository and query it for the needed metadata objects. There will be two main APIs available: This call will be made by Murano Dashboard. In response, the Metadata Repository will inspect all the service manifest files, process them and gather the filenames of all UI definitions mentioned for the services which have “enabled” property set to true. Then the repository will create a tar.gz archive containing all these files (preserving the directory structure) and put the archive into the response stream. This call will be made by Conductor. In response, the Metadata Repository will inspect all the service manifest files, process them and gather the filenames of all workflow definitions, heat templates, agent templates and scripts mentioned for the services which have “enabled” property set to true. Then the repository will create a tar.gz archive containing all these files (preserving the directory structure) and put the archive into the response stream.
 * 1) Get UI descriptions of all active services
 * 1) Get conductor’s metadata for all active services

When processing the services for either of these calls, the Repository should act in a service-driven manner, according to the following algorithm:
 * 1) Iterate over the files in data/services directory, open all manifest files, parse them, process (filtering by the manifest version, if needed) their contents, checking for “enabled” flag.
 * 2) For each enabled service create a separate list of the referenced file names
 * 3) For each filename in each list, validate that the file is present in the repository. If at least one file is not present, empty the whole list corresponding to this service.
 * 4) Join the non-empty filelists into a single set of filenames (without duplicates)
 * 5) Built the resulting archive from this set of filenames

For example, let’s say we have 3 enabled services in the repository: Service1, Service2, Service3. Service1 references files A, B, C Service2 references files C, D Service3 references files D, E Let’s say that file B is missing in the repository. In this case, the resulting archive should contain files C, D, E

Server-side caching
When the Metadata Repository creates as tar.gz archive for either of the APIs described above, it may preserve the resulting file. All subsequent calls of the same API may immediately return the same file, without the need to re-create the archive. The file is preserved until the Repository is modified by any of the Administrative operations (see below) - in that case the cached copy is deleted and any subsequent API call will lead to the generation of a new archive.

Client-side caching
The clients of Metadata Repository (Murano Conductor and Murano Dashboard) may also store their copy of the received metadata archive locally. When they need to fetch metadata in the next time, they may compute a hash sum of the existing copy and pass the hash to the Repository as the additional parameter of the appropriate API call. In this case, the Metadata repository will match the received hash sum with the hash of its current archive (either a server-side stored copy, or a newly-generated archive if the copy does not exist). If the hashes match (which means that the client has got the up-to-date copy of the metadata), the API call should return HTTP 304 (Not Modified) status code, which will notify the client that it may reuse its cached copy of the metadata. If the hashes differ, the Metadata Repository will return the archive with the HTTP response (and the HTTP return code should be 200 in this case), so the client may update its local copy.

Metadata administrative operations
The metadata repository will provide administrative APIs to allow modification of the metadata files. This APIs will be grouped by the type of metadata and provide the basic file operations: uploading new files, deleting existing ones, creating subdirectories, deleting directories.

All the operations which modify the repository will be properly synchronized with the metadata querying API calls (i.e. the modification of the repository should not start while the archive creation for some query is in progress, and vice versa)

All the operations will outdate (delete) the appropriate server-side cache of the metadata archive. These cache invalidations should be independent, i.e. uploading a new UI definition should not delete the archive created for conductor, uploading a conductor-related file should not delete the archive created for dashboard. Uploading of service manifest should delete both archives.

There will be also a read-only API to get any given metadata file by path.

Service catalog operations
This is a second-priority feature, as it actually duplicates the functionality provided by other features and is required for convenience only. Also, proper implementation of this functionality may require a database, while everything else may be done on plain filesystem. So, it is proposed to postpone the implementation of this section till the future revisions of the Repository. Because of that, the spec of this feature may change as the metadata format may already be changed by that time.

Metadata Repository will provide APIs to work with the list of Services registered in the Repository. There will be the following APIs: List all the services Returns a list of all Services registered in metadata repository with their primary properties (display name, fully-qualified name, description, author, version). In case of shared files (some files are indeed common for multiple services), some files may already exist in the destination directories. In this case, overwriting should be possible only if the contents of the files are identical. Otherwise, nothing will be added to the repository and the the upload will fail.
 * 1) List metadata objects of a given service
 * 2) Returns the list of files referenced in the manifest file of a given service. The service is identified by fully-qualified name, the files are accompanied by type and the path relative to their type-based directory. The output is sufficient for the client to get the contents of any file by using API described above in section 8.
 * 3) Download the complete service definition.
 * 4) Returns a tar.gz archive with service’s manifest file and all other metadata files referenced in it. The archive will preserve the directory structure according to the types of the files
 * 5) Upload the complete service definition.
 * 6) Uploads a tar.gz archive containing a manifest file of some service and all its references metadata files in the appropriate directories. The archive will be extracted and the files will be placed to the proper directories.

API defined in 4p. should comply to the same rules as APIs defined in p8: its usage should invalidate all caches and should be synchronized with read-operations.

Proposed architecture
The Metadata repository will consist of repository-api component (running as a daemon on a dedicated or shared linux machine and listening to the preconfigured HTTP port) and a web UI (implemented as a plugin for Horizon). At first stage of the development, the metadata will be stored in plain filesystem, without any database. In future modification, a relational database may be introduced to store manifest data, and MogoDB/MongoFS may be used to store the metadata objects.

API definitions
These are formal API definitions. See sections 5, 8 and 9 for detailed API descriptions. APIs should be RESTful, HTTP based, using standard HTTP methods (get, put, post, delete) and returning standard HTTP error codes. All API calls should be made with X-Auth-Token HTTP header containing valid keystone token identifying the user, tenant and user’s role within the tenant APIs should support both HTTP and HTTPS. Examples below will be given for HTTP, for “example.com” host and port 8080. The API should be hosted in the directory “murano-repository” on the host. All APIs should support versioning via URL.

List all the services
To be defined later, see this chapter for a details

List metadata objects of a given service
To be defined later, see this chapter for a details

Download the complete service definition
To be defined later, see this chapter for a details

Upload the complete service definition
To be defined later, see this chapter for a details

Web UI description
A Horizon-application will be used to provide a tenant administrator with the tooling needed to modify the Metadata repository. It will allow to browse the metadata objects of each types, as well as service listing, when section 9 is implemented.

Role-based access control
The simplest way to implement RBAC will be to have isolated Metadata Repository for every tenant. Each tenant will have independent directory created, where all the files of the tenant will be stored, so no tenant can access files of different tenant. Common metadata objects (corresponding to the services which come bundled with Murano) should be present in the directories of each tenant, however this should be done by simply copying the contents from some pre-configured “dist” directory into the newly created repository. Once the contents is copied, all the further modifications are not propagated into the tenant’s copy. The copying should occur on the first access to the repository by the given tenant, by using either API.�

Task breakdown by delivery
Initial (PoC) delivery
 * Proves the concept and provides repository functionality for Conductor. Only client APIs are added, i.e. only the ways to get metadata from the repository are supported. We will put metadata to the repository by manually putting files into the filesystem (scp, ftp, or whatever).
 * The delivery will break to the following tasks:
 * Core services for manipulation with metadata in filesystem, services to parse Service Manifests, create archives etc. Conductor’s client API
 * Modifications in Conductor to fetch metadata from the Repository

Support for UI definitions
 * Adds support for storing definitions of Dynamic UI.
 * Still no easy way for users to upload their own metadata - only manual coping to the filesystem
 * The delivery will break to the following tasks:
 * API implementation for Dashboard
 * Modifications in Dashboard to fetch UI definitions from the Repository

Adding administrative functionality
 * This delivery will allow users to easily upload new metadata into the Repository.
 * This may be first public delivery - it can already be demonstrated as it already defines all the needed features of Metadata Repository, at least in their basic state.
 * The delivery will break to the following tasks:
 * Implementing Administrative APIs
 * Creating web interface to interact with Administrative APIs

Optimizing data transfer
 * This delivery will improve the performance of the Metadata Repository by adding the caching functionality.
 * The delivery will break to the following tasks:
 * Server-side caching
 * Client-side caching

Adding RBAC
 * This delivery will add role-based access control, by isolating metadata storage of different tenants into the different directories.

Implementing batch operations with services
 * The final planned delivery of simple Metadata Repository will add the Service Catalog functionality defined in p9. Most useful features will be ability to dowloan entire service package (p9.3) and upload such package to a different repository