Mistral/Blueprints/PoC

Mistral Proof of Concept

We need to create a small PoC project illustrating main Mistral ideas.

Etherpads

• https://etherpad.openstack.org/p/TaskServiceDesign
• https://etherpad.openstack.org/p/MistralDesignAndDependencies

Requirements

Capabilities implemented in the scope of PoC should include:

• Basic Workflow Engine capable of analyzing task graphs, resolving dependencies between tasks and processing them in the right order
• Simple storage (RDBMS for now) access layer to it for storing task graphs and workflow execution states
• Basic DSL for defining tasks, task dependencies, actions and action transports (currently HTTP and AMQP)
• REST API for uploading task graphs and executing workflows
• Necessary test suite to validate the above functionality

Design

Picture 1. Mistral PoC Architecture

The suggested design consists of the components described below. For PoC we suggest running all the components within the same process but design them so that we can easily split them moving forward (arrows on the picture will become network interactions).

Mistral REST API

Mistral API server built with Pecan/WSME. Here's the etherpad that describes the suggested API: https://etherpad.openstack.org/p/MistralAPISpecification

The main capability of the API server is to provide actions for uploading workbooks (description of tasks, actions etc. written in Mistral DSL) and running workflows.

Mistral Python Client

Standard python client that encapsulates all the HTTP protocol interactions with Mistral REST API and provides a convenient python interface to work with Mistral service.

Demo App

Demo App is an application written to demonstrate Mistral PoC capabilities. It will be publishing a workflow into Mistral and have Mistral run it at a certain time (on-demand and on-schedule). Some of the workflow actions may be handled by the application itself (via exposing HTTP endpoints).

The demo workbook definition may look like:

    Service:
       name: Nova
       type: OPENSTACK_SERVICE
       parameters:
           baseUrl:
       actions:
           create-vm:
             parameters:
                 url: servers
                 method: POST
             task-parameters:
                flavor_id:
                  optional: false
                image_id:
                  optional: false    
           delete-vm:  
             .....
    Workflow:
       tasks:
         create-vms:
             action: Nova:create-vm
             parameters:
                image_id: 1234
                flavor_id: 42
         attache-volumes: 
              .....
         backup-vms:
            action:Nova:create-backup
            parameters:
               ....
            dependsOn: [detach-volume]
       events: 
         backup-vm:
            type: periodic
            tasks: backup-vms
            parameters:
                cron-pattern:1 0 * * *

And can be uploaded to Mistral using the following HTTP requests:

POST /workbooks/

{
    "name" : "demo_workbook",
    "description" : "A workbook containing a demo workflow definition for Mistral PoC"
}

PUT /workbooks/demo_workbook/definition

[YAML definition described above]

DSL Parser

Takes a workbook definition written in Mistral DSL creates all the domain model objects so that other components can use them instead of a raw DSL. It can also be responsible for validating definitions according to DSL grammar.

Event Scheduler

This is the component responsible for executing workflows at a right time. Strictly speaking, it's not supposed to execute workflows itself, it should tell Workflow Execution to do that. Event Scheduler only reads necessary information about events from a workbook (DSL) and maintains the queue of upcoming events.

Workflow Execution

The core component of Mistral. It takes commands from other components to run workflows (given a workbook name and a target task) assuming that all necessary objects are stored in DB. When running a workflow Workflow Execution creates one Execution object (see API spec) and multiple Task objects that hold state of corresponding tasks. Execution and Task objects are exposed via REST API.

DB

A storage for Mistral objects like workbooks, tasks, executions, events, actions. For PoC the suggested storage is MySQL. However, moving forward we may want to use something like MongoDB that could be the better fit for storing workbook definitions (DSL). That way, we wouldn't need to create all the objects as relational model and could just keep everything in Mongo and select required objects right from definitions written in Mistral DSL. Currently, we'd like to avoid a dependency like MongoDB to simplify installation and make the project more understandable for most developers.