Murano/UnifiedAgent

Introduction
This document defines architecture specification and requirements that Murano Agents must meet. It doesn’t enforce any specific Agent implementation. There can be more than one Agent implementation written in different programming languages for different operating systems and platforms as long as they confirm to this specification.

Murano Agent vNext Architecture has two major improvements over vCurrent:
 * 1) vNext Agents can execute Execution Plans targeting different deployment platforms. vCurrent Agents designed to use PowerShell deployment platform only. This makes possible development Agents for operation systems other than Microsoft Windows
 * 2) vNext Agents support non-liner execution of scripts and functions comprising the Execution Plan. While vCurrent Agent executes them one by one sequentially vNext Agent has ability to control the order of execution using conditional branching and loops. It also gives Execution Plans ability to pass data between scripts and functions (like making output of function 1 go to input of function 2)

While format of Execution Plans for Agents vNext is not compatible with vCurrent format it is designed in a way that makes possible automatic conversion of vCurrent format to vNext. Because of that vNext Agents are required to support vCurrent Execution Plans as long as they support PowerShell execution.

Murano Agent vNext
Murano Agent is an application that is responsible for receiving and execution commands called Execution Plans from remote source (normally from Murano Conductor).

Murano Agent vNext is responsible for:
 * 1) Listening to one or more communication channels waiting for Execution Plan arrival
 * 2) Validates received Execution Plans. Agent must check that it has Executors for all mentioned script types and validate all mandatory parts of Execution Plan are present before starting executing the Plan. If validation fails error result need to be returned to Plan originator.
 * 3) Convert vCurrent Execution Plans to vNext ones
 * 4) Prepare environment and execute Execution Plan script. Agent is responsible with providing Execution Plan script with API to call other scripts mentioned in Execution Plan and service commands of Agent itself.
 * 5) Manage and coordinate work of Executors that do actual script execution
 * 6) Handle API calls from Execution Plan script. Some of the calls may require sending messages to Execution Plan originator
 * 7) Send result of Plan execution back to originator

Murano Agent vNext required to:
 * 1) Support at least one channel of communications. Reference Agent implementation must support at least AMQP(S) communications. All connection and protocol parameters must be configurable via Agent’s config file.
 * 2) Track Execution Plan originator and channel the Plan was received. Plan execution result need to be sent to Plan originator via the same channel it was received. Agent is responsible for response message ID to be the same as request message ID.
 * 3) Handle communication and data error. Agent must not crash or exit in case of connectivity and/or protocol errors (including wrong Execution Plan format)
 * 4) Be fault tolerant. If for any reason Agent exits after Execution Plan script has finished but before its result was sent back Agent must sent them next time it starts before peeking any new Plans. Ideal Agent implementation would be ready for sudden shutdown during Execution Plan script run and resume next time from the last executed point in script. Less ideal implementation will restart script execution. Anyway Agent is required not to lose received  Execution Plans because of unexpected application shutdown.

Deployment Platforms
Deployment Platform is a term used by Murano to refer to various scripting programming languages, configuration management systems and administration tools that can be invoked from Execution Plans. The list of possible Deployment Platforms is open and can grow over time. Agents are not required to support all of them, but the more platforms they support the more powerful they are. At minimum Agents are required to support “Application” Platform. Below is a (partial) list of possible Deployment Platforms:
 * Application. Calls external executable (script) file. Input arguments are converted to command-line string using simple DSL. Most trivial example of such DSL is printf (C/C++) or String.Format (C#) format strings). Execution result is an application exit code. Also other variations of the same Platform may exist like running application and extracting some data from its stdout using regular expressions.
 * PowerShell. Executes PowerShell scripts/functions.
 * Python
 * Bash
 * Bat/CMD
 * Puppet
 * Cheff
 * SaltStack
 * VBScript
 * CShell

Execution Plans
Execution Plans are a minimal unit of execution that can be triggered in Murano Workflows. Each Execution Plan originally belong to some Murano Service (it may be Abstract Service Mixin).

Execution Plans are not a low-level commands (like copy file from A to B) but a script with a semantic that is meaningful to Service user. But the Execution Plan contains various scripts that do perform some low-level actions. Those scripts may be reused across several Execution Plans.

Execution Plans received by Agent in a form of JSON-encoded document with a dictionary as its root element. Agent must verify correctness of that statement before attempting to execute anything.

Below is a list of keys that can be present in that dictionary. All of them are optional.
 * FormatVersion. Current specification define FormatVersion 2.0.0. vCurrent format has the version of 1.0.0. Agent compares versions using SemVer convention. If this FormatVersion attribute is absent or has a value which is less than 2.0.0, Agent must assume that ExecutionPlan is in vCurrent format and convert it to vNext format before further processing. As defined by SemVer major version number change mean incompatible breaking changes. So vNext (v2.0) Agent must not try execute Plans with FormatVersion >= 3.0.0
 * Action. As for this specification all incoming messages must have this attribute equal to “Execute” or the be entirely omitted. Future version may have additional actions (Cconfigure” as a possible example). Response messages will have Action attribute set to “Execute:Result”.
 * Service. This as an ID of the service Execution Plan belongs to. This is to be used for State API for settings namespace isolation. If Service ID is not provided State API must not be available and cause Execution Plan to fail if it uses State API.
 * ID. This is This is an execution plan ID. This ID is generated by the sender and must be globally-unique string. Two messages having the same ID are considered to be equal (and thus duplicates)
 * Name. Human-readable name for the Execution Plan to be used for logging (ThisIsMyExecutionPlanName).
 * Version SemVer version of Execution Plan (default = “0.0.0”). This is the version of Execution Plan itself. Every time Execution Plan content changes (main script, attached scripts, properties etc.) version should be incremented. Version attribute is used for logging and tracing. This is in contrast with FormatVersion which is used to distinguish Execution Plan format (vCurrent, vNext, future formats)
 * Body. This is a string body of Execution Plan script in plain text Python.
 * Parameters. Dictionary of type String->JsonObject that maps parameter names to their value.
 * Scripts. Dictionary that maps script names to script definitions. See below for exact format specification.
 * Files. Dictionary that maps file IDs to file information structure. See below for exact format specification.

Is it highly recommended that Execution Plans would be idempotent (i.e. can be repeated any number of times having the system in the exactly the same state and with the same result each time). Developers can make use of States API for that matter.

Scripts
Scripts are the buildings blocks of Execution Plans. As the name implies those are the scripts for different Deployment Platforms.

Each script may consists of one or more files. Those files are script’s program modules, resource files, configs, certificates etc.

Scripts may be executed as a whole (like a single piece of code), expose some functions that can be independently called in Execution Plan script or both. This depends on Deployment Platform and Executor capabilities.

Scripts are specified using “Scripts” attribute of Execution Plan. This attribute maps script name to a structure (document) that describes the script. It has the following properties:
 * Type: Deployment Platform name that script is targeted to.
 * Version: optional minimum version of deployment platform/executor required by the script.
 * EntryPoint: ID of the file that contains entry point for the script (eg. main file).
 * Files. This is an optional array of additional files (IDs) that are required for the script
 * Options: an optional dictionary of type String->JsonObject that contains additional options for script Executor (see below). If not provided than empty dictionary is assumed.

Type and EntryPoints attributes are mandatory. Execution plan must fail immediately if it contains any scripts without those attributes. The same is also true if Files entry of Execution Plan does not contain any of mentioned script files (entry point or additional files).

Executors
Executors are the program modules that are responsible for executing scripts for specific Deployment Platform. This specification doesn’t enforce any particular way of implementing Executors. They can be implemented as a built-in classes/modules/packages etc, dynamically loaded plugins or even as an out-of-process services. Anyway all executors must have compatible API so that Agent can talk to all Executors using the same protocol.

Here is how scripts are executed:
 * 1) Agent prepares a folder for the script files and puts the script entry point file and all mentioned additional files to that folder (it may be symlinks to the files)
 * 2) Agent chooses appropriate Executor for the script based on a script’s Type attribute
 * 3) Agent asks Executor to load the script file providing its entry point path. This happens once per script even if there are more than one calls to that script in Execution Plan script body.
 * 4) If there is a need to execute script file as a whole or some function containing in it Agent asks Executor to do it passing function name (None if executing script as a whole) and arguments obtained from Execution Plan script
 * 5) Executor executes the script (or function) and returns its result back to the Agent. Agent is blocked during execution (waiting for result).
 * 6) If execution results in error Executor must convert it to a raised exception

There are several methods that can be utilized by Executor in order to do the execution:
 * Embed scripting engine and execute script in own process
 * Call some external RPC API
 * Wrap supplied script in some service code that would talk to executor using some sort of IPC like named pipes and execute the wrapper as a standalone process

Optimal execution method depends on particular Deployment Platform.

It is recommended that script execution timeout could be configured as part of script’s Options entry.

Files
Files is an Execution Plan’s entry that describes files that are passed as part of Execution Plan. This is a dictionary that maps file ID to a document describing the file. It has the following attributes: “Text”. Body attribute contains string content of the file “Base64”. Body attribute contains base64 encoded string content of the (binary) file “ID”. Body attribute is a file ID in Murano Metadata Service “URI”. Body attribute is an absolute file URI
 * Name. Filename. May include slashed to represent files in nested folders.
 * BodyType. One of the following:
 * Body. Contain file data or valid file reference

Smart Agent implementation would (fetch and) store files in some sort of cache upon first access and make use of symlinks to refer to the file from several script folders.

Execution Plan script

This is a simple Python script that orchestrates how scripts are executed. As Python is a dynamic programming language Agent can publish functions to the Python script engine so that the Execution Plan’s script would be represented as a Python functions.

For example if there are 3 scripts in Execution Plan named “script1”, “script2” and “script3” (they all can be of a different type!) than the following could be an example of Execution Plan script:

result = script1( ‘foo’,   args.argument1,   args[‘argument2’],   named_parameter=args.bar) if result: for i in range(0, result): t = script2(i) script3(t)

This demonstrates the very advanced capabilities of Execution Plan script. Usually this would be more like a liner script execution one by one.

The Python scripting engine can be limited in capabilities. Python script should not assume it can import other modules especially those that are not part of Python distribution. It must not rely on being executed on specific Python version or implementation and thus must use only the simplest Python statements that are guarantee to exist in any Python version that can installed on host machine.

If the underlying script executor supports invocation of individual functions within the script then the script objects is used to access them:

script4.scriptFunction

There are also 2 predefined object names that cannot be used as a script name: args that holds Execution Plan arguments (those that are in Parameters entry of Execution Plan). They can be accessed using an attribute or an indexer syntax by their name api to access API functions exposed by Agent itself (see below)

Execution Result
Upon execution end Agent must send Execution Result to Execution Plan originator containing the result. It is a JSON-encoded document with the following attributes: 0 = no error 1 = unknown/internal/generic error, error during script execution 2 = incorrect input (Execution Plan is badly formatted) 3 = unsupported Deployment Platform - Execution Plan has some scripts of unsupported type 4 = SyntaxError, TabError, ImportError, SyntaxError etc. in Execution Plan script 5 = Invalid set of options for one of the scripts 6 = Attempt to access non-existing Execution Plan parameter 7 = Some required files are missing in the Files entry 8 = Error fetching file, IOError while storing the file 9 = Unsupported FormatVersion 10 = timeout occured 100 + X = user error X
 * FormatVersion - execution result format version. If this attribute equals to “1.0.0” or absent then it is assumed that the rest of the document is in vCurrent format. Versions 2.0.0 must be used for format that meets this specification
 * ID. Globally unique message ID generated by Agent
 * SourceID. ID of attribute of Execution Plan that we are sending result of.
 * Action. “Execute:Result” for execution result
 * ErrorCode.
 * Body. JsonObject containing value returned from Execution Plan script or exception details (error message, stack trace, nested exception etc)
 * Time - ISO-8601 timestamp string containing the date/time when result was generated (using client clock)

API
Agent exposes additional API to Execution Plan script via api object so that the script will use api.methodCall(...) code to invoke API methods

Rebooting API

 * api.reboot. Schedules a reboot after Execution Plan finish but before the results would be sent back to the Plan originator
 * api.waitReboot(timeout=0). Blocks script execution until reboot happens (suppose the last invoked script initiated the reboot). Upon reboot the script will continue from this point (or be restarted if Agent implementation does not support continuation)
 * api.expectReboots(count). Tells agent the maximum number of reboots that may happen during script execution before it assumed to be in dead loop.

State API
States are an API to a local agent storage. It is a generic key-value persistent storage where keys are strings and values are any JSON-compatible object. The values are persisted immediately and remain in database until explicit removal. Keys are bound to Service ID so that Execution Plans belonging to different services cannot have key collisions.

State API helps developer persist system state when scripts cannot be made idempotent.


 * api.setState(key, value) - sets state
 * api.getState(key) - gets state or None if it doesn’t exist (or equals to None)
 * api.removeState(key) - removes state if it exists

File API

 * api.putFile(file_id path) - fetches (if needed) file from Files entry and stores (or puts symlink in specified path. Relative path may be used to store it relative to Execution Plan workdir (that would be wiped upon completion)
 * file_id api.addFile(fileInfo) - adds another entry to Files dictionary of Execution Plan

Miscellaneous functions

 * api.version. Returns Agent version
 * api.setExitCode(code). Sets user exit code (X in 100 + X) that would be returned instead of generic error code


 * api.logInfo(object, exception_info=False, notify=True), api.logDebug(object, exception_info=False, notify=True), api.logWarning(object, exception_info=False, notify=True), api.logError(object, exception_info=False, notify=True), api.logFatalError(object, exception_info=True, notify=True) - write a record to local log file and (if notify==True) send log record to Execution Plan originator

Log Message
Log message is a message that Agent sends to (or enqueues for, depending on communication channel) client as a reaction to api.logXXX calls. It has the following format:

{ “FormatVersion”: “2.0.0”, “ID”: “globally-unique message ID”, “SourceID”: “ID of an Execution Plan (optional)”, “Level”: “debug|info|warning|error|fatal”, “Body”: “message text, exception string etc”, “Action”: “log”, “Time”: “ISO-8601 client time of the message/exception”, “Tag”: “optional client tag if needed (contains in config file)” }

Backward compatibility
Agents that support PowerShell Deployment Platform and AMQP communication channel must also support vCurrent Execution Plan format by auto-converting them to vNext format. The result of such Execution Plan must be converted back to vCurrent Execution Result. Here is how it can be done:

vCurrent Execution Plan -> vNext Execution Plan:
 * 1) FormatVersion = “2.0.0”, Action = “execute”, ID = amqp_message_id, Name = “Auto-converted”
 * 2) Walk through all commands and put their parameter into args object using key = command_name + ‘_’ + argument_name
 * 3) Convert vCurrent Scripts to Files document with each file of type “Base64”. Use names like “script{index}.ps1” as a filename
 * 4) Generate script entry-point with just a dot-sourcing of other generated script files. Add it to Files document with a type “Text”
 * 5) Generate single Script entry (let it be named “ps”) with a generated EntryPoint file and all other files mentioned in Files attribute
 * 6) Generate Execution Plan script as the following: for each function name generate statements like ps.functionName(arg1 = args[‘functionName_arg1’], arg2 = args[‘functionName_arg2’]). Put them into try-except block
 * 7) Convert Reboot flag into api.reboot call

vNext Execution Result -> vCurrent Execution Result:

If 1 < ErrorCode < 100 then result would be { “IsException”: true, “Result”: result[‘Body’] }

Else if ErrorCode == 1 or ErrorCode >= 100: { “IsException”: false, “Result”: { “IsException”: true, “Result”: result[‘Body’] } }

Otherwise: { “IsException”: false, “Result”: { “IsException”: false, “Result”: result[‘Body’] } }