OpenStack - User contributions [en]

Fuel/Plugins

2016-02-22T16:54:52Z

Adidenko: /* Configuration of Fuel Plugins with new roles */

== What is Pluggable Architecture ==
Beginning with 6.0 version, Fuel features the ability to install plugins along with your environment. Fuel plugins are downloadable software components that enable you to add new capabilities to your environments in a flexible, repeatable and reliable manner. There is no need to install drivers and patches manually after Fuel deploys your cloud – plugins do this for you. 

Fuel plugins enable you to install and configure additional capabilities for your cloud, such as additional storage types and networking functionality. For example, a Load Balancing as a Service (LBaaS) plugin allows you to add network load balancing functionality to your cloud so that incoming traffic can be spread across multiple nodes. Or you might want to use the GlusterFS plugin so that you can use a Gluster file system as a backend for block storage (Cinder). 

Fuel offers an open source framework for creating these plugins, so there’s a wide range of capabilities that you can enable Fuel to add to your OpenStack clouds. If you’re a hardware vendor that has created drivers that enable OpenStack to work with your product, you can create a plugin so Fuel can deploy those drivers when it’s standing up new clouds. Or you might simply want to enable OpenStack functionality that’s not readily available in Fuel. Because Fuel includes a pluggable framework, you’re not limited to what’s provided “out of the box”. 

You are also free to use [[DriverLog#How_To:_Add_a_new_Fuel_Plugin_to_DriverLog|DriverLog]] as the single registry for all Fuel Plugins.

'''Architecture limitations''' 
* Core-functionality Fuel plugins can only be installed before configuring and deploying the environment. Otherwise, you will have to redeploy the environment to enable the core plugin. Apllication level plugins can be installed later on top of already deployed environments.
* Fuel plugins cannot be upgraded from major to another major version.
* Fuel plugins for SDN solutions cannot create a new networking option in the Fuel web UI/UI wizard.

=== 6.1 features ===

* plugins are shipped in RPMs
* plugins support updates from major to minor version (1.0.0 to 1.0.1)

=== 7.0 features ===

* [https://blueprints.launchpad.net/fuel/+spec/role-as-a-plugin define a new role in Fuel via a plugin] see [[Fuel/Plugins#Configuration_of_Fuel_Plugins_with_new_roles|Configuration of Fuel Plugins with new roles]].
* [https://blueprints.launchpad.net/fuel/+spec/vip-reservation reserve VIPs via Fuel plugin's metadata], see [https://wiki.openstack.org/wiki/Fuel/Plugins#Virtual_IP_reservation_via_Fuel_Plugin.27s_metadata| Virtual IP reservation via Fuel plugins's metadata].
* [https://blueprints.launchpad.net/fuel/+spec/ui-plugins-list view installed plugins]
* [https://wiki.openstack.org/wiki/Fuel/Plugins#How_to_separate_services_from_Controller_with_a_plugin separate services from Controller with a plugin]

=== 8.0 features ===

* [https://blueprints.launchpad.net/fuel/+spec/plugin-after-deployment install a plugin after deployment]
* [https://blueprints.launchpad.net/fuel/+spec/external-dashboard-links-in-fuel-dashboard plugin links and metadata in environment dashboard]
* [https://blueprints.launchpad.net/fuel/+spec/component-registry component registry]
* [https://blueprints.launchpad.net/fuel/+spec/segment-settings-tab-logically settings tab plugin assignment]

====Plugin links and metadata in environment dashboard====

You can make an API request via your Puppet manifests and register a non-standard dashboard.

'''Examples'''

* absolute-dashboard-link.pp:
<pre><nowiki>
notice('PLUGIN: fuel_plugin_example_v4/absolute-dashboard-link.pp')

$cluster_id = hiera('deployment_id')
$master_ip = hiera('master_ip')
$network_metadata = hiera_hash('network_metadata', {})
$os_public_vip = $network_metadata['vips']['public']['ipaddr']

$dashboard_name = 'Demo Plugin Dashboard #1'
$dashboard_desc = 'A Sample Absolute Dashboard Link'
$dashboard_link = "http://${os_public_vip}/dashboard"

$json_hash = { title => $dashboard_name,
description => $dashboard_desc,
url => $dashboard_link, }

$json_message = inline_template('<%= require "json"; JSON.dump(@json_hash) %>')

exec { 'create_dashboard_link':
command => "/usr/bin/curl -H 'Content-Type: application/json' -X POST \
-d '${json_message}' \
http://${master_ip}:8000/api/clusters/${cluster_id}/plugin_links",
}

</nowiki></pre>

* relative-dashboard-link.pp:
<pre><nowiki>
notice('PLUGIN: fuel_plugin_example_v4/relative-dashboard-link.pp')

$cluster_id = hiera('deployment_id')
$master_ip = hiera('master_ip')

$dashboard_name = 'Demo Plugin Dashboard #2'
$dashboard_desc = 'A Sample Relative Dashboard Link'
$dashboard_link = "/dashboard"

$json_hash = { title => $dashboard_name,
description => $dashboard_desc,
url => $dashboard_link, }

$json_message = inline_template('<%= require "json"; JSON.dump(@json_hash) %>')

exec { 'create_dashboard_link':
command => "/usr/bin/curl -H 'Content-Type: application/json' -X POST \
-d '${json_message}' \
http://${master_ip}:8000/api/clusters/${cluster_id}/plugin_links",
}

</nowiki></pre>

* deployment_tasks.yaml:
<pre><nowiki>
- id: fuel_plugin_example_v4
type: group
role: [fuel_plugin_example_v4]
tasks:
- hiera
- globals
required_for: [deploy_end]
requires: [deploy_start]
parameters:
strategy:
type: parallel

- id: fuel_plugin_example_v4-controller-deployment
type: puppet
groups: [primary-controller, controller]
required_for: [connectivity_tests, deploy_end]
requires: [netconfig, deploy_start]
parameters:
puppet_manifest: "deploy.pp"
puppet_modules: "."
timeout: 3600

- id: fuel_plugin_example_v4-deployment
type: puppet
groups: [fuel_plugin_example_v4]
required_for: [deploy_end]
requires: [deploy_start]
parameters:
puppet_manifest: "deploy.pp"
puppet_modules: "."
timeout: 3600
retries: 10

- id: fuel_plugin_example_v4-post-deployment-sh
type: shell
role: [fuel_plugin_example_v4]
required_for: [post_deployment_end]
requires: [post_deployment_start]
parameters:
cmd: bash deploy.sh
retries: 3
interval: 20
timeout: 180

- id: fuel_plugin_example_v4-absolute-dashboard-link
type: puppet
role: [fuel_plugin_example_v4]
required_for: [post_deployment_end]
requires: [post_deployment_start]
parameters:
puppet_manifest: "absolute-dashboard-link.pp"
puppet_modules: "/etc/puppet/modules"
timeout: 180

- id: fuel_plugin_example_v4-relative-dashboard-link
type: puppet
role: [fuel_plugin_example_v4]
required_for: [post_deployment_end]
requires: [post_deployment_start]
parameters:
puppet_manifest: "relative-dashboard-link.pp"
puppet_modules: "/etc/puppet/modules"
timeout: 180cluster_id}/plugin_links",
}

</nowiki></pre>

* node_roles.yaml:
<pre><nowiki>
fuel_plugin_example_v4:
name: "Set here the name for the role. This name will be displayed in the Fuel web UI."
description: "Write description for your role"
has_primary: false # whether has primary role or not
public_ip_required: true # whether requires public net or not
weight: 100 # weight that will be used for ordering on fuel ui
</nowiki></pre>

* Example of a plugin API added to a post-install task — post_install.sh:
<pre><nowiki>
#!/bin/sh

PLUGIN_NAME=fuel_plugin_example_v4
DASHBOARD_TITLE=Dashboard
DASHBOARD_DESC="A Sample Dashboard Link"
DASHBOARD_URL="/dashboard"

function obtain_token {

# Request a token for admin user
TENANT_NAME=admin
ADMIN_USERNAME=`python -c "import sys; import yaml; f = open('/etc/fuel/astute.yaml'); astute = yaml.load(f); print astute['FUEL_ACCESS']['user'];"`
ADMIN_PASSWORD=`python -c "import sys; import yaml; f = open('/etc/fuel/astute.yaml'); astute = yaml.load(f); print astute['FUEL_ACCESS']['password'];"`
TENANT_ID=admin

REQUEST="{\"auth\": {\"tenantName\":\"$TENANT_NAME\", \"passwordCredentials\": {\"username\": \"$ADMIN_USERNAME\", \"password\": \"$ADMIN_PASSWORD\"}}}"
RAW_TOKEN=`curl -s -d "$REQUEST" -H "Content-type: application/json" "http://localhost:5000/v2.0/tokens"`
TOKEN=`echo $RAW_TOKEN | python -c "import sys; import json; response = json.loads(sys.stdin.read()); print response['access']['token']['id'];"`

}

function plugin_link {

which fuel > /dev/null
if [[ $? -eq 0 ]]; then

local num_retries=10
local i=0

while true; do
# Fail if number of retries exeeded
if [[ $i -gt $((num_retries + 1)) ]]; then
# Report that plugin not registered
echo "WARNING: Plugin failed to register before the timeout."
echo " Plugin dashboard link will not be added."
return 1
fi

LAST_PLUGIN_ID=`fuel plugins -l | grep $PLUGIN_NAME | cut -d ' ' -f1`
if [ "$LAST_PLUGIN_ID" != "" ]; then
PLUGIN_ID=$LAST_PLUGIN_ID
echo "Plugin ID is: $PLUGIN_ID"
curl -H 'Content-Type: application/json' -H "X-Auth-Token: $TOKEN" -X POST -d \
"{\"title\":\"$DASHBOARD_TITLE\",\"description\":\"$DASHBOARD_DESC\",\"url\":\"$DASHBOARD_URL\"}" \
http://127.0.0.1:8000/api/v1/plugins/$PLUGIN_ID/links
return 0
fi

sleep 1
i=$((i++))
done
fi

}

obtain_token
echo $TOKEN

plugin_link &

</nowiki></pre>
====Component compatibility registry====

Component registry is a component compatibility mechanism in Fuel. See [https://specs.openstack.org/openstack/fuel-specs/specs/8.0/component-registry.html blueprint].

'''Specifying component requirements'''

Consider two components: Component A and Component B.
Component A requires Component B to function.

In a DSL model this relation can be described explicitly:

<pre><nowiki>
- name: 'A'
requires:
- name: 'B'
</nowiki></pre>

The DSL model can be described in the Fuel openstack.yaml file for core components and in the components.yaml file for components provided from a plugin.

In the Fuel web UI, if Component B is not selected, then Component A will be disabled and respective message displayed.

Consider three components: Component A, Component B, and Component C.
Component A requires Component B and Component B to function.

This relation can be described explicitly:

<pre><nowiki>
- name: 'A'
requires:
- name: 'B'
- name: 'C'
message: 'C and B'
</nowiki></pre>

In this case Component A can be selected only when both Component B and Component C are selected.

There is currently a limitation for cases when Component A requires Component B or Component C or more.

Example of requirements for the DVS plugin:

<pre><nowiki>
- name: 'network:neutron:ml2:dvs'
label: 'Neutron with VMware DVS'
description: 'Neutron with VMware DVS ML2 plugin'
requires:
- name: 'network:neutron:core:ml2'
- name: 'hypervisor:vmware'
message: 'The VMware DVS plugin requires vCenter as the hypervisor option.'
</nowiki></pre>

'''Specifying component compatibility'''

Compatibility of Component A with Components B means that Component A has been tested and proved to function with Component B.

DSL model example:

<pre><nowiki>
- name: 'A'
compatible:
- name: 'B'
- name: 'C'
</nowiki></pre>

If either Component B or Component B is selected in the Fuel web UI, then the UI representation of Component A will have a green tooltip.

'''Specifying component incompatibility'''

Incompatibility of Component A with Component B means that Component A has been tested and proved to not function with Component B.

DSL model example:

<pre><nowiki>
- name: 'A'
incompatible:
- name: 'B'
message: 'B incompatible with A'
</nowiki></pre>

Selecting Component A or Component B will disable the other one in the Fuel web UI. You will be able to select only Component A or Component B, but not both.

'''Components with unspecified compatibility'''

The Fuel web UI shows a grey tooltip with a respective message for the components with unspecified compatibility.

====Settings tab plugins assignment====

There is a `groups` attribute in the ''environment_config.yaml'' file that you can use to assign your plugins to.

Plugins can form any group in one of the default groups: General, Security, Compute, Storage, Logging, OpenStack Services.
If your plugin uses a custom group name it will automatically go to the `Other` group.

See [https://github.com/openstack/fuel-plugins/blob/master/fuel_plugin_builder/templates/v4/plugin_data/environment_config.yaml.mako#L2-L5 environment_config.yaml template].

== How to develop a plugin for Fuel ==

=== Planning to create a plugin for Fuel===

==== Entry development requirements ====

When planning to write up a plugin for Fuel, mind the following recommendations:

* Provide deb and rpm packages together with their dependencies. For instructions on creating packages, see [https://www.google.com/url?q=https%3A%2F%2Ffedoraproject.org%2Fwiki%2FHow_to_create_an_RPM_package&sa=D&sntz=1&usg=AFQjCNGKbtd1Lk5xHa3voWMPBjnbspqOrg Fedora project] wiki and [http://packaging.ubuntu.com/html/ Ubuntu Packaging Guide].
* Create puppet manifests according to the Official OpenStack documentation. For nice code examples and workflow, see [[Puppet-openstack|Puppet in OpenStack]].

==== Repo ====

It is recommended that you create a repo for your plugin in /openstack namespace.

'''NOTE:'''
Previously (before September 2015), plugin repos were created in Stackforge namespace. Due to it retirement, all plugin projects are now registered
under /openstack.

Your plugin project should look like ''https://github.com/openstack/fuel-plugin-<your_plugin_name>.''

====== How to create a project ======

# Make sure you're registered at the following resources:
#* LaunchPad. If not, see [http://docs.openstack.org/infra/manual/developers.html the official OpenStack documentation].
#* review.openstack.org. It is required for developers to be included into core and release groups.
# Request the repo creation in Fuel project. Enter [https://launchpad.net/fuel Fuel project at Launchpad] and click '''Report a bug''' link. [[File:report-a-bug.png]]
# In '''Summary''', specify ''Create a Fuel Plugin project in /Openstack''. [[File:bug-report-sum.png]]
# Bug description should consist of:
#* Plugin name (for example, HA Fencing)
#* Plugin functionality overview (for example, enables STONITH-based fencing in HA mode)
#* Developer's contact information (email, skype, etc.) - please make sure core group members are registered at review.openstack.org. Otherwise they wont be added as a core members.
#* List of developers with contact information (name, email on which review.openstack.org is registered) to enter core reviewers group ([https://review.openstack.org/#/admin/groups/691,members example])- used to merge changes
#* List of developers with contact information ( (name, email on which review.openstack.org is registered)) to enter release group ([https://review.openstack.org/#/admin/groups/692,members example]) - used to create release branches and tags in the repo. You will need to review the commit for your project and make sure both lists of developers to enter core and release groups are correct.
# Click '''Extra''' options. In the menu, specify ''devops'' tag. [[File:bug-report-devops.png|200px]]
# After the repo is created (that means, your bug should be marked as ''Fix Committed'' or ''Fix Released'') you can start filling your repo with the required files:
#* Plugin code itself
#* Documentation - see [[Fuel/Plugins#Creating_documentation_for_Fuel_Plugins|the corresponding section]] for more details.

====== Requesting community help for code review ======

Once your code is being prepared and uploaded into repo under ''/openstack'' project, you
are free to request the OpenStack community for review to make sure the code meets the development requirements
and common guidelines for Fuel Plugin Framework as listed below. 
This is aligned with the [http://docs.openstack.org/infra/manual/developers.html Gerrit workflow] used in the OpenStack community.
To request review, you can enter '''#fuel-dev''' channel in IRC.

==== Working with tags and branches ====

To track the release cycle in a more efficient manner, you can use:
* release branches
* tags

Here is the difference between these 2 concepts:
* A tag represents a version of a particular branch at a moment in time.
* A branch represents a separate thread of development that may run concurrently with other development efforts on the same code base. Changes to a branch may eventually be merged back into another branch to unify them.
 
It is recommended that you used branches to let end users build a your plugin themselves.
 
For this, you need to:
* create a corresponding branch.
* edit README file to provide build instructions. For more details, see [[Fuel/Plugins#Creating_documentation_for_Fuel_Plugins|the corresponding section]].
 

You can also use tags for tracking progress within one branch.
Here are 2 examples:
* tagging the plugin repo - [https://github.com/stackforge/fuel-plugin-neutron-vpnaas VPNaaS plugin] repo:
[[File:repo-tags-1.png]]
* creating a separate branch - [https://github.com/stackforge/fuel-plugin-neutron-lbaas LBaaS plugin] repo:
[[File:Plugin-branch.png]]

==== Branches naming convention ====

Since plugins are published in [[Fuel/Plugins#Add_your_plugin_to_DriverLog|DriverLog]], the only way for end users to get the plugin RPM is to build it themselves.
'''Thus, you should take the following into consideration:'''
* the branch name should coincide with the compatible Fuel version.
 
'''SEE ALSO'''
* To learn more about best practices in creating branches, see [http://nvie.com/posts/a-successful-git-branching-model A successful Git branching model]
blogpost.
* For instructions on creating a tag, see the official [http://docs.openstack.org/infra/manual/drivers.html#tagging-a-release Project Driver's Guide].
 
'''NOTE:'''
* You should add a tag once you get all the required files added into the repo and checked
if they contain copyrights. Otherwise, you will have to create new tags.
* You should create tags with gpg key using console gnupg. '''Please, ensure it is present and not expired.'''
 

==== Creating branches ====

There are 2 ways of creating branches:
* from CLI:
<pre><nowiki>
git push <remote> <localref>:<remotebranch>
</nowiki></pre>

where <remote> is the name of your gerrit remote or the full remote url, <localref>
is the refname (could be a branch or something else) and <remotebranch> is the name of the branch you want created from it.

* from the web UI:
*# Make sure you are the core reviewer.
*# Enter review.openstack.org.
*# in Project menu, click Branches. After the project setup opens, enter a new branch name and click Create branch button. Note, that 'Initial revision' field can be left blank.
[[File:Plugin-create-branch-1.png]].

===== Deleting branches =====

If you would like to delete a branch, you have 2 different ways to do that:
# Contact openstack-infra core team via mailing list. See the [http://lists.openstack.org/pipermail/openstack-infra/2015-July/002921.html example request] here.
# Report a bug in Fuel project in Launchpad and assign it to Fuel DevOps team. The flow is similar to the one described [[Fuel/Plugins#How_to_create_a_project|here]]
# Request in #openstack-infra IRC channel on freenode.net. You can contact the following core members there: fungi, clarkb, jeblair, pleia2.
 
Note, that there is no way to delete a branch manually.

==== CI ====

It's recommended to set up CI for your Fuel plugin. The section below provides summary instructions on CI components and their roles.

===== Tools =====

Here is the list of key issues, used in terms of Fuel Plugins CI:

{| class="wikitable"
|-
| CI || [http://en.wikipedia.org/wiki/Continuous_integration Continuous Integration]
|-
| CD || [http://en.wikipedia.org/wiki/Continuous_delivery Continuous Delivery]
|-
| OSTF || [http://docs.mirantis.com/fuel-dev/develop/ostf_contributors_guide.html OpenStack Testing Framework]
|-
| BVT || Build Verification Tests
|-
| JJB || [http://ci.openstack.org/jenkins-job-builder/ Jenkins Job Builder]
|-
| VM || virtual machine
|-
| Fuel || [[Fuel]]
|-
| Fuel CI || [https://ci.fuel-infra.org/ Fuel CI]
|-
| GitHub || [https://github.com GitHub Server]
|-
| Openstack || [https://github.com/openstack Openstack]
|-
| Gerrit || [http://docs.openstack.org/infra/manual/developers.html OpenStack review tool]
|}

Easy way to build own CI https://github.com/openstack/fuel-plugin-ci

As to the tools, the following ones are used:

* GitHub repository with Gerrit code-review tool
* Jenkins CI-server - provides full information about jobs status, scheduler, test results, etc.
* Jenkins Job Builder - tool for easy adding and managing Jenkins jobs. It provides functionality to store jobs’ configuration in a human-readable .yaml markup language and convert them to Jenkins-specific .xml.

===== Steps to configure Gerrit integration =====
# [http://docs.openstack.org/infra/system-config/third_party.html Create and configure] launchpad user for voting as third-party developer
# Add credentials (username, public key) for gerrit plugin configuration in Jenkins
# Send an email to the '''openstack-dev''' mailing list nominating your system for voting permissions

===== Steps to prepare development & testing environments (mandatory) =====
# OpenStack and/or Gerrit repository should be created. See [[#Repo|Repo]] section for more details.
# Preferable quantity of test labs should be allocated (plugin-specific).
# Specific Hardware resources should be installed and configured (plugin-specific).
# Test labs should be configured for setup environments and test running. See [https://docs.mirantis.com/fuel-dev/devops.html Fuel development documentation]

===== Steps to configure CI server (optional) =====
# We recommend for all plugin developers to have their own CI server. It provides better versioning, collecting test-results, deduplicating the same jobs, easier configuration and managing.
# We recommend using Jenkins with [http://ci.openstack.org/jenkins-job-builder/ Jenkins Job Builder plugin] which provides easy jobs management and their configuration storage.
# We recommend to install JJB from git and take example jobs from [[Fuel/Plugins#Example_jobs|the section below]].You may find the main job helpful: this should be plugins.yaml and dependent builders: docs/builders/.sh.
# We recommend creating a pre-commit-hook to check your code:
<pre><nowiki>
#!/bin/bash
# Save this script to <PROJECT>/.git/hooks/pre-review and make it executable
set -e
set -o pipefail

find . -name '*.pp' | xargs -P1 -L1 puppet parser validate --verbose
find . -name '*.pp' | xargs -P1 -L1 puppet-lint \
--fail-on-warnings \
--with-context \
--with-filename \
--no-80chars-check \
--no-variable_scope-check \
--no-nested_classes_or_defines-check \
--no-autoloader_layout-check \
--no-class_inherits_from_params_class-check \
--no-documentation-check \
--no-arrow_alignment-check\
--no-case_without_default-check
find . -name '*.erb' | xargs -P1 -L1 -I '%' erb -P -x -T '-' % | ruby -c
fpb --check ./
</nowiki></pre>

===== Example jobs =====

* deploy-plugin.sh:
<pre><nowiki>
#!/bin/bash
set -ex

export SYSTEM_TESTS="${WORKSPACE}/utils/jenkins/system_tests.sh"
export LOGS_DIR=${WORKSPACE}/logs/${BUILD_NUMBER}
export VENV_PATH='/home/jenkins/venv-nailgun-tests-2.9'
YOUR_PLUGIN_PATH="$(ls ./*rpm)" #Change this to appropriate fuel-qa variable for your plugin
export YOUR_PLUGIN_PATH #

sh -x "${SYSTEM_TESTS}" -w "${WORKSPACE}" -V "${VENV_PATH}" -i "${ISO_PATH}" -t test -o --group="${TEST_GROUP}"
</nowiki></pre>

* prepare_env.sh:
<pre><nowiki>
#!/bin/bash

set -ex

export VENV_PATH="/home/jenkins/venv-nailgun-tests-2.9"

rm -rf "${VENV_PATH}"

REQS_PATH="${WORKSPACE}/fuel-qa/fuelweb_test/requirements.txt"

virtualenv --system-site-packages "${VENV_PATH}"
source "${VENV_PATH}/bin/activate"
pip install -r "${REQS_PATH}" --upgrade
django-admin.py syncdb --settings=devops.settings --noinput
django-admin.py migrate devops --settings=devops.settings --noinput
deactivate
</nowiki></pre>

* syntax-build-plugin.sh
<pre><nowiki>
#!/bin/bash
set -ex

find . -name '*.erb' -print 0 | xargs -0 -P1 -I '%' erb -P -x -T '-' % | ruby -c
find . -name '*.pp' -print 0| xargs -0 -P1 puppet parser validate --verbose
find . -name '*.pp' -print 0| xargs -0 -P1 puppet-lint \
--fail-on-warnings \
--with-context \
--with-filename \
--no-80chars-check \
--no-variable_scope-check \
--no-nested_classes_or_defines-check \
--no-autoloader_layout-check \
--no-class_inherits_from_params_class-check \
--no-documentation-check \
--no-arrow_alignment-check

fpb --check ./
fpb --build ./
</nowiki></pre>

* plugins.yaml:
<pre><nowiki>
- project:
name: plugin_name #Your plugin mame
path_to_fuel_iso: $PWD #Path to FuelISO
plugin_repo: plugin_repo #Your plugin repo name at stackforge
email_to: emails_list #List of emails separated by comma
test_group: test_group #Test group in fuel-qa for deployment tests of your plugin
jobs:
- 'prepare_env'
- '{name}.build'
- '{name}.{dist}.deploy':
dist: 'centos'
- '{name}.{dist}.deploy':
dist: 'ubuntu'

- job-template:
name: 'prepare_env'
builders:
- shell:
!include-raw-escape './builders/prepare_env.sh'
description: 'Prepare environment to testing'
logrotate:
numToKeep: 10
parameters:
- string:
name: 'GERRIT_REFSPEC'
default: 'refs/heads/master'
scm:
- git:
branches:
- $GERRIT_BRANCH
refspec: $GERRIT_REFSPEC
url: 'https://review.openstack.org/stackforge/fuel-qa'
choosing-strategy: gerrit
clean:
before: true
publishers:
- email:
notify-every-unstable-build: true
recipients: '{email_to}'

- job-template:
name: '{name}.build'
builders:
- shell:
!include-raw-escape './builders/syntax-build-plugin.sh'
description: '<a href=https://github.com/stackforge/{plugin_repo}>Build {name} plugin from fuel-plugins project</a>'
logrotate:
numToKeep: 10
parameters:
- string:
name: 'GERRIT_REFSPEC'
default: 'refs/heads/master'
scm:
- git:
branches:
- $GERRIT_BRANCH
name: ''
refspec: $GERRIT_REFSPEC
url: 'https://review.openstack.org/stackforge/{plugin_repo}'
choosing-strategy: gerrit
clean:
before: true
triggers:
- gerrit:
trigger-on:
- patchset-created-event #Trigger plugin build for every gerrit patchset
projects:
- project-compare-type: 'PLAIN'
project-pattern: '{plugin_repo}'
branches:
- branch-compare-type: 'ANT'
branch-pattern: '**'
silent: true
server-name: 'review.openstack.org'
publishers:
- archive:
artifacts: '*.rpm'
- email:
notify-every-unstable-build: true
recipients: '{email_to}'

- job-template:
name: '{name}.{dist}.deploy'
builders:
- copyartifact:
project: '{name}.build'
which-build: last-successful
- inject:
properties-content: |
OPENSTACK_RELEASE={dist}
TEST_GROUP={test_group}
ISO_PATH={path_to_fuel_iso}
- shell:
!include-raw-escape './builders/deploy-plugin.sh'
description: 'fuel-qa system test for {name}'
logrotate:
numToKeep: 10
parameters:
- string:
name: 'GERRIT_REFSPEC'
default: 'refs/heads/master'
scm:
- git:
branches:
- $GERRIT_BRANCH
refspec: $GERRIT_REFSPEC
url: 'https://review.openstack.org/stackforge/fuel-qa'
choosing-strategy: gerrit
clean:
before: true
wipe-workspace: false
publishers:
- archive:
artifacts: 'logs/$BUILD_NUMBER/*'
- email:
notify-every-unstable-build: true
recipients: '{email_to}'
</nowiki></pre>

===== CI/CD Workflow =====

[[File:cicdwf.png]]

 
In terms of a specific plugin, we recommend to go through the following CI pipeline:

# Prepare labs and start or update the lab (when a new Fuel ISO has been built):
#* a) Download current ISO from the [https://ci.fuel-infra.org/ Fuel CI]. Depending on Fuel version specified in plugin’s requirements, Jenkins downloads released ISO(s) and/or currently developed and passed BVT test on core CI. (for now, this should be done manually; in future, API should be provided to inform external CIs about a new stable ISO). You can also implement an internal storage with ability to download the latest stable ISO once it's out.
#* b) Deploy this ISO and prepare the required amount of labs for testing using '''fuel-dev''' and '''fuel-qa''' repositories and running it in console: <code>$ fuel-main/utils/jenkins/system_tests -t test -j dis_fuelweb_test -i (path to downloaded Fuel-ISO) -o --group=setup -V ${VIRTUAL_ENV} -k</code> You can find all the information about script installation and usage in [https://docs.mirantis.com/fuel-dev/devops.html Fuel development documentation]:
#* c) Create/restore the required quantity of empty VMs from snapshots. The script from the previous list item uses ''dos.py'' utility for managing VMs and their snapshots (it was configured and installed on the previous step). You can find all information about dos.py just running <code>dos.py -h</code>.
# Gerrit review job will start to build plugin. See [http://docs.openstack.org/infra/manual/developers.html Gerrit workflow] for more details.
#* a) use preconfigured [https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+Trigger Gerrit Trigger] to start your job after new Gerrit Patch arrives
#* b) run code syntax checker and unit tests according to the instructions from [[Testing]]
#* c) run puppet linter (see [[Puppet-openstack/Development|Puppet OpenStack]] page for more details)
#* d) build plugin (plugin should pass Fuel Plugin Builder requirements)
#* e) trigger plugin testing
# Vote on Gerrit patch’s page and add review result in comment using [[GerritJenkinsGit|Gerrit Trigger]]. (optional)
# Plugin testing (all three steps are part of system_tests.sh runner from [https://review.openstack.org/#/admin/projects/stackforge/fuel-qa fuel-qa] repository) :
#* a) install a plugin
#* b) configure an environment
#* c) deploy environment with inactive plugin
#* d) run OSTF tests.
# Run plugin-specific functional tests to check that current plugin version provides expected functionality.
# Publish resulting aggregated logs to the log storage. You can do it with archiving logs.

===== <big>Automation test cases and test framework</big> =====

You should follow this recommendation on how to write automation tests and configure test framework. Follow the links below for more information:
* [https://github.com/stackforge/fuel-qa tests] for 6.1 Fuel release
* [https://github.com/stackforge/fuel-qa/blob/master/fuelweb_test/tests/plugins/plugin_example/test_fuel_plugin_example.py example of writing plugin test cases]
* [http://docs.mirantis.com/fuel-dev/devops.html instructions on configuring an environment]

'''First of all, you should prepare environment and download Fuel ISO.'''

1. Clone GIT repository:
<pre><nowiki>
git clone https://github.com/stackforge/fuel-qa
</nowiki></pre>
2. Activate virtual env with running:
<pre><nowiki>
source ~/venv-nailgun-tests-2.9/bin/activate
</nowiki></pre>
3. Export Fuel ISO path with running:
<pre><nowiki>
export ISO_PATH=path-to-iso
</nowiki></pre>
4. Enter this folder:
<pre><nowiki>
cd fuel-qa/
</nowiki></pre>
Start tests by running this command:
<pre><nowiki>
./utils/jenkins/system_tests.sh -t test -w $(pwd) -j fuelweb_test -i $ISO_PATH -o --group=setup
</nowiki></pre>
Alternatively you can install empty setup with 1, 3, 5 or 9 slaves for the manual testing:
<pre><nowiki>
fuel-qa$ ./utils/jenkins/system_tests.sh -t test -w $(pwd) -j fuelweb_test -i $ISO_PATH -o --group=prepare_slaves_5
</nowiki></pre>
system_tests file is used as a runner for tests from fuel-qa repository.

5. For more information on how tests work and additional options for test run, read the usage information with running:
<pre><nowiki>
./utils/jenkins/system_tests.sh -h
</nowiki></pre>

'''In the section below, you can find information about main files and modules:'''
* system_tests.sh - a file where tests start execution. This file processes parameters specified from command line and invokes run_tests.py
* run_tests.py - used to import your test files inside this file to run your test then.
* settings.py - contains environment variables used for environment customization. With this file, you can set such variables like path to ISO, nodes quantity, etc.

Models folder with files provides the main logic of the project:
* environment.py - contains methods for environment deploying, virtual machines creation and networking for them, installing Fuel on the Fuel Master node, etc.
Environment creation process uses devops manager fuel-devops/devops/manager.py. Devops manager is used for virtual machines creation and uses lower level libvirt driver. Also environment model contains methods for ssh interaction.
* nailgun_client.py - contains functionality for nailgun handlers, methods and API that are supported by the nailgun client can be found [https://docs.mirantis.com/fuel-dev/develop/nailgun/development/api_doc.html here]. Nailgun client uses http client that is located in helpers folder. Nailgun client is used in fuel web client.

Fuel web client contains such methods as:cluster creation, OSTF tests launch, adding nodes to the cluster, etc.

Helpers folder contains the following files:
* checkers.py - has methods for ssh client to verify nodes access and other.
* common.py - has methods for OpenStack API access, instances creation, etc.
* decorators.py - has different decorators, the most usable is ‘’log_snapshot_on_error’’; it's recommended to use this decorator for all tests, in case of any error diagnostic and environment snapshots will be created.
* os_actions.py - has methods to work with OpenStack.

'''When writing your first test case, please mind the following:'''
* for writing your first test case, you can use ‘’test_fuel_plugin_example.py’’.
* when creating your own test class, you have to inherit this test class from TestBasic class located in ‘’base_test_case.py’’ where fuel web client initialization is performed.
* each test class and method have to be decorated with ‘’@test’’.
* each class in test group has groups to run all test cases together and each test case has groups to separate run.
* test cases have depends_on method or test and it means that this test case does not run until depends_on method or test will be done.

'''Test execution order:'''
# Base test cases are executed: these are the tests that set up environment and install the Fuel Master node.
# After these tests are passed, snapshots are created which will be used by tests for creating clusters.
# Revert to previously created snapshots.
# Set up cluster and deploy it.
# Run Health check test (OSTF).

For test execution debugging you can use dos.py
You can create snapshot with the following command:
<pre><nowiki>
dos.py snapshot <myenv> --snapshot-name=<snapshot_name>
</nowiki></pre>
You can revert snapshot with:
<pre><nowiki>
dos.py revert <myenv> --snapshot-name=<snapshot_name>
</nowiki></pre>

==== Fuel-qa and Fuel Plugins ====
Currently, the system tests for Fuel are kept in [https://github.com/stackforge/fuel-qa fuel-qa] repo.
Note that for implementing Fuel Plugin CI, the fuel-qa can be used as the baseline framework.
This means, you can use the framework without committing any tests directly to fuel-qa repo.

=== Preparing an environment for plugin development ===

Prepare your environment for plugin development in three easy steps:

1. Install the standard Linux development tools.

* For Ubuntu 14.04 LTS, run:
''sudo apt-get install createrepo rpm dpkg-dev''

* For Centos 6.5, run:
''yum install createrepo rpm rpm-build dpkg-devel''

2. Install the Fuel Plugin Builder. To do that, you should first get pip:
''easy_install pip''

3. Then, install Fuel Plugin Builder (fpb) itself:
''pip install fuel-plugin-builder''

If you need to install the latest version of the Fuel Plugin Builder, follow the instruction below:

1. Clone the repository:
"git clone https://github.com/stackforge/fuel-plugins.git"

2. Go to the 'fuel_plugin_builder' folder:
"cd fuel-plugins/fuel_plugin_builder/"

3. Install the fpb:
"sudo python setup.py develop"

=== Using Fuel Plugin Builder tool ===

==== Plugin structure ====
To build your plugin, you should first generate its structure. It looks as follows:

[[File:Untitled drawing-3.png|500px]]

===== Generating the structure and building the plugin =====

So, to generate the plugin structure as given above, you should run the following command:

''fpb --create <fuel_plugin_name>''

As the result, you will only have to build your plugin:
''fpb --build <fuel_plugin_name>''

After your plugin is built, you can see it in your plugin's directory; for example, ''fuel_plugin_name/fuel_plugin_name-1.0.0.noarch.rpm".

=== How to use files from plugin structure ===

==== deployment_tasks.yaml====

New "deployment_tasks.yaml" file was introduced to replace the "tasks.yaml". The new file can specify tasks dependencies using new parameters "required for" and "requires".
<pre><nowiki>
type: puppet
groups: [primary-controller]
required_for: [keystone]
requires: [database]
parameters:
puppet_manifest: /etc/puppet/modules/osnailyfacter/modular/keystone/db.pp
puppet_modules: /etc/puppet/modules
timeout: 1800
</nowiki></pre>

The list of parameters for "deployment_tasks.yaml":

==== role ====
The parameter describes a role of a node where tasks will be executed. Usually, the role parameter is used for pre/post deployment tasks or for declaring a group for main deployment.

==== groups ====
The parameter describes a group of nodes with the specified role where tasks will be executed and should be explicitly declared for the main deployment.
<pre><nowiki>
- id: controller
type: group
role: [controller]
requires: [primary-controller]
required_for: [deploy_end]
parameters:
strategy:
type: parallel
amount: 6
</nowiki></pre>

The “groups” parameter is usually used for the main deployment. It conflicts with the "role" parameter. The task must have either a “role” or “group” parameter, but not both of them at the same time.

==== requires ====
The parameter specifies the list of tasks needed by the current one. The list of tasks can be obtained by running the command:
<code>fuel graph --env 1 --download</code>

==== required_for ====
The parameter specifies the list of tasks for which the current one is needed.

==== timeout ====
You can also specify execution timeout in seconds. Once specified, the deployment will fail if timeout expires. By default, timeout is set to 300 seconds.
 
Previously, when plugin developer set timeout for operation, this timeout worked differently for specific operations. For example:

* shell task type timeout = timeout * retries (Mcollective retries, 2 by default)
* puppet task type timeout = global timeout (the one set by plugin developer).
 
Now it works properly in both cases: shell and puppet task types have global timeout.

==== type: shell parameter ====
The parameter runs the specified shell command, the second applies Puppet manifests.

Here is the example of a "shell" task:

<pre><nowiki>
# This tasks will be applied on controller nodes,
# here you can also specify several roles, for example
# ['cinder', 'compute'] will be applied only on
# cinder and compute nodes
- id: task-shell-deploy
role: ['controller']
type: shell
parameters:
cmd: bash deploy.sh
timeout: 42

- id: task-shell-deploy
role: ['cinder','compute'']
type: shell
parameters:
cmd: bash deploy.sh
timeout: 42

# Task is applied for all roles
- id: task-shell-pluginlog
role: '*'
type: shell
parameters:
cmd: echo all > /tmp/plugin.all
timeout: 42
</nowiki></pre>

==== type: puppet ====
Puppet task type allows you to apply your own Puppet manifests on OpenStack nodes. For more information, see Puppet in Fuel section.

To enable this task type, add your ''site.pp'' file in ''deployment_scripts/puppet/manifests/'' directory.
Then put all required modules in ''deployment_scripts/puppet/modules'' directory.

* puppet_manifest - specify directory path for your manifest relative to deployment_scripts.
* puppet_modules - specify directory path for your modules relative to deployment_scripts.

<pre><nowiki>
# Deployment will be applied on controllers only
- role: ['controller']
type: puppet
parameters:
puppet_manifest: puppet/manifests/site.pp
puppet_modules: puppet/modules
timeout: 360
</nowiki></pre>

==== type: reboot ====
Beginning with Fuel 6.1 release for plugins with '''package_version: 2.0.0''', reboot task type allows you to reboot your node with specifying the timeout. This can be useful to apply numerous changes at the node.

<pre><nowiki>
- role: '*'
type: reboot
parameters:
timeout: 300
</nowiki></pre>

==== type: group ====
A group task consists of the list of tasks to be executed on the specified nodes.

<pre><nowiki>
- id: standalone-keystone
type: group
role: [standalone-keystone]
requires: [deploy_start, primary-standalone-keystone]
required_for: [deploy_end]
tasks: [fuel_pkgs, hiera, globals, tools, logging, netconfig, hosts, firewall, deploy_start, cluster, keystone-vip, cluster-haproxy, memcached, openstack-haproxy-stats, task-keystone]
parameters:
strategy:
type: parallel
</nowiki></pre>

When you set up a group of tasks you can also specify how they will be executed: in “parallel” or “one-by-one”.

==== strategy: type ====

* "parallel" - tasks will be executed in parallel
* "one-by-one" - tasks will be executed one-by-one

Once you choose “parallel” you can specify the maximal number of tasks that can be run in parallel using the “amount” parameter.

<pre><nowiki>
- id: controller
type: group
role: [controller]
requires: [primary-controller]
required_for: [deploy_end]
parameters:
strategy:
type: parallel
amount: 6
</nowiki></pre>

==== environment_config.yaml ====

This file describes additional attributes that will appear on the Settings tab of the Fuel web UI.
When the environment is deployed, these attributes are passed to the task executor so that the data is available in the /etc/astute.yaml file
on each target node and can be accessed from your bash or puppet scripts.

By default, your environment_config.yaml file adds text field on Fuel web UI:

<pre><nowiki>
attributes:
fuel_plugin_name_text:
value: 'Set default value'
label: 'Text field'
description: 'Description for text field'
weight: 25
type: "text"
</nowiki></pre>

For more information on Fuel web UI elements for a plugin, see Fuel plugin UI elements.

==== metadata.yaml ====

This file contains the description of your plugin:

<pre><nowiki>
# Plugin name
name: fuel_plugin_name
# Human-readable name for your plugin, it will be shown on UI
# as a name of plugin group
title: Title for fuel_plugin_name plugin
# Plugin version
version: 1.0.0
# Description
description: Enable to use plugin X
# Required fuel version
fuel_version: ['6.0']
# The plugin is compatible with releases in the list
releases:
- os: ubuntu
version: 2014.2-6.0
mode: ['ha', 'multinode']
deployment_scripts_path: deployment_scripts/
repository_path: repositories/ubuntu
- os: centos
version: 2014.2-6.0
mode: ['ha', 'multinode']
deployment_scripts_path: deployment_scripts/
repository_path: repositories/centos
# Version of plugin package
package_version: '1.0.0'
</nowiki></pre>

{| class="wikitable"
|-
! Parameter !! Usage !! Comments/Example
|-
| name || Internal name for your plugin. ||Name can consist of lowercase letters, '-' and '_' symbols.
|-
| title || Human-readable name for the plugin that will appear on the Fuel web UI. ||
|-
| description || Description of your plugin. || For example: Enables X functionality for nodes with Controller role.
|-
| version || Plugin version. || For the guidelines, see [http://semver.org/ Semantic Versioning 2.0.0.]
|-
| fuel_version || A list of plugin-compatible versions of Fuel. || For example, 2014.2-6.0.
|-
| package_version || version of plugin; Fuel uses this version to choose the way a plugin should be installed. || Example
|-
| is_hotpluggable || Set this parameter to 'true' to enable installation of a plugin on top of already deployed environment. Use this parameter only with application level plugins that have no core functionality. Also, the plugin must define a role that Fuel can apply to a new node, which has not already been provisioned. This role can be co-located with another role as long as they are being provisioned onto a new node but not the one that was previously provisioned or deployed. || See the [https://github.com/openstack/fuel-plugins/blob/master/examples/fuel_plugin_example_v4/metadata.yaml#L22 template] with the parameter set to 'false' by default.
|-
| releases || a list of OpenStack releases compatible with the plugin. || For example, 2014.2-6.0.
|-
| os || a name of supported Linux distribution || For example, Ubuntu or CentOSe
|-
| version || A version of OpenStack release ||
|-
| mode || A list plugin-compatible modes. || 'ha' is used if plugin supports High Availability;’'multinode' - if it does not.
|-
| deployment_scripts_path || A path in your plugin directory where all deployment scripts for the release are located relative to the top of the plugin directory. ||
|-
| repository_path || A path in your plugin directory where all packages for the release are located relative to the top of the plugin directory. || Example
|-
|}

=== Plugins deployment order ===

Beginning with Fuel 6.1 release, you can specify the order in which plugins are deployed.
This is especially useful if several plugins should be enabled in one environment.
For example, plugins for network configuration should be run before plugins installing software services.
For each stage name plugin developer adds a postfix, which defines the stage of
specific execution order of the task.
Let's have a look at the following sample:

The ''tasks.yam''l file of Fuel plugin '''A''':
 
<pre><nowiki>
role: ['primary-controller', 'controller']
stage: post_deployment/100
type: shell
parameters:
cmd: bash deploy.sh
timeout: 42</nowiki></pre>

The ''tasks.yaml'' file of Fuel plugin '''B''':
 
<pre><nowiki>
role: ['primary-controller', 'controller']
stage: post_deployment/50
type: shell
parameters:
cmd: bash deploy.sh
timeout: 42
</nowiki></pre>
 
During post_deployment stage execution, the task of plugin '''B'''
will be run before plugin post task of plugin '''A''', because
''post_deployment/50'' is lower than ''post_deployment/100''.
Nevertheless, in some cases plugins do not know about each other so the best
way to solve the problem is to define the convention of ranges which
plugin developers will be able to use:

{| class="wikitable"
|-
| 0 - 999 || hardware configuration, for example drivers configuration
|-
| 1000 - 1999 || reserved for future uses
|-
| 2000 - 2999 || disks partitioning and volumes configuration
|-
| 3000 - 3999 || reserved for future uses
|-
| 4000 - 4999 || network configuration
|-
| 5000 - 5999 || reserved for future uses
|-
| 6000 - 6999 || software deployment
|-
| 7000 - 7999 || reserved for future uses
|-
| 8000 - 8999 || monitoring services deployment
|}

==== How the deployment order works in specific cases ====

* If one network plugin defines stage: ''post_deployment/100''
and another one has stage: ''post_deployment/2000'', they will be
installed in the right order without knowing about each other.
* If there are two plugins which implement monitoring, plugin developers
can figure out which plugin should be installed first and tune postfixes
accordingly.
* If two tasks have the same priority, they should be sorted in alphabetical
order by name and the first in the list should be deployed first.
* If several tasks with the same postfix priority are present in a single plugin,
then they should be deployed in the same order in which they specified in the
file.
* Postfix can be negative or positive, floating or integer number.

==== Additional stages ====

Additional plugin-specific stages can be defined:
* hw_configuration
* disk_partitioning
* network_configuration
* software_installation

With the already existing stages:
* pre_deployment
* post_deployment
 
And, finally, a new stage:
* monitoring

In this case, plugin developer will be able to work with a single entity
without some additional postfixes.

=== How to display plugin restrictions to users ===

Sometimes you might need to provide restrictions for your plugin
in the Fuel UI. That means, all incompatible options (e.g. Networking Setup)
should be somehow grayed out.

==== What are restrictions? ====

Restrictions define when settings and setting groups should be available.
Each restriction is defined as a condition with optional action and message:
<pre><nowiki>
restrictions:
- condition: "settings:common.libvirt_type.value != 'kvm'"
message: "KVM only is supported"
- condition: "not ('experimental' in version:feature_groups)"
action: hide
</nowiki></pre>

* condition is an expression written in [https://github.com/stackforge/fuel-web/blob/master/docs/develop/nailgun/customization/settings.rst#expression-dsl Expression DSL]. If returned value is true, then action is performed and message is shown (if specified). 
* action defines what to do if condition is satisfied. Supported values are ''"disable"'', ''"hide"'' and ''"none"''. The ''"none"'' value can be used just to display message. This field is optional (the default value is ''"disable"''): 
** disable - plugin checkbox (placed into the ''Settings'' tab of the Fuel web UI) turns inactive and has a yellow triangle with a pop-up warning (when hovering a mouse).
** hide - when incorrect parameters are selected, the plugin checkbox will not appear in the ''Settings'' tab of the Fuel web UI at all.
** none - plugin checkbox stays active and has a yellow triangle with a warning (when hovering a mouse).
* message is a message that is displayed if the condition is satisfied. This field is optional. 
* strict is a boolean flag which specifies how to handle non-existent keys in expressions. If it is set to true (default value), exception is thrown in case of non-existent key. Otherwise values of such keys have null value. Setting this flag to false is useful for conditions which rely on settings provided by plugins:
<pre><nowiki>
restrictions:
- condition: "settings:other_plugin == null or settings:other_plugin.metadata.enabled != true"
strict: false
message: "Other plugin must be installed and enabled"
</nowiki></pre>

There are also short forms of restrictions:

<pre><nowiki>
restrictions:
- "settings:common.libvirt_type.value != 'kvm'": "KVM only is supported"
- "settings:storage.volumes_ceph.value == true"
</nowiki></pre>

===== How to add restrictions to plugin-related fields and checkbox =====

Note, that you can not only add restrictions to the checkbox, but also to the plugin-related fields.
Here is the [https://github.com/stackforge/fuel-plugin-contrail/blob/master/environment_config.yaml#L3 example implementation]:
<pre><nowiki>
attributes:
# Show contrail only in supported network config
metadata:
restrictions:
- condition: "not (cluster:net_provider == 'neutron' and networking_parameters:segmentation_type == 'vlan')"
message: "Please use Neutron with VLAN segmentation, the only network type supported with Contrail plugin."
contrail_asnum:
value: '64512'
label: 'AS Number'
description: 'AS number for BGP communication'
weight: 10
type: "text"
regex:
source: '^(?:(6553[0-5])|(655[0-2]\d)|(65[0-4]\d{2})|(6[0-4]\d{3})|([1-5]\d{4})|([1-9]\d{1,3})|([1-9]))$'
error: "Invalid AS number"
contrail_private_cidr:
value: '10.109.3.0/24'
label: 'Private network CIDR'
description: 'CIDR for private network used in Contrail inter-node communication'
weight: 20
type: "text"
regex:
source: '^(?:\d|1?\d\d|2[0-4]\d|25[0-5])(?:\.(?:\d|1?\d\d|2[0-4]\d|25[0-5])){3}(?:\/(?:[1-2]\d|[8-9]))$'
error: "Invalid network CIDR"
</nowiki></pre>
 
Such a restriction will be then displayed in the Fuel web UI as follows:
[[File:Plugin-restriction.png]]

==== Example implementation ====

Here is the [https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/fixtures/openstack.yaml#L511-L525 example implementation] of the restriction:

<pre><nowiki>
- data: "disabled"
label: "Mellanox drivers and plugins disabled"
description: "If selected, Mellanox drivers, Neutron and Cinder plugin will not be installed."
restrictions:
- "settings:storage.iser.value == true"
- data: "drivers_only"
label: "Install only Mellanox drivers"
description: "If selected, Mellanox Ethernet drivers will be installed to support networking over Mellanox NIC. Mellanox Neutron plugin will not be installed."
restrictions:
- "settings:common.libvirt_type.value != 'kvm'"
- data: "ethernet"
label: "Install Mellanox drivers and SR-IOV plugin"
description: "If selected, both Mellanox Ethernet drivers and Mellanox network acceleration (Neutron) plugin will be installed."
restrictions:
- "settings:common.libvirt_type.value != 'kvm' or not (cluster:net_provider == 'neutron' and networking_parameters:segmentation_type == 'vlan')"
</nowiki></pre>

== How to migrate plugins from 1.0.0 to 2.0.0 package version ==

Beginning with Fuel 6.1, new plugins format is supported. Note, that new format is not compatible with Fuel 6.0.

For new plugins, Fuel Plugin Builder builds RPM packages instead of fuel plugin archives.

In order to migrate from old format to new, follow these steps:
* Get the latest fuel plugin builder version, 2.0.0 or higher.
<pre><nowiki>
pip install fuel-plugin-builder
</nowiki></pre>
* Change the value of '''package_version''' parameter in '''metadata.yaml''' file from '''1.0.0''' to '''2.0.0'''.
* Run the following command:
<pre><nowiki>fpb --check plugin_path</nowiki></pre>
and fix the errors one by one or follow the instructions below.

=== Updates ===

If your plugin uses "controller" role in '''tasks.yaml''' file, make sure that you have also specified the "primary-controller".
In new plugins, "controller" and "primary-controller" should be defined explicitly. In previous version, you could indicate "controller", and then the "primary-controller" was added at the backend automatically.

=== Brand new features ===

Several obligatory fields are added to '''metadata.yaml''' file:
* '''groups''' field is used to specify group which your plugin belongs to. Either of the options is available:
** network
** storage
** storage::cinder
** storage::glance
** hypervisor
** monitoring.
If your plugin does not belong to any of these options, set an empty list as a value for "groups" parameter.
* '''authors''' field provides the list of authors. Here you should specify your or your company's name.
'''Note:''' No commas should be used in '''authors''' field
don't:
authors: ['Vyacheslav Struk, Mirantis', 'Oleksandr Martsyniuk, Mirantis']
do:
authors: ['Vyacheslav Struk', 'Oleksandr Martsyniuk']
* '''licenses''' field contains the list of licenses.
* '''homepage''' field sets a link to plugin's project.
 See the [https://github.com/stackforge/fuel-plugin-contrail/blob/master/metadata.yaml#L25 Contrail plugin] ''metadata.yaml'' file for example.

 
One more task type is introduced: the '''reboot''' task is useful if you perform node configuration which requires reboot; for example, in case of linux kernel parameters configuration.
For information about reboot task type, see [[Fuel/Plugins#type:_reboot_parameter|the Plugins wiki]].

== Plugin versioning system ==

When new functionality, minor updates or security fixes should be delivered, plugin developer creates a new version of the plugin; this can be major or minor one:
* major - changes in API, functionality, major OpenStack release introduced.
* minor - security fixes only.
 
In 6.0, plugins had .fp format only. It’s deprecated now.
In 6.1 and higher, only RPM plugins are used. Their versioning is performed as follows:

{| class="wikitable"
|-
! !! Plugin file format !! fuel-plugin value !! metadata.yaml !! major !! minor
|-
| RPM || fuel-plugin-1.0-1.0.0 || 1.0 || 1.0.0 || 1.0.0 || 1.0.1
|}

Here is the example:
Let's suppose that a plugin has 1.0.1 version in the metadata.yaml file.
The plugin file should then have a different format: plugin-1.0-1.0.1.rpm.

=== Update procedure ===
{| class="wikitable"
|-
! !! Update !! Limitations
|-
| fp || NO ||
|-
| RPM || YES || Can be updated to minor version only with <code>fuel plugins --update <fuel-plugin-file></code> command. To get a major one, user has to download it from Fuel Plugins Catalog and create a new environment from scratch.
|}

=== Versioning scheme ===
* for .fp plugins versioning is not supported at all. That means, user has to download&install the plugin from scratch.
* for RPM plugins, it looks as follows:
[[File:Rpm plugin versioning.png]]

=== Important note ===
Please, consider changing the versioning scheme for customized packages
to have clear indicator which package is installed - the ones that enter Mirantis OpenStack or customized ones.

Otherwise, there is need to check python files to understand which package is actually installed.

== How it works from the inside ==

=== Installation ===

Installation procedure consists of the following steps:

# User copies fuel_plugin_name-1.0-1.0.0-1.noarch.rpm file on the Fuel Master node with secure copy.
# Then, after it’s copied to the Fuel Master node, the user runs the following command: <code>fuel plugins --install fuel_plugin_name-1.0-1.0.0-1.noarch.rpm</code>
# Fuel client installs the contents of the fuel_plugin_name-1.0-1.0.0-1.noarch.rpm package to the ''/var/www/nailgun/plugins/fuel_plugin_name-1.0''.
# Fuel client registers the plugin using REST API Service (Nailgun); it sends a POST request with the contents of metadata.yaml file to /api/v1/plugins url.

=== Configuration ===

Configuration procedure consists of the following steps:

# When a new environment is created, Nailgun tries to find plugins which are compatible with the environment.
# Nailgun merges the contents of the environment_config.yaml files with the basic attributes of the environment and generates a separate group and the checkbox on the Fuel web UI for each plugin.
# The plugin is disabled until the user enables it. After user selected the plugin checkbox, the Fuel web UI sends the data to Nailgun. Nailgun parses the request and creates relations between Plugin and Cluster models.

====Prioritization of Repositories====

You can use priorities to configure plugin’s repositories instead of shutting them down completely, which may lead to the failed deployment.
You can specify the priorities for plugin's repositories in Nailgun’s [https://github.com/stackforge/fuel-web/blob/master/nailgun/nailgun/settings.yaml#L88-L95 settings.yaml].
Default priorities for Plugins' repos look like:
<pre><nowiki>
REPO_PRIORITIES:
plugins:
centos: 10
ubuntu: 1100
</nowiki></pre>

These priorities should be higher than OS/Fuel one, because a user may want to override some package from OS/Fuel.

====Virtual IP reservation via Fuel Plugin's metadata====

Some plugins require an additional VIP to enable proper configuration. Previously, VIP reservation was based on network metadata. Now, it is based on the network roles' description. This enables a plugin developer to create extra VIPs to be used in developer's deployment scripts.

First, a user should define VIPs in the plugin metadata. Then, install a plugin before creating an environment. In upcoming releases, the requirement to install a plugin prior to environment creation will be removed.

For example, Zabbix can be configured in a way that it receives SNMP traffic on a dedicated VIP. In that case, a plugin developer can define extra VIPs in a plugin configuration file, and use it as a puppet resource.

VIP reservation is possible only via plugin metadata. This is done by adding a new file ‘network_roles.yaml’, which looks like this:
<pre><nowiki>
- id: "name_of_network_role"
default_mapping: "public"
properties:
subnet: true
gateway: false
vip:
- name: "test_a"
alias: "alias_name"
namespace: "haproxy"
node_roles: ["primary-controller", "controller"]
- name: "test_b"
</nowiki></pre>

Note that 'alias', 'namespace', and 'node_roles' parameters are optional.

* 'default_mapping' - a name of a network to map the network role by default. So, VIP will be allocated on that network by default (that mapping can be changed in the network template). Thus, several network roles should be defined if VIPs should be allocated in different networks.

* 'name' - a string that contains a unique name within the environment used in the Nailgun database and for serialization in the orchestrator.
'''Note:''' Names for network interfaces are generated automatically using first 13 characters of a VIP’s name. Therefore, if several VIPs are introduced by the same plugin, their names must differ in first 13 characters. Also, do not use the word ‘vip’ in a VIP’s name, as it will be added automatically where necessary.

* 'alias' - a string used to solve a compatibility issue with REST API to run tests within the OpenStack Testing Framework that use the old notation.

* 'namespace' - a string that points to a network namespace to be used for landing of the VIP, null if not defined.
'''Note:''' You must specify a namespace for a VIP, otherwise it still will be put into network_metadata['vips'] in Hiera, where all available VIPs are stored, but not handled by Pacemaker. So, a plugin can use it in its own way.

* 'node_roles' - a list of node roles where VIPs should be set up. If not defined, its value will be set to ["primary-controller", "controller"].
'''Note:''' A node must have Pacemaker installed that automatically allocates resources for all VIPs. In Fuel 6.1, VIPs were processed manually.

'''Known issue:'''
It should be mentioned, that only new network roles can be requested in a plugin. Previously defined core network roles cannot be redefined/extended in a plugin.
For more information, see [https://bugs.launchpad.net/fuel/+bug/1487011 LP1487011].

====Configuration of Fuel Plugins with new roles====

Beginning with Fuel 7.0, after adding and enabling custom plugins a user can define a new role described in the plugins via Web UI as well as via Fuel CLI. In such case, in the '''Settings''' tab you can select a ''plugin'' role from a ''roles'' list in the '''Nodes''' tab and attach it to specific nodes. And vice versa, it should not be displayed in the roles list when the plugin is disabled for the cluster (environment).

If you want to disable a plugin, but there are nodes with this plugin’s role in a cluster, then follow the existing mechanism: in the '''Nodes''' tab remove the plugin’s role from all the nodes and then disable the plugin in the '''Settings''' tab.

'''NOTE:'''
When a cluster is deployed, you cannot disable a plugin and, as a result, remove plugin’s role(s) from nodes.

New node’s roles with a volume’s partition and tasks info can be described in config yaml files that will be integrated in Nailgun. The Fuel plugin builder should automatically create a new node role based on a plugin name in a yaml file.

The basic skeleton describing a node’s role in the ‘node_roles’ yaml file:

<pre><nowiki>
role_name:
name: "Some plugin role"
description: "Some description"
conflicts:
- some_not_compatible_role
limits:
min: 1
restrictions:
- condition: "some logic condition"
message: "Some message for restriction warning"
</nowiki></pre>

Description of the volumes’ partition in ‘volumes’ yaml file:

<pre><nowiki>
volumes:
- id: "role_volume_name"
type: "vg"
min_size: {generator: "calc_min_os_size"}
label: "Role specific volume"
items:
- mount: "/"
type: "lv"
name: "root"
size: {generator: "calc_total_root_vg"}
file_system: "ext4"
- mount: "swap"
type: "lv"
name: "swap"
size: {generator: "calc_swap_size"}
file_system: "swap"
volumes_roles_mapping:
role_name:
- {allocate_size: "min", id: "os"}
- {allocate_size: "all", id: "role_volume_name"}
</nowiki></pre>

Previously, pre/post deployment tasks were kept only in tasks.yaml, which does not allow specification of tasks' dependencies. Now, we have a possibility to describe deployment tasks with tasks dependencies in the deployment_tasks.yaml file. The old tasks.yaml is still supported, however it is recommended to use the new deployment_tasks.yaml.

Description of a new group in ‘deployment_tasks.yaml’:

<pre><nowiki>
- id: role-name
type: group
role: [role-name]
requires: [controller]
required_for: [deploy_end]
parameters:
strategy:
type: parallel
</nowiki></pre>

In metadata for a plugin’s role a developer can describe conflicts with other roles such as already present in ‘openstack.yaml’. Each plugin should describe the list of provided roles for proper name referencing.

User can declare several new node roles in one plugin. It can be useful for tasks order and provide granular way for a plugin developer to build their own plugins on top of others.

'''NOTE:'''
Plugins with old format also will be supported.

'''NOTE:'''
The role '*' is supported for deployment tasks. Also a list of tasks' names can be specified in a role's field.

=== Deployment ===

After environment is created and configured, user starts a deployment.
Meanwhile, Nailgun gets the list of enabled plugins from the database. For each plugin from the list, Nailgun parses tasks.yaml file:

<pre><nowiki>
- role: ['controller']
stage: post_deployment
type: shell
parameters:
cmd: bash deploy.sh
timeout: 42
- role: '*'
stage: pre_deployment
type: shell
parameters:
cmd: echo all > /tmp/plugin.all
timeout: 42
</nowiki></pre>

For example, we have a two-node environment deployed. A node has a Controller role with UID 7 and Compute role with UID 8. In this case, the task executor generates the following tasks:

<pre><nowiki>
{
"pre_deployment": [
{
"uids": ["8", "7"],
"parameters": {
"path": "/etc/apt/sources.list.d/fuel_plugin_name-1.0.0.list",
"data": "deb http://10.20.0.2:8080/plugins/
fuel_plugin_name-1.0.0/repositories/ubuntu /"
},
"priority": 100,
"fail_on_error": true,
"type": "upload_file",
"diagnostic_name": "fuel_plugin_name-1.0.0"
},
{
"uids": ["8", "7"],
"parameters": {
"src": "rsync://10.20.0.2:/plugins/fuel_plugin_name-1.0.0/deployment_scripts/",
"dst": "/etc/fuel/plugins/fuel_plugin_name-1.0.0/"
},
"priority": 200,
"fail_on_error": true,
"type": "sync",
"diagnostic_name": "fuel_plugin_name-1.0.0"
},
{
"uids": ["8", "7"],
"parameters": {
"cmd": "echo all > /tmp/plugin.all",
"cwd": "/etc/fuel/plugins/fuel_plugin_name-1.0.0/",
"timeout": 42
},
"priority": 300,
"fail_on_error": true,
"type": "shell",
"diagnostic_name": "fuel_plugin_name-1.0.0"
}
],
"post_deployment": [
{
"uids": ["7"],
"parameters": {
"cmd": "bash deploy.sh",
"cwd": "/etc/fuel/plugins/fuel_plugin_name-1.0.0/",
"timeout": 42
},
"priority": 100,
"fail_on_error": true,
"type": "shell",
"diagnostic_name": "fuel_plugin_name-1.0.0"
}
],
"deployment_info": "<Here is regular deployment info>"
}
</nowiki></pre>

{| class="wikitable"
|-
! Task !! Comment
|-
| pre_deployment || 1st subtask: Generated automatically by Nailgun. Adds a new repository for the node. Repository's path is built according to the following template: <pre><nowiki>http://{{master_ip}}:8080/plugins/{{plugin_name}}-{{plugin_version}}/{{repository_path}}</nowiki></pre> Where:
* master_ip is an IP address of the Fuel Master node
* plugin_name is a plugin name
* plugin_version is the plugin version
* repository_path is a path for a specific release in metadata.yaml file.

2nd subtask: Generated automatically by Nailgun.Using rsync, copies plugin deployment scripts on the target nodes. Path to these files is pretty similar to the repository path. The only difference is that the deployment scripts path is taken from deployment_scripts_path that is placed into metadata.yaml file. 3rd subtask: Initiated by user and taken from tasks.yaml file, converted to task executor format.
|-
| post_deployment ||Has only one task which is taken from tasks.yaml file; uids field contains a list of nodes on which user should run a particular task. In this example, tasks.yaml file has "role: ['controller']" and this role is assigned to controller
|-
| deployment_info || Contains configuration information, required for deployment and not related to plugins.
|}

== Bugs ==

Once you have encountered a bug in a specific plugin, plugin development team should quickly learn about
that and start creating a fix or workaround.

The Fuel Plugins Launchpad project uses a widely spread bug triage workflow.
It is based on reporting bugs, providing information for bug reports and working out the fix.
In terms of the triage process, we use '''Importance''' and '''Status''' to reflect progress on bug fixing.

=== Importance criteria ===

==== Development bugs ====

* Critical = can't deploy anything and there's no trivial workaround; data loss; or security vulnerability
* High = specific hardware, configurations, or components are unusable and there's no workaround; or everything is broken but there's a workaround
* Medium = specific hardware, configurations, or components are working incorrectly; or is completely unusable but there's a workaround
* Low = minor feature is broken and can be fixed with a trivial workaround; or a cosmetic defect
* Wishlist = Not really a bug, but a suggested improvement

==== Documentation bugs ====
* Critical = following the instructions from documentation can cause outage or data loss
* High = documentation includes information that is not true, or instructions that do not yield the advertised outcome
* Medium = important information is missing from documentation (e.g. new feature description)
* Low = additional information would improve reader's understanding of a feature
* Wishlist = cosmetic formatting and grammar issues

=== Status ===

All bugs, both development and documentation can have the following statuses:
* '''New'''. When you find a bug, you should file it into [https://launchpad.net/fuel-plugins Fuel Plugins project] in Launchpad.
New is the default state assigned to the bug automatically when it was reported.
Make sure there is no bug already filed for the same issue (use advanced search
filters at LaunchPad), then enter the details of your report.

* '''Incomplete'''. The bug report is incomplete and needs more information before it can be triaged.
If you lack information to properly reproduce or assess the importance of the
bug, you should ask the original reporter for more information.

* '''Confirmed'''. Someone besides the original reporter believes that this report describes a genuine
bug in enough detail that a developer could start to work on a fix.
The person must make sure that milestone, importance and assignee are
present and set properly. Still the bug could not be reproducible or confirmed as genuine.

* '''Triaged (optional).''' The bug supervisor believes the bug report contains all information a developer
needs to start work on a fix. It is clear how to fix the bug and solution can be posted
in the comments. Sometimes the implementation of the fix will be straightforward and
you would jump directly to bugfixing, but in some other cases, you would just post your
complete debugging analysis and give someone else the opportunity of fixing the bug.

* '''In progress'''. At this stage, a developer starts working on the fix. During that time, in order to avoid
duplicating the work, this status should be set immediately when developer starts the work on the issue. Note, that when the commit is created (with sending the first patch
set out to review) and Closes-Bug tag is put into the commit message, this status is
assigned to the bug automatically. If the developer was not specified manually, then
his/her name will appear in the bug report after sending out the first patch set for
review.

* '''Fix committed'''. Once the change is reviewed, accepted, approved by Core Reviewer (with the
subsequent merge to the master branch), it will automatically move to Fix committed
status.

* '''Fix released'''. Someone besides the assignee has verified that the issue does not manifest after the fix is applied.
The status is left for QA to mark bugs as verified after fix is merged to the master
branch.

=== How to report a bug ===

# Make sure you're registered at LaunchPad. If not, see the official [http://docs.openstack.org/infra/manual/developers.html OpenStack Developer's Guide].
# Report a bug in Launchpad in the Fuel Plugins project: enter https://launchpad.net/fuel-plugins.
# Click '''Report a bug''' link: [[File:Report-a-bug-lp.png]]
# Fill in Summary field for a bug. It is going to be the headline. The headline must cointain plugin name (for example, EMC Plugin) in square brackets. Please be descriptive but short and to the point:
#* Bad - "<plugin-name> doesn’t install"
#* Good - "<plugin name> fails to install in HA mode when “Neutron with VLANs” network is selected"
# Enter “Further information”. This is a bug description. Let's focus on every issue it has:
# Description of the environment. Provide enough relevant information:
#* Output of http://fuel-master-node:8000/api/version/
#* Operating System
#* Reference Architecture (HA / non-HA)
#* Network model (Nova-network, Neutron+VLAN, Neutron+GRE, etc.)
#* Related Projects installed (Savanna, Murano, Ceilometer)
# Steps to reproduce:
#* Bad: Run the deployment in HA configuration
#* Good:Install the Fuel Master node. Copy the plugin to the Fuel Master node and install it. Create a new cluster.
# Expected result:
#* Good: The plugin is deployed, up and running.
# Actual result:
#* Bad: Plugin fails to install.
#* Good: Plugin installation fails with the error message “xxx”. Please see the attached screenshot which shows the errors on “Logs” tab.
# Workaround:
#* Bad: Don’t use “Neutron with VLANs”
#* Good: Apply patch/Change configuration from x to y.
# Impact:
#* Bad: Needs to be fixed.
#* Good: Deployment cannot be completed, because customer requires us to implement a use case for Savanna and “Neutron with VLANs”. Changing configuration to “Neutron with GRE” is not as acceptable option here so this has to be addressed as soon as possible.
# Select visibility for the bug under “This bug contains information that is” field. Either leave it as “Public” by default.
# Add attachments under “Extra Options” section
# Logs
# Diagnostic snapshot
# Screenshots
# Add <plugin-name> tag.
# After everything is entered, select the “Submit bug report” button.

 '''Important note:'''
please, keep private bugs in internal bugtrackers.
Do not report any of those at LaunchPad.

=== Common recommendations for bug triage workflow ===

When triaging bugs, please take into consideration the following:
* Review all New bugs. Plugin development team is responsible for the review. To create a plugin’s team, follow [[Fuel/Plugins#How_to_create_plugin_development_team_in_Launchpad|How to create plugin development team in Launchpad]].
* Target each '''New''' bug to the release milestone under development and set the corresponding bug status.
* Consider changing the bug title to be more specific: replace generic statements like "deployment timeout exceeded" with description of the root cause (if identified) or symptoms that are unique for that bug. Feel free to use tags in the bug title and the report with creating new tags manually when required.
* Assuming all bugs reported in Fuel Plugins project are public, request the missing information, and set the bug status to Incomplete. Add a comment using the following template: ''We didn't receive required details in order to correctly troubleshoot the problem, so we are setting this bug into Incomplete state. Please provide the requested information and we will investigate this issue further.'' If you think it was set to Incomplete by mistake, please comment in the bug.
* If there is sufficient information on how to reproduce the bug and milestone, importance and assignee are properly set, set the status to '''Confirmed'''.
* If there is sufficient information to start implementing a fix, set the status to '''Triaged.'''
* If the root cause of the bug is identified, include it in the bug description.
* Review all '''Incomplete''' bugs.
* If an I'''ncomplete''' bug has new updates, use the same steps as described above for '''New''' bugs.
* If an '''Incomplete''' bug did not have any updates for 4 weeks, close it as '''Invalid'''. Add a comment using the following template: ''This bug was incomplete for more than 4 weeks. We cannot investigate it further so we are setting the status to Invalid. If you think it is not correct, please feel free to provide more information and reopen the bug, and we will look into it further''.
* During new release scoping, review all bugs targeted at that release milestone, make sure it is updated daily by the assignee (or bug reporter if the bug is not assigned to a person).
* Every status change of a bug should be accompanied by a comment explaining the reason why the change was made.

==== Bug report tags ====

For tracking bugs better, please, use different tags:
* every plugin should have its own tag that has <plugin name> format.
* every documentation issue should be marged as ''docs''.

=== How to create plugin development team in Launchpad ===

To make the bug tracking process for plugins as simple as possible, plugin developers should
create their own teams in Launchpad. Here are the step-by-step instructions:

# Enter [https://launchpad.net Launchpad] site.
# Click '''Register a team''' link: [[File:Registerteam.png]]
# Fill in the fields. You should also provide the following links to the plugin repo, README.md file and Fuel Plugins Catalog: [[File:Fillin.png]]
# Select Membership policy: [[File:Membership-pol.png]]
# Click '''Create Team''' button to finish.

For information on adding members and running your team, see the official [https://help.launchpad.net/Teams Launchpad] guidelines.

== Tutorials ==

=== How To: Build and install a plugin from source ===

These instructions explain how to build a plugin package from the source repository. This is useful when you want to install a plugin's version that isn't available on the [https://software.mirantis.com/download-mirantis-openstack-fuel-plug-ins/ Fuel plugins catalog] yet.

1. [[Fuel/Plugins#Preparing_an_environment_for_plugin_development|Install]] the ''fuel-plugin-builder'' script.

2. Clone the plugin's repository.
<pre><nowiki>git clone https://git.openstack.org/stackforge/fuel-plugin-neutron-fwaas</nowiki></pre>

3. Go to the plugin's directory and switch to the desired Git branch or tag (optional).

4. Build the plugin's package.
<pre><nowiki>fpb --build ./fuel-plugin-neutron-fwaas</nowiki></pre>

5. The resulting RPM package should be available in the current directory. Follow the rest of the installation procedure as described in the README.md file of the plugin.

=== How To: Debug UI ===

UI elements are described in environment_config.yaml file.
To check how your built plugin looks on the Fuel web UI, install and create an environment:

1. Enter plugin directory
<pre><nowiki>cd fuel_plugin_name</nowiki></pre>

2. Edit environment_config.yaml file

3. Build a plugin
<pre><nowiki>fpb --build <plugin_name></nowiki></pre>

4. Install plugin, use "--force" parameter to replace the plugin if you have it installed:
<pre><nowiki>fuel plugins --install fuel_plugin_name-1.0.0.fp --force</nowiki></pre>

5. Create new environment
<pre><nowiki>fuel env --create --release 1 --name test</nowiki></pre>

6. Check that UI correctly shows elements from environment_config.yaml file

=== How To: Debug deployment ===

To show how it works, let's create a simple plugin with an error in the deployment script.

1. Create a plugin:
<pre><nowiki>fpb --create fuel_plugin_name</nowiki></pre>

2. Add an error in the default deployment script (fuel_plugin_name/deployment_scripts/deploy.sh):

<pre><nowiki>
#!/bin/bash
# It's a script which deploys your plugin
echo fuel_plugin_name > /tmp/fuel_plugin_name
# Non-zero exit code means, that a script executed with error
exit 1
</nowiki></pre>

3. If you do not want to run plugin build, but you want to check that plugin format is correct, you can use --check parameter with the following command:

<pre><nowiki>fpb --check fuel_plugin_name</nowiki></pre>

4. Build and install the plugin:

<pre><nowiki>fpb --build fuel_plugin_name/
fuel plugins --install fuel_plugin_name/fuel_plugin_name-1.0.0.fp</nowiki></pre>

5. Use the Fuel web UI or CLI to create an environment:
<pre><nowiki>fuel env create --name test --rel 1 --mode multinode --network-mode nova</nowiki></pre>

6. Enable the plugin on Fuel web UI Settings tab and then add several nodes. The first node has Controller role, the second node has Cinder and Computes roles.
<pre><nowiki>
fuel node set --node 1 --env 1 --role controller
fuel node set --node 2 --env 1 --role compute,cinder
</nowiki></pre>

7. Check that Nailgun generates correct configuration data that a user can set on Fuel web UI:
<pre><nowiki>
fuel deployment --default --env 1
cat deployment_1/controller_1.yaml
...
fuel_plugin_name:
fuel_plugin_name_text: Set default value
...
</nowiki></pre>

8. Now can see that the file for target node contains plugin data.

'''NOTE:'''
<pre><nowiki>
The command mentioned above is useful when you do not know how your configuration data from Fuel web UI Settings tab will look like in /etc/astute.yaml file on target nodes.
</nowiki></pre>

9. Perform provisioning without deployment for two nodes:
<pre><nowiki>fuel --env 1 node --provision --node 1,2</nowiki></pre>
'''NOTE:'''
<pre><nowiki>
To reduce the time required for testing, make a snapshot after nodes are provisioned. Note that if you use virtual machines, make snapshots of your target nodes.
</nowiki></pre>

10. Now you can run deployment:
<pre><nowiki>fuel --env 1 node --deploy --node 1,2</nowiki></pre>

11. The deployment fails with the following message:
<pre><nowiki>Deployment has failed. Method deploy. Failed to deploy plugin fuel_plugin_name-1.0.0 </nowiki></pre>

12. You can see an error in /var/log/docker-logs/astute/astute.log task executor logs:

<pre><nowiki>
[394] Shell command failed. Check debug output for details
[394] 13edd324-6a11-4342-bc04-66c659e75e35: cmd: bash deploy.sh
cwd: /etc/fuel/plugins/fuel_plugin_name-1.0.0/
stdout:
stderr:
exit code: 1
</nowiki></pre>

13. It fails due to the changes in deploy.sh script that you made in step 2. Let's assume that we do not know what happened and try to debug the problem:
<pre><nowiki>
# Go to the first node
ssh node-1
</nowiki></pre>

14. All plugin deployment scripts are copied to the separate directory on the target node; in this case, it is /etc/fuel/plugins/fuel_plugin_name-1.0.0/:
<pre><nowiki>
cd /etc/fuel/plugins/fuel_plugin_name-1.0.0/
# The directory contains our deploy.sh script, lets run it
bash deploy.sh
# And check exit code
echo $? # Returns 1
</nowiki></pre>

'''NOTE:'''
<pre><nowiki>
If you use puppet for your plugin deployment, run the following command on the target node to check if your puppet manifests work correctly:

puppet apply --debug --modulepath=/etc/fuel/plugins/fuel_plugin_name-1.0.0/puppet/modules:/etc/puppet/modules /etc/fuel/plugins/fuel_plugin_name-1.0.0/puppet/manifests/site.pp</nowiki></pre>

15. Now we can see that deployment fails due to non-zero exit code error. To fix the problem and check that the proposed solution works, edit the /var/www/nailgun/plugins/fuel_plugin_name-1.0.0/deployment_scripts/deploy.sh script on the Fuel Master node. Note that there is no need to rebuild and reinstall a plugin:
<pre><nowiki>
#!/bin/bash

# It's a script which deploys your plugin
echo fuel_plugin_name > /tmp/fuel_plugin_name

# Now our deployment script returns 0 instead of 1
exit 0
</nowiki></pre>

16. If you run the deployment again, it goes successfully:
<pre><nowiki>
fuel --env 1 node --deploy --node 1,2
</nowiki></pre>

'''WARNING:'''
<pre><nowiki>
During the testing of your deployment scripts, make sure that your scripts are idempotent: they should work correctly when applied several times. Run environment deployment at least twice and check that your plugin works properly. The reason for this workflow is the following: Fuel can run deployment of your plugin several times in case the first deployment try failed. Also, your deployment scripts can be executed during OpenStack patching.</nowiki></pre>

17. To make sure that plugin works without errors, revert snapshots which you made in step 6, and run deployment again:

<pre><nowiki>
fuel --env 1 node --deploy --node 1,2
</nowiki></pre>

18. In the same way with no plugin reinstallation, you can edit /var/www/nailgun/plugins/<fuel_plugin_name>-1.0.0/tasks.yaml file. Note that in this case to make sure that your tasks have a valid format, you should at least run the following command:
<pre><nowiki>
fpb --check /var/www/nailgun/plugins/fuel_plugin_name-1.0.0/
</nowiki></pre>

== Plugin elements in the Fuel web UI ==

Here is an example of the environment_config.yaml file that contains all possible Fuel web UI elements:
<pre><nowiki>
attributes:

# Text field
fuel_plugin_name_text:
type: "text"
weight: 10
value: "Default text"
label: "Text field label"
description: "Field description"
regex:
source: '\S'
error: "Error field cannot be empty"

# Select
fuel_plugin_name_select:
type: "select"
weight: 20
value: "value2"
label: "Select label"
description: "Select description"
values:
- data: "value1"
label: "Value 1 label"
- data: "value2"
label: "Value 2 label"
- data: "value3"
label: "Value 3 label"

# Checkbox
fuel_plugin_name_checkbox:
type: "checkbox"
weight: 30
value: false
label: "Checkbox label"
description: "Checkbox description"

# Radio button
fuel_plugin_name_radio:
type: "radio"
weight: 40
value: "disabled"
label: "Radio buttons label"
values:
- data: "data1"
label: "Label data1"
description: "Description data1"
- data: "data2"
label: "Label data2"
description: "Description data2"
- data: "data3"
label: "Label data3"
description: "Description data3"
</nowiki></pre>

After plugin is installed, additional elements will appear on the Settings tab of the Fuel web UI.
Here are UI elemens for Elasticsearch-Kibana plugin:

[[File: Ui-elements.png]]

== Puppet in Fuel ==

Fuel does not use [https://docs.puppetlabs.com/learning/agent_master_basic.html#what-do-agents-do-and-what-do-masters-do master Puppet]. Task executor copies manifest from the Fuel Master node and runs puppet apply command on each target node.
It is recommended that you use puppet tasks in your plugin instead of running puppet in shell tasks.
Task executor has [https://github.com/stackforge/fuel-astute/blob/122cdaab/mcagents/puppetd.rb code with special logic] which handles errors, if puppet apply command returns zero/non-zero exit code. Note that it does not mean that command is succeed or failed. That means, it returns '2' if there were changes during the execution: task executor parses /var/lib/puppet/state/last_run_summary.yaml file to determine the status of puppet run.

== How to separate services from Controller with a plugin ==

Starting with Fuel 7.0, you can create and implement a plugin that allows any role or task to be broken up into sections and deployed on a custom role. This includes the high availability feature enabling as well.

Fuel includes a granular deployment framework that relies on a task based-deployment schema defining how each node role is deployed. To modify the granular deployment framework, you can create a plugin that inserts, overrides, or skips any given task. In addition, you can add any custom role that fulfils a task normally tied to the Controller role. Generally speaking, all Controller services are deployed on the Controller nodes directly.

: '''Note:''' In the granular deployment, all dependencies are preserved when skipping a task. A custom role must deploy either before or after the Controller role. For example, you cannot deploy tasks A, B, C, and D on the Controller, pause it, deploy the custom ''MyRole'' role, then finish tasks E through Z on the Controller role. For this reason, it is best to deploy your plugin in a self-sufficient way that does not rely on controller services.

If you want to change the deployment configuration, override the default setting using the '''Hiera''' tool.

=== Hiera overview ===

In spite of Puppet is static in configuration, it can customize deployment based on the metadata fed in from an external data source. In this case, Fuel uses Hiera to define node configuration.

[http://docs.puppetlabs.com/hiera/ Hiera] is a key/value lookup tool for configuration data, but you can configure it to define the priority of the data sources.

Configuration items are stored in 3 different data structures:

* simple key/value,
* arrays,
* hashes.

'''Examples:'''

Key/value
role: “controller”
primary_controller: true 
Array:
amqp_hosts: [“192.168.0.3:5673”,”192.168.0.4:5673”,”192.168.0.5:5673”] 
Hash:
mysql:
root_password: “OvEuZFyt”
wsrep_password: “Secr3t”

Hiera also offers a hierarchy of configuration data. It has several layers of precedence in its hierarchy by default. The default ''/etc/hiera.yaml'' looks as follows:

---
:backends:
- yaml 
:hierarchy:
- override/node/%{::fqdn}
- override/class/%{calling_class}
- override/module/%{calling_module}
- override/plugins
- override/common
- class/%{calling_class}
- module/%{calling_module}
- nodes
- globals
- astute 
:yaml:
:datadir: /etc/hiera
:merge_behavior: deeper
:logger: noop

Any values defined in ''/etc/hiera/globals.yaml'' takes precedence over values defined in ''/etc/hiera/astute.yaml''. The same is true for any value defined higher up. If a plugin defines a new YAML file and inserts itself in the hierarchy, it can override a given value as well.

Queries to hiera come in three forms:

* priority,
* array merge,
* hash merge.

The defined values can either override or be merged with other values. As an example, you can either merge the array of ''memcache_roles'' from ''astute.yaml'' and a custom plugin YAML or take the highest priority. The difference is in how puppet calls it:
Merged lookup:
hiera_array(‘memcache_roles’)
Priority lookup:
hiera(‘memcache_roles’)

In most cases, array lookups should be done with a priority lookup. In contrast, hash lookups in Hiera should always be done using the <code>hiera_hash</code> function.

'''Example:'''

astute.yaml contains:
mysql:
root_password: “OvEuZFyt”
wsrep_password: “Secr3t”
plugins.yaml contains:
mysql:
enabled: false

The results are vastly different if you do a priority lookup versus a hash lookup:

Priority lookup:
$ hiera mysql
{“enabled”=>false}
Hash lookup:
$ hiera -h mysql
{“enabled”=>false,“root_password”=>“OvEuZFyt”,”wsrep_password”=>“Secr3t”}

Hash merging in Hiera allows a plugin developer to tweak an individual setting in the configuration without duplicating all of the other values in a hash.

==== How a plugin can use Hiera ====

A plugin can modify any Hiera configuration data for a deployment to inform existing tasks how to deploy. For example, if you want to run ''memcached'' on a separate ''my-role'' role , you can override the ''memcache_roles'' array to say ''my-role'' instead of ''primary-controller, controller''.

=== Deploying services into separate nodes ===

Rabbit and MySQL can be easily turned off on the Controller role by overriding the metadata in Hiera, as described above. This can be accomplished by disabling a service inside its respective service hash, indicated by a value ''enabled: false''. However, some services such as Keystone is not so simple and require more configuration. For the specific details proceed with the sections below.

==== Defining a custom role in a plugin ====

If you want to deploy separate services on a different node from the Controller, you need to define a custom role first.

To define custom node roles in a plugin, fill out the ''node_roles.yaml'' file in your plugin’s base directory. A custom role that replaces an existing task must contain some particular fields to achieve this objective.

In addition to the default attributes, you need to include the following attributes in the plugin:

; <code>update_required</code>
: Corosync hosts, memcached hosts, and AMQP hosts are enumerated explicitly in configuration. All existing nodes for the specified roles will be refreshed before adding/removing a given node. This list should include all consumers of the service you are deploying.

; <code>public_ip_required</code>
: Internal-only services (like RabbitMQ) should not have a public IP.

; <code>has_primary</code>
: Required for any deployment that includes corosync.

; <code>conflicts</code>
: Roles containing sensitive data should not be combined with Compute. In our example, we also conflict with the Controller role because it could break if there is not a 1:1 mapping of controller and ''custom_role'' roles to all other nodes.

'''See Also:'''

* [https://wiki.openstack.org/wiki/Fuel/Plugins#Configuration_of_Fuel_Plugins_with_new_roles custom node roles details]
* [https://github.com/stackforge/fuel-plugin-detach-database/blob/master/node_roles.yaml node_roles.yaml example]

==== Configuring Hiera ====

===== Database separation =====

Database separation is intended to be as flexible as possible for various deployment configurations including remote database creation, user creation, and different database servers for different services.

The list of the parameters configurable for MySQL and its metadata is as follows:
* MySQL root password (default is auto-generated)
* db_host or Database VIP
* db_create (defaults to true)
* db_user (defaults to service name)
* db_password (defaults to auto-generated value)

Additionally, for the MySQL hash on the Controller roles, you need to set mysql to ''enabled: false''.

===== Adding in HA =====

High availability is key to any production deployment. Each service has HA configured through corosync or haproxy.

The following services are managed by corosync:

* Virtual IP addresses
* haproxy
* MySQL/Galera
* RabbitMQ
* Heat
* Neutron agents

This means that you need to include one or more of the following tasks in your custom role:

* cluster
* cluster-haproxy
* virtual_ips
* openstack-haproxy-SERVICENAME

: '''Note:''' Since RabbitMQ only requires corosync and not haproxy or a virtual IP, it only requires the cluster task for high availability. On the other hand, MySQL and Keystone would require all four of these.

The metadata requirements for corosync and haproxy are prepared to accept custom values through Hiera. For corosync itself, you need one key value that is ''corosync_roles''. This is necessary because UDP configuration of Corosync requires all nodes to explicitly allow communication with each other. The value should equal the role name for your plugin as follows:

corosync_roles:
- primary-standalone-keystone
- standalone-keystone

: '''Note:''' This should only apply when evaluating your custom role. Refer to the [https://github.com/stackforge/fuel-plugin-detach-keystone/blob/master/deployment_scripts/hiera-override.pp#L54-L65 detach-keystone plugin] as an example.

If the service you want to detach requires HAProxy, include the following arrays to your plugin:
* servicename_nodes
* servicename_ipaddresses
* servicename_nodes_names

See [https://github.com/stackforge/fuel-plugin-detach-keystone/blob/master/deployment_scripts/hiera-override.pp#L45-L52 detach-keystone plugin] as an example.

If the service being separated requires memcached, specify ''memcache_roles'' in your override hiera YAML. This is necessary if your role is evaluated before the Controller role. However, in case your role is deployed after the Controller role, you can use memcached from the Controller role leaving this value unchanged.

===== RabbitMQ separation =====

Configure Hiera as follows:

# Set the ''amqp_hosts'' array on all nodes. Since the original list of hosts is generated in the globals task of the granular deployment, you need to override this value without merging. As there is no virtual IP to manage, it is the only value to control.
# Set ''rabbit_hash'' to ''enabled: false'' for the Controller roles.

===== Keystone separation =====

To meet a specific use case, decoupling the Keystone service from the Controller role, proceed with the following steps:

: 1. On the Controller node, view the task list for a created environment:

fuel graph --env 1 --download

The tasks we are concerned with for keystone are:

; ''keystone''
: creates the keystone service

; ''keystone-db''
: prepares the DB for keystone

; ''openstack-haproxy-keystone''
: enables haproxy for keystone

; ''disable_keystone_service_token''
: cleanup task for ''admin_token''

: 2. Shift the tasks listed above to a custom keystone role (it is the “standalone-keystone” role in our example):

* Disable original keystone tasks creating a YAML file called ''deployment_tasks.yaml'' and defining new tasks with the type <code>skipped</code> preventing these tasks from running on the Controller role:

id: original_task_name
type: skipped

* Create a new task that reproduces the original one, but replace the requirements and role for the original task: 

'''Original task'''

- id: keystone-db
type: puppet
groups: [primary-controller]
required_for: [keystone]
requires: [database]
parameters:
puppet_manifest: /etc/puppet/modules/osnailyfacter/modular/keystone/db.pp
puppet_modules: /etc/puppet/modules
timeout: 1800

'''New tasks'''

- id: keystone-db
type: skipped 
- id: task-keystone-db
type: puppet
role: [primary-standalone-keystone, standalone-keystone]
required_for: [task-keystone]
requires: [deploy_start, keystone-hiera-override, netconfig]
parameters:
puppet_manifest: /etc/puppet/modules/osnailyfacter/modular/keystone/db.pp
puppet_modules: /etc/puppet/modules
timeout: 1800

: 3. Ensure all new tasks depend on each other.

For each new task you create, ensure its dependencies line up with the tasks you marked as skipped. In the example above, ''task-keystone-db'' is a dependent task for ''task-keystone'', which is another custom task that performs the same function.

'''See also:'''

* [https://github.com/stackforge/fuel-plugin-detach-keystone/blob/master/deployment_tasks.yaml#L60 deployment_tasks.yaml sample]

==== Verifying changes ====

# Create an environment to test your task graph with 1 Controller node and 1 node with your custom role.
# Run <code>fuel plugins --sync</code>
# Run <code>fuel graph --env 1 --download</code>

If successful, these should run without errors.
If there is a syntax mistake, it returns a 500 error. For the proper YAML syntax, see the [https://github.com/stackforge/fuel-plugins example plugins].

=== Procedures and limitations ===

==== Procedure for RabbitMQ ====

# Create a new custom role.
# Enable corosync (for HA).
# Override the ''amqp_hosts'' array on all hosts that consume AMQP.
# Disable rabbitmq on the Controller role.

==== Procedure for Galera ====

# Create a new custom role.
# Enable corosync (for HA).
# Override mysql hash: enable it for the custom role, disable it for the Controller role.
# Provide ''database_vip'' and ''haproxy'' for your role.

==== Procedure for Keystone ====

# Disable Keystone base task, haproxy task, and DB task on the Controller node.
# Create a new custom role.
# Add ''service_endpoint'' and ''public_service_endpoint'' VIPs in the custom role.
# Enable corosync (for the VIP, but keystone itself is stateless).
# Enable memcached.
# Enable dependency on database plugin.
# For SSL deployments, enable haproxy for Keystone on public VIP

==== Limitations for Keystone separation ====

* Keystone cannot run without MySQL and memcached.
* The Controller role provides these roles, but you cannot do a partial Controller deployment, stop, deploy keystone, and continue.
* Keystone public SSL is limited to one hostname.

=== Useful links ===

* Examples of plugins:
** [https://github.com/stackforge/fuel-plugin-detach-database detach-database plugin]
** [https://github.com/stackforge/fuel-plugin-detach-rabbitmq detach-rabbitmq plugin]
** [https://github.com/stackforge/fuel-plugin-detach-keystone detach-keystone plugin]

* [http://docs.puppetlabs.com/hiera/1/ Hiera]

* [https://wiki.openstack.org/wiki/Fuel/Plugins#Virtual_IP_reservation_via_Fuel_Plugin.27s_metadata VIP reservation in plugin]

* [https://wiki.openstack.org/wiki/Fuel/Plugins#Configuration_of_Fuel_Plugins_with_new_roles Role definition in plugin]

== Creating documentation for Fuel Plugins ==

Fuel Plugins can provide brand different functionality, that's why you should not only develop a plugin itself,
but also write documentation clear to end users.

=== Documentation package ===

To give all users the understanding of what you plugins does and how it was tested,
you need to have the following files in the plugin's repo '''under /doc folder''':
 
* Test Plan - contains information on planned testing activities
* Test Report - contains information obtained during testing
* Plugin Guide - contains installation, configuration and usage instructions.

 
Please, note that your plugin's repo should also contain:
* LICENSE file with license specified correctly.
* copyright for specific repo files added at the file header. For details, see [[Fuel/Plugins#License_and_copyright|the corresponding section]].
* README file with instructions on how to build your plugin, see [https://github.com/stackforge/fuel-plugin-contrail/blob/master/README.md the examples here].
* developer's specification - contains design and implementation details and explains which problem can be solved with the plugin. It is put into '''/specs''' folder.
 
When writing, please take [[Fuel/Plugins#Style_recommendations|style recommendations]] into consideration.

=== Following Gerrit workflow ===

[http://docs.openstack.org/infra/manual/developers.html Gerrit workflow] provides the following advantages for Fuel Plugin documentation:
* aligns development and documentation writing cycles.
* provides shipping both specific plugin and related documentation at once.
* provides writing in RST (as already works for the specs).
* enables flexible updates for developers.
 

==== Documentation files structure and Sphinx usage ====

To simplify the build procedure, you can use Sphinx.

To enable it in your plugin's repo, follow these steps:
# Make sure you've got a '''/doc''' folder in your repo (according to diagram below).
# Consider splitting your Plugin Guide into smaller RST pieces. Sample can be found [https://github.com/openstack/fuel-plugin-openbook/tree/7.0/doc here].
# Check if you have '''index''' file in place. It will serve as the source to build the document. It also needs to be edited to include the RST pieces (check the sample).
# Add a Makefile using the sample. Makefiles are usually left as-is to enable PDF and html build (check the [https://github.com/openstack/fuel-plugin-openbook/blob/7.0/doc/source/conf.py sample]).
# Edit [https://github.com/openstack/fuel-plugin-openbook/blob/7.0/doc/source/conf.py '''conf.py'''] file. This file is in charge of the document's appearance and you can modify it according to your own needs:
#* [https://github.com/openstack/fuel-plugin-openbook/blob/7.0/doc/source/conf.py#L267 Enable page numbering] in a proper manner.
#* [https://github.com/openstack/fuel-plugin-openbook/blob/7.0/doc/source/conf.py#L226 Get rid of blank pages] before and after contents page.
# Commit your change.
# Try getting into '''/doc''' folder and running 'make latexpdf' out of it.
As the deliverable of this exercise, you should get a PDF version of your document in the /build folder.

Please, see the recommended file structure for Plugin Guide (as sample):
[[File: Doc-scheme-repo.png]]

=== Style recommendations ===

# Spell out acronyms and abbreviations when they are first used and avoid using abbreviations that are ambiguous. For example, "OS" could mean either "Operating System" or "OpenStack".
# Do not add many links to external resources in main sections. It is recommended that you put them into Appendix section.
# Each link should have a brief description.Be sure to define any complex concepts and terms that are required to install/use/test the plugin.
# Use step-by-step instructions, structured with an introductory sentence followed by a numbered list of steps to take.
 

=== Acceptance criteria ===

Plugin-related documentation (Test Plan, Test Report, Plugin Guide, specification) should meet the following acceptance criteria: 
* the documentation is written in RST and put into the plugin's repo
* the plugin version is specified correctly;it must resemble the one reflected in the plugin's metadata.yaml file
* all links to external documentation/resources should be present and put into Appendix
* all complex concepts are explained in a detailed and clear manner
 
Please, mind some specific recommendations for the following documentation:
# '''Plugin Guide''' should contain 2 main sections for installation and usage.
#* The former should focus on 1) how/where to download a plugin 2) how to install a plugin 3) any additional prerequisite procedures (e.g. backend configuration).
#* The latter should cover 1) how to use the plugin (e.g. a set of commands) with an example (commands usage with specific values is strongly required) 2) links to external documentation.
#* If plugin has its own UI or supposes some actions to take in Horizon, the Plugin Guide must have screenshots with clearly seen UI elements.
# '''Test Plan and Test Report''' must have clearly specified prerequisites to make the plugin up and running.
# '''Plugin's specification''' must cover design and implementation peculiarities. See the [https://github.com/stackforge/fuel-specs/blob/master/specs/template.rst template].

==== Checklist ====
{| class="wikitable"
|-
! Issue !! Tick if done
|-
| Plugin version is the one specified in the plugin's metadata.yaml file (see [https://github.com/stackforge/fuel-plugin-neutron-vpnaas/blob/master/metadata.yaml#L6 the example]). ||
|-
| All key terms/acronyms/abbreviations are put into the corresponding table and clearly explained or at least refer to other resources with detailed description. ||
|-
| Plugin name doesn't somehow change within the document (like VPNaaS and vpnaas). Stick to one variant and use it along the text.||
|-
| If configuration of specific components (if any) is not covered within the document, then there must be a link pointing to the detailed step-by-step instructions.||
|-
| All UI elements are properly seen in the screenshots and or even highlighted to point at a specific action/element. ||
|-
| All instructions are put into numbered lists to track the task accomplishment and navigate along the document. ||
|-
| All instructions concerning Fuel UI wizard or Fuel web UI refer to the official documentation. ||
|}

== Add your plugin to DriverLog ==

Once you plugin is ready, you can add it to DriverLog, following instructions from [[DriverLog#How_To:_Add_a_new_Fuel_Plugin_to_DriverLog|DriverLog wiki page]].
Before adding, please make sure that:
{| class="wikitable"
|-
! Issue !! Example !! Tick if done
|-
| [[Fuel/Plugins#Acceptance_criteria|Plugin's documentation is added to /doc folder.]] || [https://github.com/stackforge/fuel-plugin-midonet/tree/master/doc Midonet plugin] ||
|-
| [[Fuel/Plugins#Documentation_files_structure|Plugin's specification is added to /specs folder]]. || [https://github.com/stackforge/fuel-plugin-solidfire-cinder/blob/master/specs/solidfire-plugin-specs.rst Solidfire plugin] ||
|-
| [[Fuel/Plugins#Creating_documentation_for_Fuel_Plugins|Plugin's README file contains instructions on how to build the plugin.]] || [https://github.com/stackforge/fuel-plugin-contrail/blob/master/README.md Contrail plugin] ||
|-
| [[Fuel/Plugins#License_and_copyright|Plugin's LICENSE file contains the relevant license.]] || [https://github.com/stackforge/fuel-plugin-swiftstack/blob/master/LICENSE Swiftstack plugin] ||
|-
|}

== License and copyright ==

Fuel Plugins are licensed under [https://www.apache.org/licenses/LICENSE-2.0 Apache 2.0].
 The plugin's repo should contain the following:
* the license as a separate file (LICENSE) - please, use [http://www.apache.org/licenses/LICENSE-2.0 the official license text].
* all files in the repository (except for configuration files and README) must have a copyright, see [https://github.com/stackforge/fuel-library/blob/e64d5408c0274950fd88e37bfbd94991941bbe4d/deployment/puppet/cobbler/manifests/profile/centos63_x86_64.pp#L1 fuel-library repo].
 
Note, that configuration files can have the following format: 
* yaml (see [https://github.com/stackforge/fuel-plugin-contrail/blob/master/metadata.yaml the example]).
* erb (see [https://github.com/stackforge/fuel-plugin-contrail/blob/master/deployment_scripts/puppet/modules/contrail/templates/094-web_for_contrail.cfg.erb the example]).
* lua (see [https://github.com/stackforge/fuel-plugin-lma-collector/blob/master/deployment_scripts/puppet/modules/lma_collector/files/plugins/common/patterns.lua the example]).
 
The Fuel Plugins can have extra components included.
To provide users with this information, make sure that the Plugin Guide has a separate section with the following table:
{| class="wikitable"
|-
! Component !! License
|-
| Component1 || License1
|-
| Component2|| License2
|}

== FAQ ==

'''''Where can I find Fuel Plugin Builder source code?'''''

fuel-plugin-builder is located in fuel-plugins repository.

'''''Are there any plugins examples?'''''

All plugins for Fuel now have their own repos under Stackforge project.

'''''How can I reuse Puppet modules from Fuel?'''''
According to the design, every plugin should have all necessary components to be then deployed. This means, every plugin should have its own copy of Fuel Puppet modules. If you do not want to keep copy of Fuel library manifests in your repository, you can use pre_build_hook to download the required modules during the plugin build. To do that, add the following code into your hook:
<pre><nowiki>
#!/bin/bash
set -eux
ROOT="$(dirname `readlink -f $0`)"
MODULES="${ROOT}"/deployment_scripts/puppet/modules
mkdir -p "${MODULES}"
REPO_PATH='https://github.com/stackforge/fuel-library/tarball/f43d885914d74fbd062096763222f350f47480e1'
RPM_REPO="${ROOT}"/repositories/centos/
DEB_REPO="${ROOT}"/repositories/ubuntu/

wget -qO- "${REPO_PATH}" | \
tar -C "${MODULES}" --strip-components=3 -zxvf - \
stackforge-fuel-library-f43d885/deployment/puppet/{inifile,stdlib}

</nowiki></pre>

The code then copies inifile and stdlib modules from fuel-library repository.

'''WARNING'''
<pre><nowiki>
To reuse existing Puppet manifests you can also specify several Puppet modules in your task with colon separator: for example, puppet_modules: "puppet/modules:/etc/puppet/modules". Note that we do not recommend using this approach, because Fuel puppet modules can be changed during OpenStack update procedure; this can lead to compatibility failures.
</nowiki></pre>

'''''How can I download the packages which are required for a plugin?'''''
Use wget in your pre_build_hook script to download packages in the required directories:
<pre><nowiki>
#!/bin/bash
set -eux

ROOT="$(dirname `readlink -f $0`)"
RPM_REPO="${ROOT}"/repositories/centos/
DEB_REPO="${ROOT}"/repositories/ubuntu/

wget -P "${RPM_REPO}" http://mirror.fuel-infra.org/fuel-plugins/6.0/centos/glusterfs-3.5.2-1.mira2.x86_64.rpm
wget -P "${DEB_REPO}" http://mirror.fuel-infra.org/fuel-plugins/6.0/ubuntu/glusterfs-client_3.5.2-4_amd64.deb
</nowiki></pre>

It downloads two packages in your plugin's directories before Fuel Plugin Builder starts building repositories.

'''''Why is there no /etc/astute.yaml file, when I run pre_deployment task?'''''
If you have task with "stage: pre_deployment" parameter set, you will not find /etc/astute.yaml file on the target node during the task execution. The file /etc/astute.yaml is a symlink that is created before Fuel deploys a role.
Target node can have several roles and each role contains its own file with deployment data.
Here is the example of a node with ID 2 and two roles, Controller and Cinder:
<pre><nowiki>
root@node-2:~# ls -l /etc/ | grep yaml
-rw------- 1 root root 8712 Nov 19 12:48 controller.yaml
-rw------- 1 root root 8700 Nov 19 12:48 cinder.yaml
</nowiki></pre>
Let's assume that we need deployment data file for Controller role.
We can use '/etc/controller.yaml' file directly in deployment script.

'''''What is the user context that fuel plugin hooks are invoked as: root or fuel user?'''''
Plugins hooks are executed under the root.

'''''Are the keystone IP addresses and user name or password available to the fuel plugin post_deployment hook?'''''
Keystone admin_token, admin_password are in /etc/astute.yaml and can be used in the post_deployment hook. For connection to keystone we should use management_vip:5000. management_vip is also in the astute.yaml. Also keystone url can be fetched from /root/openrc (OS_AUTH_URL), but this is hackish way.

'''''Do multiple openstack controller nodes mean multiple neutronserver nodes?Neutron server is installed on each controller node.
Are all the controller nodes' private IP (not VIP) available and accessible to the postdeployment hook?(because we want to run the avicontroller, network etconly once, but we have to run the plugin install on all neutronserver nodes)? Since there are multiple controller nodes, then fuel infrastructure automatically runs the plugin postdeployment hook on each of them.'''''

The private IP is referred to the IP of VM, or what we call management IP - all people in OpenStack name it differently. On the post_deployment step all networks, controllers, e.t.c. are configured and run.

'''''In pre-deployment, the astute.yaml is a symlink to the role specific yaml (controller, cinder etc) before the role is deployed, but in post-deployment how is astute.yaml organized - merged from all role yamls?'''''
The astute.yaml files are not merged during deployment process. Result file just astute.yaml for last deployed role. Specific astute.yaml can be found by role name.

'''''How can I add a custom role?'''''
Current Plugin Architecture does not provide a way to define role. Starting with Fuel 6.1 you can use the Operating System role as a workaround.

In the tasks.yaml file use the base-os string to specify which task should be used for the role deployment.

<pre><nowiki>
- role: ['base-os']
stage: post_deployment
type: shell
parameters:
cmd: bash deploy.sh
timeout: 42
</nowiki></pre>

When creating documentation for your plugin, ask the user to assign the Operating System role on the required node and specify the name to give this node, such as zabbix-01.
In the deployment script you can query the user_node_name variable and decide if the script should deploy Zabbix related packages on the node.

<pre><nowiki>
...
user_node_name: zabbix-01
...
</nowiki></pre>

== Channels of communication ==

If you have some questions left, feel free to use the '''openstack-dev''' mailing list (use ''[fuel][plugin]'' prefix). For instructions on mailing list usage, see [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev openstack-dev]
mailing list instructions.

Fuel/How to contribute

2015-06-30T11:25:06Z

Adidenko: /* Test and report bugs */

=Join the Fuel team=
=== First steps ===
Read the [http://docs.openstack.org/infra/manual/developers.html Developer's Guide] and learn how to setup the accounts and git.

Clone the repositories:
* git clone git://github.com/stackforge/fuel-main.git
* git clone git://github.com/stackforge/fuel-web.git
* git clone git://github.com/stackforge/fuel-astute.git
* git clone git://github.com/stackforge/fuel-ostf.git
* git clone git://github.com/stackforge/fuel-library.git
* git clone git://github.com/stackforge/fuel-docs.git

You can [https://review.openstack.org/#/q/project:%255Estackforge/fuel-.*,n,z search gerrit for all reviews related to Fuel], or use the [https://review.openstack.org/#/dashboard/?foreach=%28project%3A^stackforge%2Ffuel-.*%29+status%3Aopen+NOT+label%3AWorkflow%3C%3D-1+NOT+label%3ACode-Review%3C%3D-2&title=Fuel+Inbox&My+Patches+Requiring+Attention=owner%3Aself+%28label%3AVerified-1+OR+label%3ACode-Review-1+OR+label%3AWorkflow%3C%3D-1%29&Fuel+Specs=NOT+label%3AWorkflow%3E%3D1+NOT+owner%3Aself+project%3Astackforge%2Ffuel-specs&Needs+Approval=NOT+label%3AWorkflow%3E%3D1+NOT+label%3AWorkflow%3C%3D-1+NOT+label%3AVerified%3C%3D-1+NOT+owner%3Aself+label%3ACode-Review%3E%3D2+NOT+label%3ACode-Review-1&5+Days+Without+Feedback=NOT+label%3AWorkflow%3E%3D1+NOT+label%3AWorkflow%3C%3D-1+NOT+label%3AVerified%3C%3D-1+NOT+owner%3Aself+NOT+project%3Astackforge%2Ffuel-specs+NOT+label%3ACode-Review%3C%3D2+age%3A5d&No+Negative+Feedback=NOT+label%3AWorkflow%3E%3D1+NOT+label%3AWorkflow%3C%3D-1+NOT+label%3AVerified%3C%3D-1+NOT+owner%3Aself+NOT+project%3Astackforge%2Ffuel-specs+NOT+label%3ACode-Review%3C%3D-1+NOT+label%3ACode-Review%3E%3D2+limit%3A50&Other=NOT+label%3AVerified%3C%3D-1+NOT+owner%3Aself+NOT+project%3Astackforge%2Ffuel-specs+label%3ACode-Review-1+limit%3A20 gerrit dashboard for Fuel] for a prioritized list of open reviews (this URL was generated using [https://github.com/stackforge/gerrit-dash-creator/blob/master/dashboards/fuel.dash Gerrit dashboard generator], you need to be logged into Gerrit for it to work).

Please note that OpenStack Infra uses strict checks on your git commit messages. The most important rules are:
* Keep your commit title short (<80 symbols). It must not end with dot.
* Body of commit message should be separated from title by one empty line
* If patch closes bug, then it must have in a body "Closes-Bug: #12345", where 12345 is your LP bug number
* If patch relates to bug, but doesn't completely solve an issue, then you must use "Related-Bug: #12345"
See the [[GitCommitMessages|Git commit messages]] wiki page for more details.

=How and where to get help=
=== Subscribe to mailing lists ===
* [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev OpenStack developers mailing list]. If you are hacking on Fuel, or have technical question about Fuel internals, you are very welcome in this ML with subject containing "[Fuel]".
* [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack OpenStack users mailing list]. Fuel traffic is there with subject containing "[Fuel]". You are welcome to ask questions related to Fuel usage, including errors, etc.
* [http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-announce OpenStack announcements mailing list]. Announcements about the OpenStack project like product release information, security advisories, important discussions. This is a low-traffic, read-only list.
* There are even more mailing lists out there, visit [[MailingLists]] to see the other OpenStack related lists.

=== IRC at freenode.net ===
* The general Fuel channel is called "#fuel", and it is logged [http://irclog.perlgeek.de/fuel/ here].
* The Fuel development channel is called "#fuel-dev", and it is logged [http://irclog.perlgeek.de/fuel-dev/ here].

=== Quick reference dev links ===
* Development environment [http://docs.mirantis.com/fuel-dev/develop/env.html http://docs.mirantis.com/fuel-dev/develop/env.html]
* How to build your Fuel ISO [http://docs.mirantis.com/fuel-dev/develop/env.html#building-the-fuel-iso http://docs.mirantis.com/fuel-dev/develop/env.html#building-the-fuel-iso]
* Nailgun development [http://docs.mirantis.com/fuel-dev/develop/nailgun.html http://docs.mirantis.com/fuel-dev/develop/nailgun.html]
* Health Check (OSTF) contributors guide [http://docs.mirantis.com/fuel-dev/develop/ostf_contributors_guide.html http://docs.mirantis.com/fuel-dev/develop/ostf_contributors_guide.html]
* Contributing to Fuel Library (puppet modules) [http://docs.mirantis.com/fuel-dev/develop/module_structure.html http://docs.mirantis.com/fuel-dev/develop/module_structure.html]
* Devops framework - distributed testing of Fuel envs [http://docs.mirantis.com/fuel-dev/devops.html http://docs.mirantis.com/fuel-dev/devops.html]

=Contributing=
Fuel is large project within which Programming, Linux and Networking intersect. It is written in Python, Ruby and JavaScript. It also has a number of build scripts in bash & make. Documentation is written in Sphinx. New contributors are always welcome.

==Fuel library for puppet manifests==
===Adding new puppet modules to fuel-library===
'''Case A. Pulling in changes from an existing module'''

If you are pulling in changes to an existing fork from a module that is the work of another project and is already tracked in separate repo then:
1. Create a review request with a unmodified copy of the upstream module from whichever point you are working from and no other related modifications.
* This review should also contain the commit hash from the upstream repo in the commit message.
* The review should be evaluated to determine its suitability and either rejected (for licensing, code quality, outdated version requested) or accepted without requiring modifications.
* The review should not include code that calls this new module.
2. Any changes necessary to make it work with Fuel should then be proposed as a dependent change(s).

'''Case B. Adding a new module'''

If you are adding a new module that is a work purely for Fuel and will not be tracked in a separate repo then submit incremental reviews that consist of working implementation of features for your module.
If you have features that are necessary, but do not work fully yet, then prevent them from running during the deployment. Once your feature is complete, submit a review to activate the module during deployment.

If you are going to add a new fork of upstream module - please don't do. Instead, follow the upstream module's contribution rules and provide this module as an additional plug-able component for Fuel. This can be done as a part of [[Fuel/Plugins|plug-able architecture]] and distributed as a package. Note, if upstream module doesn't fit your requirements, you should first adapt the module for your needs upstream or, for emergency cases, provide custom patch-sets as a part of module as a Fuel plugin distribution.

Modules with licenses incompatible with Apache 2.0 (GPL, etc.) will not be accepted. *Please pay attention to the license information.*

===Contributing to existing fuel-library modules===
As developers of Puppet modules, we tend to collaborate with the Puppet OpenStack community. As a result, we contribute to upstream modules all of the improvements, fixes and customizations we make to improve Fuel as well. That implies that every contributor must follow Puppet DSL basics, [[Puppet-openstack#Developer_documentation|puppet-openstack dev docs]] and [[Puppet-openstack#Rspec_puppet_tests|Puppet rspec tests requirements]].

The most common and general rule is that upstream modules should be modified only when bugfixes and improvements could benefit everyone in the community. And appropriate patch should be proposed to the upstream project prior to Fuel project.

In other cases (like applying some very specific custom logic or settings) contributor should submit patches to openstack::* classes

Fuel library includes custom modules as well as ones forked from upstream sources. Note that Modulefile (or metadata.json), if any exists, should be used in order to recognize either given module is forked upstream one or not. In case there is no Modulefile in module’s directory, the contributor may submit a patch directly to this module in Fuel library. Otherwise, he or she should submit patch to upstream module first, and once merged or +2 recieved from a core reviewer, the patch should be backported to Fuel library as well. Note that the patch submitted for Fuel library should contain in commit message the upstream commit SHA or link to github pull-request (if the module is not on stackforge) or Change-Id of gerrit patch.

=Fuel development environments=
===VirtualBox===
The quickest way to get started with Fuel is to set up an environment under VirtualBox. This is a great way start testing, for instance by using the nightly builds to try to verify, confirm or triage bugs. A quick-start script and instructions can be found at https://software.mirantis.com/quick-start/.

===Fuel ISO-build and development environment===
Setting up an environment to build the Fuel ISO will allow you to easily build against specific commits in order to do more complicated bug test/confirm/triage/fix work, or augmenting Fuel with new features. In addition to preparing the environment for ISO building, the steps necessary to prepare a dev environment for the other Fuel components (nailgun, astute and doc building) can be found there. Instructions for preparing the environment can be found at http://docs.mirantis.com/fuel-dev/develop/env.html.

===Fuel DevOps environment===
Setting up a Linux machine as a Fuel DevOps environment will allow you to automate the build/verify/test process, in addition to building the ISO and doing other Fuel-related development. Using libvirt, the devops environment is able to deploy multiple complete OpenStack environments on VMs. Instructions for preparing a devops environment can be found at http://docs.mirantis.com/fuel-dev/devops.html.

=Documentation=
There are two types of documentation in Fuel: [http://docs.mirantis.com/fuel-dev/ development] and [http://docs.mirantis.com/ user]. Source code of the development documentation can be found at [https://github.com/stackforge/fuel-web/tree/master/docs fuel-web repository]; users, at [https://github.com/stackforge/fuel-docs fuel-docs repository]. Both repositories use the [http://sphinx-doc.org/rest.html reStructuredText format]. Documentation is treated as source code, we use the same [[Fuel/How_to_contribute#First_steps|development process]] for updating documentation as we do for Fuel code.

=== Writing documentation ===
Keeping the Documentation up to date is hard and it is a critical part in every project. Fuel developers use the OpenStack [https://wiki.openstack.org/wiki/Documentation/DocImpact DocImpact] process to highlight code changes that need to be reflected in documentation. When a patchset should be documented or requires documentation changes, the reviewer should ask the commiter to add 'DocImpact' tag to their commit message, and makes sure the commit message provides enough information about the impact of the code change. When a commit with a DocImpact tag is merged, a new bug will be automatically created and assigned to the Fuel Documentation team. Technical writers then work with the commiter to reflect the commit in the documentation.

A bug generated from a DocImpact commit should be assigned to fuel-docs team until a technical writer picks it up. Technical writer collects information from the original commit author, and adds it to the right documentation areas. It's commit author's responsibility to provide an informative commit message in the first place, to answer technical writer's questions, and to review documentation commits that address that DocImpact bug.

=== Update and expand current documentation ===
Feel free to file bug if you find a problem with the documentation, and provide with a fix for it via same code contribution policy and procedures. There is [http://docs.mirantis.com/fuel-dev/develop/env.html#building-documentation building documentation page] to help with an environment setup for development documentation.

=== Writing release notes ===
==== New features ====
For every blueprint listed on the milestone page in Launchpad (e.g. https://launchpad.net/fuel/+milestone/5.1), find answers to the following questions:
* What is the visible impact of the change for a user?
* Was this previously documented as a Known Issue?

If it is a user visible improvement, add a New Features entry about it. Explain when this feature is important, how it can be enabled, and what are the current expectations and limitations.

If it was a Known Issue before, add a Resolved Issues entry.

==== Known and Resolved issues ====
Scan the release milestone page (e.g. [https://launchpad.net/fuel/+milestone/6.0 fuel 6.0]). Review all bugs targeted for the current milestone that satisfy one of the following conditions:
* "release-notes" tag
* Critical priority
* High priority and "customer-found" tag.

Sort the bugs table on the milestone page by Number from highest to lowest, so that you can see when you've reached the bugs that you have already reviewed. If bug status for the current milestone is Fix Committed or Fix Released, consider it for Resolved Issues list. Find answers to:
* Was this problem present in the previous GA release? (If it was, it will be targeted for at least one previous release series, with target milestone set to a maintenance release in that series.)
* How does one confirm if their environment is affected? (Identify specific configurations affected by the bug, include error messages or other symptoms that distinguish this bug from others.)
* If you're affected, how do you recover?
* What was the root cause and what had to be changed to resolve the issue?

If bug status for the current milestone is Won't Fix and it meets the criteria above, consider it for the Known Issues list. Questions to answer are:
* How does one confirm if their environment is affected?
* Is there a workaround? (How do you prevent this problem from affecting you?)
* What is the recovery procedure?

==== Reviewing the release notes ====
Responsibility for every section of release notes is assigned to a technical writer. The writer identifies features and bugs that have to be reflected in their section. For each feature and bug, the writer finds a responsible software engineer: either feature owner, or committer of the most recent bugfix.

If sufficient description (see question lists above) cannot be derived from bug description, blueprint description and specification, and commit messages of related commits, the writer contacts the engineer privately with a request for more information. When a description of a feature or bug is published in a gerrit review, the writer subscribes the engineer to that review, and once again privately sends a request with a link to the documentation review in Gerrit and instructions on how to identify specific documentation files and lines that this engineer must review (e.g. LP bug number, specific RST file, section header or label, etc.)

Fuel engineers must prioritize such information and review requests so that they are able to respond within one week during normal development process, and within one business day during the period between Hard Code Freeze and GA release. Standard escalation process is used when no response to request for information or request for review is received in time. Technical writer is responsible for tracking status of information and review requests and escalations for their section of release notes.

=Bugs=
===Verify bugs===
This is very easy to get started, but help here is actually very valuable. Especially when it comes to bugfixes which are relevant to only certain types of hardware. When bugfix is merged, bug is automatically moves to "Fix Committed" status. In Fuel, we set "Fix Released" status for a bug only when it was verified by someone else than a developer. You can simple check out [https://bugs.launchpad.net/fuel/+bugs?field.status%3Alist=FIXCOMMITTED all fix committed bugs] and start verifying them. It is required to be in fuel-bugs LP group to be able to change bug status. However you can leave a comment in bug report, and it will be enough for bug supervisors to change bug status. By your request, you can be included into fuel-bugs team to manage bugs yourself.

=== Test and report bugs ===
You can always build a developers ISO from master following instructions at http://docs.mirantis.com/fuel-dev/develop/env.html#building-the-fuel-iso. Test different deployment modes, and file bugs. Please do not hesitate to share your findings in IRC for any details you would want to know. Feel free to file bug in a format you want, however it would be great to see a bug in a certain format, with all details in place - to easily reproduce and save a time of developers. You can follow the following simple guidance, which a little bit extend official OpenStack [[Bugs]] page.

The bugs you report for Fuel can be either “Public” or “Private”. Private bugs become visible only for Dev, QA, and Support teams. The guidance is as follows:
* Normally you would want to report a “Public” bug
* However, if a bug contains any sensitive information, or security vulnerability information, please report a “Private” bug

Before creating a new bug, it's always a good idea to check if a bug like that already exists in the project, and update an existing bug instead of creating something that will later have to be discarded as a duplicate. However, it is important to make sure that both bugs are really related, dealing with a duplicate is much easier than untangling several unrelated lines of investigation mixed into one bug. Do not ever reopen bugs with generic catch-all titles like "Horizon crashed" or "HA doesn't work" that can cover whole ranges of root causes. Do not create new bugs with such titles, either: be as specific about the nature of the problem as you can.

Here is how you file a bug:
* Go to https://launchpad.net/fuel
* Select “Report a bug” on the right
* Fill in “Summary” field for a bug. It is going to be the headline. Please be descriptive but short and to the point.
** Bad - "Savanna doesn’t install"
** Good - "Savanna fails to install in HA mode when “Neutron with VLANs” network is selected"
* Enter “Further information”. This is a bug description. It should contain the following sections:
** Steps to reproduce
*** Bad: Run the deployment in HA configuration
*** Good:
***# Create a new cluster
***# Select HA mode, “Neutron with VLANs” for network, and choose to install Savanna
***# Leave defaults the rest of the settings. Run cluster installation
** Expected result:
*** Good: Savanna is deployed, up and running.
** Actual result:
*** Bad: Savanna fails to install.
*** Good: Savanna installation fails with the error message “xxx”. Please see the attached screenshot which shows the errors on “Logs” tab.
** Workaround
*** Bad: Don’t use “Neutron with VLANs”
*** Good: Switch network mode to “Neutron with GRE” for the deployment to complete successfully.
** Impact
*** Bad: Needs to be fixed.
*** Good: deployment cannot be completed, because customer requires us to implement a use case for Savanna and “Neutron with VLANs”. Changing configuration to “Neutron with GRE” is not as acceptable option here so this has to be addressed as soon as possible.
** Description of the environment. Provide enough relevant information:
*** Output of http://fuel-master-node:8000/api/version/
*** Operating System
*** Reference Architecture (HA / non-HA)
*** Network model (Nova-network, Neutron+VLAN, Neutron+GRE, etc.)
*** Related Projects installed (Savanna, Murano, Ceilometer)
* Select visibility for the bug under “This bug contains information that is” field. Either leave it as “Public” (default) or set it to “Private” per the above guidance
* Add attachments under “Extra Options” section
** Logs
** Diagnostic snapshot
** Screenshots
* After everything is entered, select the “Submit bug report” button

=== Confirm and triage bugs ===
Fuel project uses the same bug management practices as in other OpenStack projects. Please follow [[Bugs]] and [[BugTriage]] pages.

When triaging the bugs in the Fuel deployment process (as opposed to bugs in OpenStack components deployed by Fuel), use following additional criteria for bug importance:

* Critical = can't deploy anything and there's no trivial workaround; data loss; or security vulnerability
* High = specific hardware, configurations, or components are unusable and there's no workaround; or everything is broken but there's a workaround
* Medium = specific hardware, configurations, or components are working incorrectly; or is completely unusable but there's a workaround
* Low = minor feature is broken and can be fixed with a trivial workaround; or a cosmetic defect
* Wishlist = not a bug, either close as Won't Fix, or associate with a blueprint (create new blueprint if no existing blueprint covers the feature requested in the bug)

Additional guidelines for documentation bugs:

* Critical = following the instructions from documentation can cause outage or data loss
* High = documentation includes information that is not true, or instructions that yield the advertised outcome
* Medium = important information is missing from documentation (e.g. new feature description)
* Low = additional information would improve reader's understanding of a feature
* Wishlist = cosmetic formatting and grammar issues

Also Fuel uses different meaning for two bug states:

* Confirmed = somebody who is not submitter looked into the bug and made sure that it is properly triaged. The person must made sure that milestone, importance and assignee are properly set. Still the bug '''could be not''' reproduced or confirmed as genuine.
* Fix Released = somebody who is not assignee verified that the issue does not reproduce after it was fixed. The status is left for QA to mark bugs as verified after they are fixed.

When triaging bugs, please use the following routine:

# Review all '''New''' bugs.
# Assign each New bug to the release series under development (focus of development according to [https://launchpad.net/fuel/+series Fuel project timeline]) and set corresponding bug status and milestone for that release series.
# Consider changing the bug title to be more specific, replace generic statements like "deployment timeout exceeded" with description of the root cause (if identified) or symptops that are unique for that bug.
# Consider visibility status of the bug. If it contains sensitive information, set it to '''Private'''. If it describes a security vulnerability, set it to '''Private Security'''.
# If the bug does not contain sufficient information as described in [[Fuel/How_to_contribute#Test_and_report_bugs]], request the missing information, and set the bug status to '''Incomplete'''. Add a comment using the following template: ''We didn't receive required details in order to correctly troubleshoot the problem, so we are setting this bug into Incomplete state. Please provide reqested information and we will investigate this issue further. If you think it was incorrectly set to Incomplete, please comment in the bug''.
# If there is sufficient information on how to reproduce the bug and milestone, importance and assignee are properly set, set the status to '''Confirmed'''.
# If there is sufficient information to start implementing a fix, set the status to '''Triaged'''.
# If the root cause of the bug is identified, include it in the bug description.
# Review all '''Incomplete''' bugs
# If an Incomplete bug has new updates, use the same steps as described above for New bugs.
# If an Incomplete bug did not have any updates for '''4 weeks''', close it as '''Invalid'''. Add a comment using the following template: ''This bug was incomplete for more than 4 weeks. We cannot investigate it further so we are setting the status to Invalid. If you think it is not correct, please feel free to provide requested information and reopen the bug, and we will look into it further.''
# During '''Code Freeze''' for a release, review all bugs targeted at that release milestone, make sure it is updated daily by the assignee (or bug reporter if the bug is not assigned to a person).
# If you know that the bug is related to some blueprints, link this bug to these blueprints. To do this, go to corresponding blueprints page and hit 'Link a bug report' link.

Remember that every status change of a bug should be accompanied by a comment explaining the reason why the change was made.

Useful LaunchPad searches and tools:

* [https://bugs.launchpad.net/fuel/+bugs?field.status=NEW&field.project=FUEL New bugs]
* [https://bugs.launchpad.net/fuel/+bugs?field.searchtext=&orderby=-importance&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=&field.tags_combinator=ANY&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on&search=Search Incomplete bugs]
* [https://bugs.launchpad.net/fuel/+bugs?field.searchtext=&orderby=-importance&field.status%3Alist=NEW&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&assignee_option=none&field.assignee=&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=&field.tags_combinator=ANY&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on&search=Search Unassigned bugs]
* [https://bugs.launchpad.net/fuel/+bugs?field.searchtext=&field.status%3Alist=NEW&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&assignee_option=choose&field.assignee=fuel-library&field.bug_reporter=&field.bug_commenter=&field.subscriber=&field.structural_subscriber=&field.tag=&field.tags_combinator=ANY&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on&search=Search&orderby=milestone_name&start=0 Bugs in fuel-library, sorted by milestone]
* Launchpad Report Tool with bugs SLA by Mirantis https://launchpad-reports-summary.mirantis.com/project/bugs_lifecycle_report/6.1
* [https://addons.mozilla.org/en-US/firefox/addon/launchpad-helper/?src=cb-dl-created Launchpad Helper for Firefox]
* [https://chrome.google.com/webstore/detail/launchpad-helper/kjelakmoldahcakikjnfebiolcliigah?hl=en Launchpad Helper for Chrome]

=== Fix bugs ===
The most simple and easy way to contribute code into Fuel is to start with bug fixes. First of all, check out [https://bugs.launchpad.net/fuel/+bugs?field.status%3Alist=NEW&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.tag=low-hanging-fruit+&field.tags_combinator=ANY bugs with "low-hanging-fruit" tag]. You can also take any other bug. For easier management of bugs, Fuel uses multiple functional groups, and you will see bugs assigned to fuel-python, fuel-library, fuel-qa and other teams. Feel free to pick any of those. It is assigned to group, so group can get an email notification about a bug. Unless it is reassigned to particular engineer to work on that bug, then it means the bug is not taken by anyone yet, so feel free to take it. Bugs with assignee set to a person are not yet imply that someone works on them (but it should in ideal world). If it is even in "in progress" state, feel free to ping a person and ask if he really works on that bug, and feel free to assign yourself otherwise.

=== Backport bugfixes to stable release series ===
When you create a new bug with High or Critical priority or upgrade an existing bug, always check if this bug is present in the supported stable release series (at least one most recent stable release). If it is present there, target it to all affected series (even if you don't expect to be able to eventually backport a fix). If it's not present in stable releases, document that on the bug so that other people don't have to re-check.

When you propose a fix for a bug, always start with fixing it in the master branch. If the fix only applies to the stable branch and is not relevant in master, explain that in the commit message. For a bug targeted for stable release series, cherry-pick the fix commit onto the stable/x.x branch for each series it is targeted for, '''after''' the fix was merged to master (creating backports prematurely leads to possible inconsistency between master and stable/x.x versions of the same commit). Use the same Change-Id and topic (git review -t bug/<id>) to make it easier to track down all backports for that bug. You can also use "Cherry Pick To" button in Gerrit if the commit can be applied to the stable branch without conflicts.

When you approve a bugfix commit for master branch, use the information available so far on the bug and in the review request to review and maybe update backporting status of the bug. Is its priority high enough to need a backport? Is it targeted to all affected release series? Are there backport commits already? If yes, are they up to date with the commit that was merged to master?
* For all series where backport should exist and doesn't, create a backport review request yourself.
* For all other affected series, change bug status to Won't Fix and explain in bug comments.
* Do not merge commits into stable branches until it was merged into master, unless commit message explicitly documents why it doesn't apply to master.

When you find a bug in a stable release, do not simply target it to the next release in that maintenance release series. Instead, target it to the release designated as "current focus" on the [https://launchpad.net/fuel/ Fuel home page in Launchpad]. If it's a High or Critical priority bug, additionally target it to the releases series it was originally found in, and, if applicable, all release series in between.

If a bug is present in a maintenance release and not present in current release series, it should have one of the following statuses in the current release series:
* Incomplete if root cause analysis is not finished
* Fix Committed if root cause analysis shows that the bug is fixed by changes merged after the maintenance release the bug was found in
* Invalid if root cause analysis shows that the bug was introduced by a change specific to this maintenance release series

=Enhancements=
=== Propose enhancements ===
Enhancements for Fuel are proposed using the mechanism of blueprints. The process is very similar to the bugs:
* Select "Register a Blueprint". Typically, you’ll find this link on the right side of the launchpad page.
** Enter a Name for the blueprint. If possible, include the name of the sub-component of fuel as the first word of the name (or just fuel for a broader blueprint), then a very short descriptor using dashes between words. For example, [https://blueprints.launchpad.net/fuel/+spec/ui-declarative-wizard ui-declarative-wizard] or [https://blueprints.launchpad.net/fuel/+spec/fuel-stop-deployment fuel-stop-deployment].
** Enter a Title for the blueprint. The title should describe the feature as clearly as possible in up to 70 characters. This title is displayed in every feature list or report.
* Enter a Summary of the blueprint.
** Summarize the idea of the blueprint in a user story. If you have an existing user story in a public document, include the link.
** If a public document isn’t yet created and the user story is extensive, you can use https://etherpad.openstack.org/ to document the full user story. You can use the Name that you used for the blueprint as the Pad name.
* Propose your blueprint by selecting the milestone in which you plan to complete the blueprint.
* You won’t need to enter any other fields, so now just choose Register Blueprint.

If you’re just entering a new idea without a design, you can leave the specification URL field blank. If you have a design, here is how you get it published and approved:
* Upload a design specification in the "specs/<release>" folder in fuel-specs
** e.g. [https://review.openstack.org/96429 access-control-master-node]
** it should be based on the [http://git.openstack.org/cgit/stackforge/fuel-specs/tree/specs/template.rst template], see the instructions in the template for more details
** get it reviewed by submitting your patch using Gerrit, in the usual way: [http://docs.openstack.org/infra/manual/developers.html#development-workflow Development Workflow]
** at the end of each release, non-completed specs will be removed
** you need to re-submit for the following release, should the blueprint slip
* Once your design specification has been approved and merged into fuel-specs git repo:
** Update your blueprint's specification URL to point to the design specification in fuel-specs git
** Note, this should link to git (as in the template link above), not to the gerrit change

When done, drop an email about the new blueprint to the mailing list to attract attention of Fuel developers, who are likely to provide early feedback and request additional information.

Please read [[Blueprints|OpenStack Blueprints]] page for detailed definition of blueprints and their lifecycle.

=== Implement new features ===
To implement a new feature, you will need to address the following:
# File a blueprint, with all required feature details, as described in [[Fuel/Where_to_contribute#Propose_enhancements|Propose enhancements]] section above and on [[Blueprints]] page
# PTL or core developers of Fuel should approve a blueprint, when all required information is provided.
# Please no active contribution unless there is an agreement on design. To get an approval, you can discuss a feature in the mailing list.
# Every new feature must not break other features, and whether work fine with features we had before, or be disabled in combination with features which do not work with the new one. Short list of features to check compatibility with:
#* All distributions (currently CentOS & Ubuntu)
#* Networking (nova-network, Neutron)
#* Combination with other roles on a node. For example, MongoDB role should not be placed with Ceph.
#* Disk partitioning (will it use existing partition or new one needs to be created?)
#* Network verification
# Other requirements for features:
#* logging of services should be done via rsyslog for Fuel Library contributions
#* implementation should be scalable, or otherwise limitations should be clearly stated in a design doc. It is unacceptable to degrade scalability and performance of already existing features.
# Start implementation and propose code changes frequently and by small pieces. This is the only way to successful merge - it is unrealistic to review changesets with 1000+ lines of code.
# Every feature should have a set of tests:
#* [https://github.com/stackforge/fuel-web/tree/master/nailgun/ui_tests CasperJS] and [https://github.com/stackforge/fuel-main/tree/master/fuelweb_ui_test Selenium] tests are for Fuel Web UI
#* [https://github.com/stackforge/fuel-web/tree/master/nailgun/nailgun/test Nailgun uses Python Unit Test framework] for unit and integration tests
#* [https://github.com/stackforge/fuel-astute/tree/master/spec Astute uses RSpec] for unit and integration tests
#* Consider extending [http://docs.mirantis.com/fuel-dev/develop/ostf_contributors_guide.html OSTF tests] for implemented feature
#* [https://github.com/stackforge/fuel-main/tree/master/fuelweb_test System tests], which use [http://docs.mirantis.com/fuel-dev/devops.html Devops framework], are run against an ISO, i.e. when all components are integrated. System tests last for up to 6-8 hours and run every night. As these tests are ran against integrated environment, their result represents a higher interest. Every large feature which has pieces in more than one Fuel component must be covered by system tests.
#* If feature requires special hardware, or special deployment setup, then existing CI should be extended with such hardware and required changes into devops framework. It should be discussed with the Fuel community in mailing list in order to come up with a joint decision.
# Documentation. New features must be well documented in both [[Fuel/Where_to_contribute#Documentation|user and development documentation]].
# Please propose changes for a feature before feature freeze date. Otherwise, changes won't be even reviewed before code freeze, and we will not let changes to land into master before milestone proposed branch is created. This will allow Fuel team to concentrate on stabilization of the current release, and put all the force into squashing bugs. Rules are similar to core OpenStack projects, see [[FeatureFreeze]] and [[ReleaseCycle]].

=Tests=
=== Unit and integration tests for Nailgun ===
[https://github.com/stackforge/fuel-web/tree/master/nailgun/nailgun/test Nailgun uses Python Unit Test framework] for unit and integration tests. Help in writing test is highly appreciated - we always want better coverage! Please checkout [http://docs.mirantis.com/fuel-dev/develop/env.html#setup-for-nailgun-unit-tests how to setup development environment for Nailgun tests], and follow [[Fuel/How_to_Test_Your_Code|How to Test Your Code]] guidance.

= Code review rules =
When contributing code changes to Fuel, please use the [http://docs.openstack.org/infra/manual/developers.html#development-workflow development workflow] and follow [[Fuel/Code_Review_Rules|Fuel specific rules and recommendations]].