Fuel/Library and Upstream Modules

Fuel Library and Upstream Modules
The fuel-library project leverages upstream puppet modules as dependencies to provide configuration management for the software being installed by Fuel. In order to align more closely with the upstream OpenStack puppet community and other opensource puppet modules that are currently being used, we have decided to leverage librarian-puppet-simple to manage the inclusion of upstream modules into fuel-library for testing and building the fuel-library package. This work was added under the fuel-puppet-librarian blueprint.

With the choice to leverage a Puppetfile and librarian-puppet-simple to manage the inclusion of upstream modules, we are specifically requiring that only *tags* be used within the Puppetfile so that we are pinning to a specific version of an upstream module. This ensures that we only include an fixed version for any given version of the Puppetfile within fuel-library. All changes to the Puppetfile will trigger the Fuel CI to test the upstream versions of the modules to ensure we are not breaking anything when we upgrade from one version to the next (e.g. 1.1.0 -> 1.1.1).

librarian-puppet-simple integration
As part of the fuel-library code base, we have added a Puppetfile that will be used to designate the inclusion of an upstream puppet module. See change I5e628f159d2d11121af741bcc1218f292cd2b96e for the specific changes made to integrate librarian-puppet-simple. Librarian-puppet-simple only supports git references and does not do any dependency resolution. So the developer including or updating a module is responsible for ensuring all dependancies are properly captured. Librarian-puppet-simple will be used during the noop test runs and as part of the fuel-library package building process to pull down the required upstream modules for our deployment needs.

Puppetfile
The Puppetfile that has been added to fuel-library/deployment contains the upstream modules, a git repository and a specific reference that should be used to populate the fuel-library/deployment/puppet folder for development, testing and build purposes. It should be noted that the reference used should be a specific git tag reference. We will not allow a branch as it would allow the version of the upstream module to change between builds of the same fuel-library code. We will be validating the Puppetfile per the guidelines set in this page (pending merge of I623929df3a309110d26e166ac9067c715b14188b).

Puppetfile Example
Below is a puppet file example for librarian-puppet-simple. It should be noted that this format is slightly different than the regular librarian-puppet format in that the mod name is not the vendor name but rather the folder name that we will put the module into. For example 'puppetlabs-stdlib' should be 'stdlib'.

# # # # mod 'stdlib', :git => 'https://review.fuel-infra.org/p/puppet-modules/puppetlabs-stdlib.git', :ref => '4.6.0' mod 'concat', :git => 'https://review.fuel-infra.org/p/puppet-modules/puppetlabs-concat.git', :ref => '1.2.4'
 * 1) !/usr/bin/env ruby
 * 2) ^syntax detection
 * 3) See https://github.com/bodepd/librarian-puppet-simple for additional docs
 * 1) Important information for fuel-library:
 * 2) With librarian-puppet-simple you *must* remove the existing folder from the
 * 3) repo prior to trying to run librarian-puppet as it will not remove the folder
 * 4) for you and you may run into some errors.
 * 1) Examples #
 * 2) From git repo
 * 3)  mod "stdlib",
 * 4)    :git => 'https://github.com/puppetlabs/puppetlabs-stdlib.git',
 * 5)    :ref => '4.6.x'
 * 1)    :git => 'https://github.com/puppetlabs/puppetlabs-stdlib.git',
 * 2)    :ref => '4.6.x'
 * 1) From tarbal
 * 2)  mod "stdlib",
 * 3)    :tarbal => 'https://forgeapi.puppetlabs.com/v3/files/puppetlabs-stdlib-4.6.0.tar.gz'

update_modules.sh
A script named update_modules.sh has been added to the fuel-library/deployment folder that can be used to pull down all the required modules via librarian-puppet-simple. This script can also be passed parameters to leverage bundler for the installation of librarian-puppet-simple for development needs. This script should be used to ensure that a development environment is up to date with the required upstream modules.

remove_modules.sh
A script named remove_updates.shhas been added to the fuel-library/deployment folder that can be used to remove the librarian managed modules from the current working directory. Alternatively, you can also perform a 'git clean -d -x -f' in the your working copy of fuel-library to remove any unmanaged files.

Noop modifications for librarian-puppet-simple
In order to be able to run our noop tests to be able to run, we need to pull down the upstream modules at the specified version. The noop test runner has been updated to optionally call librarian-puppet prior to executing tests to populate the required puppet modules. fuel_noop_tests.rb has been updated with a -u option that will call librarian-puppet to pull down upstream modules. The fuel_noop_tests.rb calls our update_modules.sh script that has been created to manage the librarian-puppet-simple tasks.

fuel-library rpm build changes
As part of the fuel-library RPM build process, we have updated the RPM spec to pull down upstream modules by leveraging the update_modules.sh script so they are included as part of the fuel-library RPM. This change is part of I5e628f159d2d11121af741bcc1218f292cd2b96e and will attempt to run update_modules.sh (if it exists) during the RPM build so that the upstream modules are bundled into the rpm for deployment. It should be noted that this update_modules.sh run does *not* use bundler so ruby21-rubygem-librarian-puppet-simple has been added as a BuildRequires for the fuel-library package. Also to provide a mechanism that does not require internet access to build the fuel-library packages, the fuel-main scripts were updated to support a tarbal instead of leveraging the Puppetfile as part of I02e02ef62bcc65b96fe17806105394302b2a92d1

Puppetfile Module Definitions
Module definitions in the Puppetfile *MUST*:
 * Reference a tag for the :ref
 * Reference an official fuel-infra.org mirror (either review.fuel-infra.org or github.com/fuel-infra)
 * If a fuel-infra mirror is not a possibility (and there is a very good reason why, e.g. bug 1479017), reference the official upstream repository

Module definitions in the Puppetfile *MUST NOT*:
 * Reference a sha1 hash for the :ref
 * Reference a branch for the :ref
 * Reference a personal fork git repository

Examples
Good

mod 'stdlib', :git => 'https://review.fuel-infra.org/p/puppet-modules/puppetlabs-stdlib.git', :ref => '4.6.0'

mod 'concat', :git => 'https://github.com/fuel-infra/puppetlabs-concat.git', :ref => '1.2.4'

Bad

mod 'stdlib', :git => 'https://github.com/Yelp/puppetlabs-stdlib.git', :ref => 'master'

mod 'concat', :git => 'https://github.com/puppetlabs/puppetlabs-concat.git', :ref => '1.2.x'

New Module
To add a new upstream puppet module into the system, you will need to run through several steps. Change I4198b56e843d5b4d5f43b4c990c8af07de14524b is an example of a new module being added into the fuel-library code base.


 * 1) Request a fuel-infra repository mirror for the new upstream module (and any required dependencies) so that CI can be used to leverage this mirror for building and testing.  See bug 1477320 as an example of this type of request.
 * 2) Update Puppetfile with required module information. You will need to add the module name, git repository and specific reference to the tag to be used.
 * 3) Add "puppet/ " to the fuel-library/deployment/.gitignore file to ensure it won't be checked in
 * 4) Submit for review

Existing Module Migration
For existing modules, we need to remove them from the code base as part of the migration. Change I16fd42f666cf142c5efc7362ed0a3b9bbcded249 provides an example of the migration of an existing module within the fuel-library code base.


 * 1) Request a fuel-infra repository mirror for the module being migrated.
 * 2) Update Puppetfile with the required module information.
 * 3) Remove existing from fuel-library/deployment/puppet/
 * 4) Add "puppet/ " to the fuel-library/deployment/.gitignore
 * 5) Submit for review

Updating Existing Module
Updating a module is should be a simple update to a reference in a Puppetfile.
 * 1) Update Puppetfile with git tag reference
 * 2) Submit for review

Custom Upstream Module Changes
All upstream module changes should occur in the upstream and we should pull them down via an updated git tag reference in our Puppetfile.

The rules for using a custom upstream module change are as follows:
 * 1) A fix for a *Critical* bug that has spent more than 7 days in an upstream review (since first patchset), relaxed to 24h between SCF and RTM.
 * 2) A fix for a *High* or *Critical* bug that received -2 in upstream review but +2 from 2 fuel-library core reviewers.

If your change does not fall in to these categories, you should be looking at alternative ways to address the issue that does not require upstream modification. If something is related to a technical decision for fuel-library, see if there is an alternative within fuel-library that can be leveraged to address the issue or fix it in the upstream.

If your change meets the above rules for a custom change on fuel-infra, you can use the following workflow.


 * 1) Submit the patch upstream module
 * 2) Above criteria for custom patch must be met
 * 3) Cherry-pick the commit to a fuel-custom branch on review-infra (i.e 6.0.0-MOS)
 * 4) Have change reviewed and approved by the core reviewers for the module on fuel-infra (e.g. fuel-library and MOS-Puppet folks)
 * 5) Once review has been approved, have a custom tag to reference the change is not a standard upstream version. For example 6.0.0 -> 6.0.0-MOS-1
 * 6) Create fuel-library patch with the updated Puppetfile with new git reference, e.g. 6.0.0-MOS-1
 * 7) Submit for review in fuel-library

It is expected that when the upstream patch ultimately lands, that the custom patch be reverted out of fuel-infra and the landed patch be used.