Difference between revisions of "Fuel/Library and Upstream Modules"

Revision as of 16:06, 29 July 2015

Fuel Library and Upstream Modules

The fuel-library project leverages upstream puppet modules as dependencies to provide configuration management for the software being installed by Fuel. In order to align more closely with the upstream OpenStack puppet community and other opensource puppet modules that are currently being used, we have decided to leverage librarian-puppet-simple to manage the inclusion of upstream modules into fuel-library for testing and building the fuel-library package. This work was added under the fuel-puppet-librarian blueprint.

librarian-puppet-simple integration

As part of the fuel-library code base, we have added a Puppetfile that will be used to designate the inclusion of an upstream puppet module. See change I5e628f159d2d11121af741bcc1218f292cd2b96e for the specific changes made to integrate librarian-puppet-simple. Librarian-puppet-simple only supports git references and does not do any dependency resolution. So the developer including or updating a module is responsible for ensuring all dependancies are properly captured. Librarian-puppet-simple will be used during the noop test runs and as part of the fuel-library package building process to pull down the required upstream modules for our deployment needs.

Puppetfile

The Puppetfile that has been added to fuel-library/deployment contains the upstream modules, a git repository and a specific reference that should be used to populate the fuel-library/deployment/puppet folder for development, testing and build purposes. It should be noted that the reference used should either be a specific git commit hash or a tag. If you use a branch, you may run into a chance that the branch has changed between builds resulting in unexpected code changes.

Puppetfile Example

Below is a puppet file example for librarian-puppet-simple. It should be noted that this format is slightly different than the regular librarian-puppet format in that the mod name is not the vendor name but rather the folder name that we will put the module into. For example 'puppetlabs-stdlib' should be 'stdlib'.

#!/usr/bin/env ruby
#^syntax detection
# See https://github.com/bodepd/librarian-puppet-simple for additional docs
#
# Important information for fuel-library:
# With librarian-puppet-simple you *must* remove the existing folder from the
# repo prior to trying to run librarian-puppet as it will not remove the folder
# for you and you may run into some errors.
#
############
# Examples #
############
# From git repo
#  mod "puppetlabs/stdlib",
#    :git => 'https://github.com/puppetlabs/puppetlabs-stdlib.git',
#    :ref => '4.6.x'
#
# From tarbal
#  mod "puppetlabs/stdlib",
#    :tarbal => 'https://forgeapi.puppetlabs.com/v3/files/puppetlabs-stdlib-4.6.0.tar.gz'
#
mod 'stdlib':
  :git => 'https://review.fuel-infra.org/p/puppet-modules/puppetlabs-stdlib.git',
  :ref => '4.6.0'
mod 'concat'
  :git => 'https://review.fuel-infra.org/p/puppet-modules/puppetlabs-concat.git',
  :ref => '1.2.4'

update_modules.sh

A script named update_modules.sh has been added to the fuel-library/deployment folder that can be used to pull down all the required modules via librarian-puppet-simple. This script can also be passed parameters to leverage bundler for the installation of librarian-puppet-simple for development needs. This script should be used to ensure that a development environment is up to date with the required upstream modules.

Noop modifications for librarian-puppet-simple

In order to be able to run our noop tests to be able to run, we need to pull down the upstream modules at the specified version. The noop test runner has been updated to optionally call librarian-puppet prior to executing tests to populate the required puppet modules. fuel_noop_tests.rb has been updated with a -u option that will call librarian-puppet to pull down upstream modules. The fuel_noop_tests.rb calls our update_modules.sh script that has been created to manage the librarian-puppet-simple tasks.

fuel-library rpm build changes

As part of the fuel-library RPM build process, we have updated the RPM spec to pull down upstream modules by leveraging the update_modules.sh script so they are included as part of the fuel-library RPM. This change is part of I5e628f159d2d11121af741bcc1218f292cd2b96e and will attempt to run update_modules.sh (if it exists) during the RPM build so that the upstream modules are bundled into the rpm for deployment. It should be noted that this update_modules.sh run does *not* use bundler so ruby21-rubygem-librarian-puppet-simple has been added as a BuildRequires for the fuel-library package.

Best Practices

Puppetfile Module Definitions

Module definitions in the Puppetfile *SHOULD*:

• Reference a tag for the :ref
• Reference the official fuel-infra.org mirror
• If fuel-infra.org mirror is not a possibility (and there is a very good reason why), reference the official upstream repository

Module definitions in the Puppetfile *SHOULD NOT*:

• Reference a sha1 hash for the :ref
• Reference a branch for the :ref
• Reference a personal fork git repository

Examples

Best

mod 'stdlib':
  :git => 'https://review.fuel-infra.org/p/puppet-modules/puppetlabs-stdlib.git',
  :ref => '4.6.0'

mod 'concat'
  :git => 'https://review.fuel-infra.org/p/puppet-modules/puppetlabs-concat.git',
  :ref => '1.2.4'

Good

mod 'stdlib':
  :git => 'https://github.com/puppetlabs/puppetlabs-stdlib.git',
  :ref => '4.6.0'

mod 'concat'
  :git => 'https://review.fuel-infra.org/p/puppet-modules/puppetlabs-concat.git',
  :ref => '1.2.4'

Bad

mod 'stdlib':
  :git => 'https://github.com/Yelp/puppetlabs-stdlib.git',
  :ref => 'master'

mod 'concat':
  :git => 'https://github.com/puppetlabs/puppetlabs-concat.git',
  :ref => '1.2.x'

Workflows

New Module

To add a new upstream puppet module into the system, you will need to run through several steps. Change I4198b56e843d5b4d5f43b4c990c8af07de14524b is an example of a new module being added into the fuel-library code base.

1. Request a fuel-infra repository mirror for the new upstream module (and any required dependencies) so that CI can be used to leverage this mirror for building and testing. See bug 1477320 as an example of this type of request.
2. Update Puppetfile with required module information. You will need to add the module name, git repository and specific reference to the tag to be used.
3. Add puppet/<modulename> to the fuel-library/deployment/.gitignore file to ensure it won't be checked in
4. Submit for review

Existing Module Migration

For existing modules, we need to remove them from the code base as part of the migration. Change I16fd42f666cf142c5efc7362ed0a3b9bbcded249 provides an example of the migration of an existing module within the fuel-library code base.

1. Request a fuel-infra repository mirror for the module being migrated.
2. Update Puppetfile with the required module information.
3. Remove existing <modulename> from fuel-library/deployment/puppet/
4. Add puppet/<modulename> to the fuel-library/deployment/.gitignore
5. Submit for review

Updating Existing Module

Updating a module is should be a simple update to a reference in a Puppetfile.

1. Update Puppetfile with new reference
2. Submit for review

Custom Upstream Module Changes

Ideally, all upstream module changes should occur in the upstream and we should pull them down via an updated git tag reference in our Puppetfile. Unfortunately since that isn't always option, this is where the upstream puppet module on fuel-infra comes in. We can create a custom branch based off an upstream tag to contain changes that might not be upstream compatible or are needed in a pinch. It should be noted that this is not an acceptable general workflow and should only be used in rare cases with specific approval. For this type of change, a change request will need to be made against the fuel-infra mirror of the module and the git reference should be updated in our Puppetfile.

1. Submit update to fuel-infra mirror targeting a custom branch.
2. Once review has been approved, create custom tag to reference the change is not a standard upstream version. For example 6.0.0 -> 6.0.0-MOS-1
3. Update Puppetfile with new git reference, e.g. 6.0.0-MOS-1
4. Submit for review