Jump to: navigation, search

Difference between revisions of "Oslo/CreatingANewLibrary"

(Creating an Initial Repository for Import: remove obsolete info about using the oslo lp project)
(update the instructions to refer to the new Project Creator's Guide in the infra manual)
Line 1: Line 1:
 
== Before You Start ==
 
== Before You Start ==
  
This is a long document. It's long because it has to be, not because we want it to be. If you follow it, everything will be fine.
+
This is a long process. It's long because it has to be, not because we want it to be. If you follow it, everything will be fine.
  
 
It is important that you perform all of the steps, in the order they are given here. Don't skip any steps. Don't try to do things in parallel. Don't jump around.
 
It is important that you perform all of the steps, in the order they are given here. Don't skip any steps. Don't try to do things in parallel. Don't jump around.
 +
 +
== Project Creator's Guide ==
 +
 +
The [http://docs.openstack.org/infra/manual/creators.html Project Creator's Guide] includes many of the details for creating a repository. It is maintained by the Infrastructure team, and should be up to date with the current process.
 +
 +
Read that guide before continuing. Some steps below will refer to sections of that guide, and you will need to use both documents together to complete this process.
  
 
== The Spec ==
 
== The Spec ==
Line 11: Line 17:
 
=== Blueprint ===
 
=== Blueprint ===
  
First, create a blueprint in the oslo-incubator repository on launchpad. Name the blueprint gradate-oslo-foo, replacing foo with the name of the library.
+
First, create a blueprint in the oslo-incubator repository on launchpad. Name the blueprint graduate-oslo-foo, replacing foo with the name of the library.
  
 
=== Spec ===
 
=== Spec ===
Line 19: Line 25:
 
The spec template includes questions that come later in this document, for which you will need to provide an answer at this point. This is the one exception to the "don't read ahead or skip around" rule mentioned earlier. Collect all of the information you need to fill in the spec.
 
The spec template includes questions that come later in this document, for which you will need to provide an answer at this point. This is the one exception to the "don't read ahead or skip around" rule mentioned earlier. Collect all of the information you need to fill in the spec.
  
== Creating an Initial Repository for Import ==
+
== Choosing a Name ==
 
 
There are three sources for new Oslo libraries:
 
 
 
# Incubated code that is graduating.
 
# A brand new library.
 
# An existing library that we are adopting.
 
 
 
These subsections cover incubated code and new libraries. Existing libraries can skip ahead to "Importing the Repository into the CI System" below.
 
  
=== Choosing a Name ===
+
In addition to the rules described in the [http://docs.openstack.org/infra/manual/creators.html Project Creator's Guide], there are currently three naming schemes used for Oslo libraries:
 
 
There are currently three naming schemes used for Oslo libraries:
 
 
# oslo.something - Used for production runtime dependencies of OpenStack projects.  Using the oslo. namespace for any other type of package will cause conflicts due to some limitations in the way Python and Pip handle namespace packages.
 
# oslo.something - Used for production runtime dependencies of OpenStack projects.  Using the oslo. namespace for any other type of package will cause conflicts due to some limitations in the way Python and Pip handle namespace packages.
 
# oslosomething - Used for non-production or non-runtime dependencies of OpenStack projects.  If you're planning to use a name like this, please discuss it with the Oslo team first - we aren't sure we like this name scheme and may suggest an alternative.
 
# oslosomething - Used for non-production or non-runtime dependencies of OpenStack projects.  If you're planning to use a name like this, please discuss it with the Oslo team first - we aren't sure we like this name scheme and may suggest an alternative.
 
# something - Used for libraries that may be generally useful outside of OpenStack.  Note: you should verify that the name you are proposing is not in use on the Python Package Index (https://pypi.python.org/pypi).
 
# something - Used for libraries that may be generally useful outside of OpenStack.  Note: you should verify that the name you are proposing is not in use on the Python Package Index (https://pypi.python.org/pypi).
  
=== Creating a Library from Scratch ===
+
== Graduating a Library from the Incubator ==
  
==== Git Repository ====
+
There are three sources for new Oslo libraries:
  
All OpenStack projects should use one of our [https://pypi.python.org/pypi/cookiecutter cookiecutter] templates for creating an initial repository to hold the source for the project.
+
# Incubated code that is graduating.
 
+
# A brand new library.
  pip install cookiecutter
+
# An existing library that we are adopting.
 
 
Choose the right template:
 
  
The template in openstack-dev/cookiecutter is suitable for libraries not sharing the oslo namespace package.
+
Most of this document covers incubated code. Brand new and existing libraries can skip ahead to "Importing the Repository into the CI System" below.
 
 
  cookiecutter https://git.openstack.org/openstack-dev/cookiecutter
 
 
 
The template in openstack-dev/oslo-cookiecutter should be used for libraries that are sharing the oslo namespace package.
 
 
 
  cookiecutter https://git.openstack.org/openstack-dev/oslo-cookiecutter
 
 
 
The output will be a new directory containing a project skeleton, ready to have your real project files added. This directory needs to be turned into a git repository that can be imported into the CI system in later steps. The simplest way to do that is to push a copy to github or other public git server.
 
 
 
  cd projectname
 
  git init .
 
  git add .
 
  git commit -m "set up project with cookiecutter template"
 
  # add your project files
 
  git remote add origin git@github.com:username/projectname.git
 
  git push -u origin master
 
 
 
It is best if the repository is not imported into the CI system until all of the relevant tests pass, since failing tests prevent other changes from being merged.
 
 
 
==== Sync Required Tools from Incubator ====
 
 
 
Some of the tools common to all Oslo libraries reside in the oslo-incubator. These need to be synced into the new repository before it is imported.
 
 
 
# clone a copy of the oslo-incubator repository
 
# Add the appropriate modules and scripts to /path/to/newproject/openstack-common.conf
 
# In oslo-incubator, run: ./update.sh /path/to/newproject
 
# In /path/to/newproject, add the new files and commit any changes
 
 
 
=== Graduating a Library from the Incubator ===
 
  
 
''Graduating a library is a multi-step process. It isn't complicated, but there are a lot of details that are easy to miss. Please tread carefully.''
 
''Graduating a library is a multi-step process. It isn't complicated, but there are a lot of details that are easy to miss. Please tread carefully.''
  
The goal with incubated code is to extract it from the incubator with all of its history, and publish the results as a new library. A lot of that process is automated, but there are inevitably some manual steps -- especially to ensure that the tests work properly.
+
The goal with incubated code is to extract it from the incubator with all of its history, and publish the results as a new library. This means that a new repository must be created outside of gerrit, and then imported, to avoid having to re-review all of the existing changesets. A lot of that process is automated, but there are inevitably some manual steps -- especially to ensure that the tests work properly.
 
 
==== Identifying the Primary Maintainer for the New Library ====
 
 
 
Each graduated library needs a primary maintainer. That may be the same person who started the code, or it may be someone who is taking over maintenance duties.
 
 
 
Talk with the Oslo team about who the person is, and make sure the team agrees to their leadership on the library before proceeding.
 
 
 
==== Identifying the Security Contact for the New Library ====
 
 
 
Each graduated library needs a contact for the OpenStack Vulnerability Management team. It may be the same as the primary maintainer, an existing Oslo team member who helps with security issues, or it may be someone else on the development team for the new library.
 
  
Talk with the Oslo and Vulnerability management teams about who the person is, and make sure they agree to their participation before proceeding.
+
Perform the steps in this section before following the directions in the [http://docs.openstack.org/infra/manual/creators.html Project Creator's Guide], then use the repository you create as the "upstream" value when adding the project to gerrit.
  
Have the Oslo PTL add the person to the https://launchpad.net/~oslo-coresec team in launchpad.
+
=== Updating the Status in the Incubator ===
 
 
==== Updating the Incubator ====
 
  
 
'''Update oslo-incubator/MAINTAINERS'''
 
'''Update oslo-incubator/MAINTAINERS'''
Line 111: Line 64:
 
We want this done in incubator before the new repo is created so we can review the changes, but don't have to pull in logging dependencies that will just be removed immediately.  This shouldn't be a problem for anyone continuing to use the incubator versions of the files either.
 
We want this done in incubator before the new repo is created so we can review the changes, but don't have to pull in logging dependencies that will just be removed immediately.  This shouldn't be a problem for anyone continuing to use the incubator versions of the files either.
  
==== Extracting a Clean Copy of the History ====
+
=== Extracting a Clean Copy of the History ===
  
 
Start by checking out a copy of the oslo-incubator to use for running the tool:
 
Start by checking out a copy of the oslo-incubator to use for running the tool:
Line 131: Line 84:
 
The script is fairly I/O intensive, and will take a while to run. It logs all of its output to /tmp/oslo-graduate.XXXX/output.log, and other scratch files are kept in the same directory.
 
The script is fairly I/O intensive, and will take a while to run. It logs all of its output to /tmp/oslo-graduate.XXXX/output.log, and other scratch files are kept in the same directory.
  
==== Manual Fixes ====
+
=== Manual Fixes ===
  
 
It is not possible to completely automate the process of reorganizing the library code and ensuring that it will work, so there are several manual tasks to be performed next.
 
It is not possible to completely automate the process of reorganizing the library code and ensuring that it will work, so there are several manual tasks to be performed next.
Line 153: Line 106:
 
* Add new library dependencies that were not already in use by the incubated code
 
* Add new library dependencies that were not already in use by the incubated code
  
==== Sync Required Tools from Incubator ====
+
=== Sync Required Tools from Incubator ===
  
 
Some of the tools common to all Oslo libraries reside in the oslo-incubator. These need to be synced into the new repository before it is imported.
 
Some of the tools common to all Oslo libraries reside in the oslo-incubator. These need to be synced into the new repository before it is imported.
Line 162: Line 115:
 
# In /path/to/newproject, add the new files and commit any changes
 
# In /path/to/newproject, add the new files and commit any changes
  
==== Publish Your Results ====
+
=== Publish Your Results ===
  
 
When you have a clean version of the library, publish it to a public git repository (e.g., on github). This copy of the project history will be used in subsequent steps.
 
When you have a clean version of the library, publish it to a public git repository (e.g., on github). This copy of the project history will be used in subsequent steps.
  
== Receive Oslo Team Sign-off ==
+
=== Receive Oslo Team Sign-off ===
  
 
Before proceeding to import the library into the OpenStack CI system, post to the openstack-dev mailing list asking for the Oslo team to review the public repository.
 
Before proceeding to import the library into the OpenStack CI system, post to the openstack-dev mailing list asking for the Oslo team to review the public repository.
  
== Importing the Repository into the CI System ==
+
== Adding the Repository to the CI System ==
 
 
=== openstack-infra/project-config ===
 
 
 
These instructions are based on the [http://ci.openstack.org/stackforge.html steps for creating a new Stackforge project] with some variations, and apply to changes made in the [http://git.openstack.org/cgit/openstack-infra/project-config openstack-infra/project-config] git repository.
 
 
 
All of the changes described in this section should be submitted together as one patchset. Refer to https://review.openstack.org/#/c/124103/ for an example.
 
 
 
==== Add the project to the master project list ====
 
 
 
Edit gerrit/projects.yaml to add a new section like:
 
 
 
  - project: openstack/project-name
 
    description: Latest and greatest cloud stuff.
 
    groups:
 
      - oslo
 
    upstream: git://github.com/awesumsauce/project-name.git
 
 
 
The projects in the file need to be listed in alphabetical order.
 
 
 
Provide a very brief description of the library, especially if it is not named oslo.something-obvious
 
  
Set the "groups" to "oslo" to use https://launchpad.net/oslo for tracking bugs and milestones. For libraries with existing launchpad projects, set the "groups" to the name of the launchpad project.
+
Follow the steps in the [http://docs.openstack.org/infra/manual/creators.html Project Creator's Guide] for importing your new repository, using the repository you have created as the "upstream" value when adding the project to gerrit.
  
Set the "upstream" URL to the repository created earlier. If the existing library being imported is already on stackforge, the import can be handled by renaming the repository instead of using the upstream URL, so leave that line out.
+
== Add the Project to the Oslo Library List ==
 
 
==== Add Gerrit permissions ====
 
 
 
Each project should have 2 groups. The first, "project-name-core", is the normal core group, with permission to +2 changes. The second, "project-name-release" is a small group of the primary maintainers with permission to push tags to trigger releases.
 
 
 
Create gerrit/acls/openstack/project-name.config:
 
 
 
<pre><nowiki>
 
[access "refs/heads/*"]
 
label-Code-Review = -2..+2 group project-name-core
 
label-Workflow = -1..+1 group project-name-core
 
abandon = group project-name-core
 
 
 
[access "refs/tags/*"]
 
pushSignedTag = group project-name-release
 
 
 
[receive]
 
requireChangeId = true
 
requireContributorAgreement = true
 
 
 
[submit]
 
mergeContent = true
 
</nowiki></pre>
 
 
 
See other oslo files in the same directory for examples.
 
 
 
==== Add Basic Jenkins Jobs ====
 
 
 
Establish the standard Python jobs, including publishing releases to PyPI and pre-release tarballs to tarballs.openstack.org.
 
 
 
Edit jenkins/jobs/projects.yaml to add, for example:
 
 
 
- project:
 
    name: project-name
 
    node: 'bare-precise || bare-trusty'
 
    tarball-site: tarballs.openstack.org
 
    doc-publisher-site: docs.openstack.org
 
    jobs:
 
      - python-jobs
 
      - openstack-publish-jobs
 
      - pypi-jobs
 
 
 
'''This page may be out of date by the time you reach this step. Look at a recently updated library to see how it was handled.'''
 
 
 
==== Configure Zuul to Run Jobs ====
 
 
 
Zuul is the gate keeper. It watches for changes in gerrit to trigger the appropriate jobs. To start, establish the rules for the basic jobs already configured, but not the full devstack-gate jobs.
 
 
 
Edit zuul/layout.yaml to add a section like:
 
 
 
<pre><nowiki>
 
  - name: openstack/project-name
 
    template:
 
      - name: merge-check
 
      - name: python-jobs
 
      - name: openstack-server-publish-jobs
 
      - name: check-requirements
 
      - name: integrated-gate
 
      - name: publish-to-pypi
 
      - name: python3-jobs
 
      - name: translation-jobs
 
</nowiki></pre>
 
 
 
'''This page may be out of date by the time you reach this step. Look at a recently updated library to see how it was handled.'''
 
 
 
You can find more info about job templates in the beginning of zuul/layout.yaml in section "project-templates:".
 
 
 
For projects with documentation hosted on readthedocs.org, include "hook-project-name-rtfd" under the post, pre-release, and release groups.
 
 
 
If you use pypi-jobs and publish-to-pypi, please ensure your project's namespace is registered on pypi. This will be required before your patch is merged.
 
 
 
==== Configure GerritBot to Announce Changes ====
 
 
 
Edit gerritbot/channels.yaml to add your new repository to the list of projects under openstack-oslo:
 
 
 
  openstack-oslo:
 
    events:
 
      - patchset-created
 
      - x-vrif-minus-2
 
    projects:
 
      - openstack/cliff
 
      - openstack/oslo.config
 
      - openstack/oslo-incubator
 
      - openstack/oslo.messaging
 
      - openstack/oslo.rootwrap
 
      - openstack/oslosphinx
 
      - openstack/oslo-specs
 
      - openstack/oslo.test
 
      - openstack/oslo.version
 
      - openstack/oslo.vmware
 
      - openstack/stevedore
 
      - openstack/taskflow
 
      - openstack-dev/cookiecutter
 
      - openstack-dev/hacking
 
      - openstack-dev/oslo-cookiecutter
 
      - openstack-dev/pbr
 
    branches:
 
      - master
 
 
 
==== Submitting Infra Change for Review ====
 
 
 
'''IMPORTANT:''' When submitting the change to openstack-infra/project-config for review, use the "new-project" topic so it receives a high priority.
 
 
 
  $ git review -t new-project
 
 
 
==== Wait Here ====
 
 
 
The rest of the process needs this initial import to finish, so coordinate with the Infra team, and read ahead, but don't do any of these other steps until the import is complete and the new repository is configured.
 
 
 
==== Update the Gerrit Group Members ====
 
 
 
Once the review is approved and groups are created, Ask the Infra team to add you to both groups in gerrit, and then you can add other members.
 
 
 
The "oslo-core" group should be added to every sub-project core list (as a group, not individually).
 
 
 
The project PTL and Oslo PTL, at least, should be added to "project-name-release", and other developers who understand the release process can volunteer to be added as well.
 
 
 
=== Updating devstack-vm-gate-wrap.sh ===
 
 
 
Check out openstack-infra/devstack-gate and edit devstack-vm-gate-wrap.sh to add the new project:
 
 
 
  PROJECTS="openstack/project-name $PROJECTS"
 
 
 
Keep the list in alphabetical order.
 
 
 
See https://review.openstack.org/#/c/72487/ for an example.
 
 
 
'''No changes will land in the new repo until this is done, so wait for these changes to be accepted before proceeding.'''
 
 
 
=== Add Project to the Requirements Mirror List ===
 
 
 
The global requirements repository (openstack/requirements) controls which packages are available in the PyPI mirror used for the CI system. It also automatically contributes updates to the requirements lists for OpenStack projects when the global requirements change.
 
 
 
Edit the projects.txt file to add the new library, adding "openstack/project-name" in the appropriate place in alphabetical order.
 
 
 
Refer to https://review.openstack.org/#/c/35845/ for an example.
 
 
 
=== Add Project to the Governance Repository ===
 
 
 
Each project managed by an official program in OpenStack needs to be listed in reference/programs.yaml in the openstack/governance repository to indicate who owns the project so we know where ATCs voting rights extend.
 
 
 
Find the "Common Libraries" section in reference/programs.yaml and add the new project to the list.
 
 
 
Common Libraries:
 
  codename: Oslo
 
  ptl: Doug Hellmann (dhellmann)
 
  mission:
 
    To produce a set of python libraries containing code shared by OpenStack
 
    projects. The APIs provided by these libraries should be high quality,
 
    stable, consistent, documented and generally applicable.
 
  url: https://wiki.openstack.org/wiki/Oslo
 
  projects:
 
    - openstack/oslo-incubator
 
    - openstack/oslo.config
 
    - openstack/oslo.messaging
 
    - openstack/oslo.rootwrap
 
    - openstack/oslo.sphinx
 
    - openstack/oslo.version
 
    - openstack-dev/cookiecutter
 
    - openstack-dev/hacking
 
    - openstack-dev/pbr
 
 
 
=== Add the Project to the Oslo Library List ===
 
  
 
Edit [[Oslo]] wiki page to add the new project to the list of libraries we maintain. Include a brief description.
 
Edit [[Oslo]] wiki page to add the new project to the list of libraries we maintain. Include a brief description.
Line 364: Line 134:
  
 
The next step is to verify that you can submit a change request for the repository, the tests run successfully, you have permission to approve changes, and the release process works.
 
The next step is to verify that you can submit a change request for the repository, the tests run successfully, you have permission to approve changes, and the release process works.
 
== Set up Launchpad ==
 
 
=== Create a new Launchpad Project ===
 
 
Visit https://launchpad.net/projects/+new and fill in the details.
 
 
Name your project using the same name as the library, unless that is taken. Try "python-" as a prefix if necessary. If that name is also taken, contact the Oslo PTL before going any further.
 
 
=== Put Your New Project in the Oslo Project Group ===
 
 
From the Overview page of your project, select "Change Details" from the right sidebar (e.g., http://launchpad.net/oslo.foo/+edit).
 
 
Find the "Part of" field and set the value to "oslo".
 
 
Save your changes.
 
 
=== Create Bug Tracker ===
 
 
From the Overview page for your project, click the "Bugs" link at the top of the page. Launchpad should suggest that you set up bug tracking.
 
 
Choose "In launchpad".
 
 
Check the box labeled "Expire 'Incomplete' bug reports when they become inactive"
 
 
Check the box labeled "Search for possible duplicate bugs when a new bug is filed"
 
 
Set the "Bug supervisor" field to "oslo-bugs".
 
 
Save your changes.
 
 
=== Create Blueprint Tracker ===
 
 
From the Overview page for your project, click the "Blueprints" link at the top of the page. Launchpad should suggest that you set up blueprint tracking.
 
 
Choose "Launchpad".
 
 
Save your changes.
 
 
=== Set up Supervisors for your Project ===
 
 
From the Overview page for your project, click the pencil "edit" icon next to the Maintainer field. Replace your name with "oslo-drivers".
 
 
From the Overview page for your project, click the pencil "edit" icon next to the Drivers field. Replace your name with "oslo-drivers".
 
 
NOTE: If either of these steps makes it so you cannot edit the project, stop and ask someone in the oslo-drivers group to help you before proceeding.
 
  
 
== Prepare an Initial Release ==
 
== Prepare an Initial Release ==
Line 426: Line 150:
  
 
Update the rest of the documentation with information about the public API, tips on adopting it, etc.
 
Update the rest of the documentation with information about the public API, tips on adopting it, etc.
 
=== Give OpenStack Permission to Publish Releases ===
 
 
New libraries without any releases need to be manually registered on PyPI.
 
 
If you already have PyPI credentials, visit https://pypi.python.org/pypi?%3Aaction=submit_form and fill in only the required fields.
 
 
If you do not have PyPI credentials, you can either create them or ask an Oslo dev who has them to handle this step for you.
 
 
Next your project needs to be updated so the "openstackci" user has "Owner" permissions.
 
 
Visit https://pypi.python.org/pypi?:action=role_form&package_name=project-name and add "openstackci" in the "User Name" field, set the role to Owner, and click "Add Role".
 
 
 
[[File:PyPI-role-maintenance.png|800x499px|thumbnail|center]]
 
  
 
=== Tagging a Release ===
 
=== Tagging a Release ===
Line 457: Line 166:
 
   git os-job $version
 
   git os-job $version
  
=== Remove Incubated Code ===
+
== Remove Incubated Code ==
  
 
After the first successful release of a newly graduated library, the module(s) should be removed from the incubator. During this phase, critical bug fixes will be merged to the previous release stable branch in the incubator version of the code. New features and minor bugs should be fixed in the released library, and effort should be spent focusing on having downstream projects consume the library.
 
After the first successful release of a newly graduated library, the module(s) should be removed from the incubator. During this phase, critical bug fixes will be merged to the previous release stable branch in the incubator version of the code. New features and minor bugs should be fixed in the released library, and effort should be spent focusing on having downstream projects consume the library.
Line 475: Line 184:
 
# add the new library
 
# add the new library
 
# add any of the library's direct dependencies that are not already listed
 
# add any of the library's direct dependencies that are not already listed
 
See https://review.openstack.org/#/c/35845/ for an example.
 
  
 
=== Document Migration Process ===
 
=== Document Migration Process ===
Line 484: Line 191:
 
Link to the documentation under [[Oslo#Libraries]].
 
Link to the documentation under [[Oslo#Libraries]].
  
== Setting up Gate Testing ==
+
== Updating devstack to Enable Gate Testing ==
  
 
The devstack gate jobs install all OpenStack projects from source so that the appropriate git revisions (head, or revisions in the merge queue) are tested together. To include the new library in these tests, it needs to be included in the list of projects in the devstack gate wrapper script. For the same feature to work for developers outside of the gate, the project needs to be added to the Oslo portion of devstack.
 
The devstack gate jobs install all OpenStack projects from source so that the appropriate git revisions (head, or revisions in the merge queue) are tested together. To include the new library in these tests, it needs to be included in the list of projects in the devstack gate wrapper script. For the same feature to work for developers outside of the gate, the project needs to be added to the Oslo portion of devstack.
 
=== Updating devstack ===
 
  
 
Check out openstack-dev/devstack.
 
Check out openstack-dev/devstack.
Line 510: Line 215:
 
   NEWPROJECT_REPO=${NEWPROJECT_REPO:-${GIT_BASE}/openstack/new-project.git}
 
   NEWPROJECT_REPO=${NEWPROJECT_REPO:-${GIT_BASE}/openstack/new-project.git}
 
   NEWPROJECT_BRANCH=${NEWPROJECT_BRANCH:-master}
 
   NEWPROJECT_BRANCH=${NEWPROJECT_BRANCH:-master}
 
See https://review.openstack.org/#/c/72437/ for an example.
 
  
 
'''Wait for these changes to be accepted before proceeding.'''
 
'''Wait for these changes to be accepted before proceeding.'''
Line 535: Line 238:
 
You can paste this task list into the "Work Items" section of the blueprint to make keeping track of the steps easier.
 
You can paste this task list into the "Work Items" section of the blueprint to make keeping track of the steps easier.
  
 +
Create Launchpad project: TODO
 +
Create Launchpad bug tracker: TODO
 +
Create Launchpad blueprint tracker: TODO
 +
Change owner of Launchpad project: TODO
 +
Give openstackci Owner permissions on PyPI: TODO
 
  Create Initial Repository: TODO
 
  Create Initial Repository: TODO
 
  Update MAINTAINERS in incubator with status and name: TODO
 
  Update MAINTAINERS in incubator with status and name: TODO
 
  Remove Oslo logging calls in incubator: TODO
 
  Remove Oslo logging calls in incubator: TODO
Run graduate.sh: TODO
 
 
  Fix the output of graduate.sh: TODO
 
  Fix the output of graduate.sh: TODO
Use cookiecutter template to make a new project: TODO
 
 
  Sync tools from incubator: TODO
 
  Sync tools from incubator: TODO
 
  Publish git repo: TODO
 
  Publish git repo: TODO
 
  Oslo team review new repository: TODO
 
  Oslo team review new repository: TODO
 +
openstack/governance reference/programs.yaml: TODO
 
  openstack-infra/project-config - gerrit/projects.yaml: TODO
 
  openstack-infra/project-config - gerrit/projects.yaml: TODO
 
  openstack-infra/project-config - gerrit/acls/openstack/project-name.config: TODO
 
  openstack-infra/project-config - gerrit/acls/openstack/project-name.config: TODO
Line 552: Line 259:
 
  openstack-infra/devstack-gate - devstack-vm-gate-wrap.sh: TODO
 
  openstack-infra/devstack-gate - devstack-vm-gate-wrap.sh: TODO
 
  openstack/requirements projects.txt: TODO
 
  openstack/requirements projects.txt: TODO
openstack/governance reference/programs.yaml: TODO
 
 
  Update list of libraries on Oslo wiki page: TODO
 
  Update list of libraries on Oslo wiki page: TODO
Create Launchpad project: TODO
 
Create Launchpad bug tracker: TODO
 
Create Launchpad blueprint tracker: TODO
 
Change owner of Launchpad project: TODO
 
 
  Make the library do something: TODO
 
  Make the library do something: TODO
 
  Update the README.rst: TODO
 
  Update the README.rst: TODO
Give openstackci Owner permissions on PyPI: TODO
 
 
  Tag a release: TODO
 
  Tag a release: TODO
 
  Remove graduated code from oslo-incubator: TODO
 
  Remove graduated code from oslo-incubator: TODO

Revision as of 20:13, 9 December 2014

Before You Start

This is a long process. It's long because it has to be, not because we want it to be. If you follow it, everything will be fine.

It is important that you perform all of the steps, in the order they are given here. Don't skip any steps. Don't try to do things in parallel. Don't jump around.

Project Creator's Guide

The Project Creator's Guide includes many of the details for creating a repository. It is maintained by the Infrastructure team, and should be up to date with the current process.

Read that guide before continuing. Some steps below will refer to sections of that guide, and you will need to use both documents together to complete this process.

The Spec

The Oslo team reviews plans for large changes, such as graduations, using the specs process common to the feature review processes of other teams within OpenStack. To prepare to graduate or adopt a library, you will need to file a spec.

Blueprint

First, create a blueprint in the oslo-incubator repository on launchpad. Name the blueprint graduate-oslo-foo, replacing foo with the name of the library.

Spec

In the openstack/oslo-specs repository, copy specs/graduation-template.rst to create a new spec using the name of your blueprint under the release code name directory. For example, specs/kilo/graduate-oslo-reports.rst. Edit the new file, filling in the details, then submit it for review just like any other code review.

The spec template includes questions that come later in this document, for which you will need to provide an answer at this point. This is the one exception to the "don't read ahead or skip around" rule mentioned earlier. Collect all of the information you need to fill in the spec.

Choosing a Name

In addition to the rules described in the Project Creator's Guide, there are currently three naming schemes used for Oslo libraries:

  1. oslo.something - Used for production runtime dependencies of OpenStack projects. Using the oslo. namespace for any other type of package will cause conflicts due to some limitations in the way Python and Pip handle namespace packages.
  2. oslosomething - Used for non-production or non-runtime dependencies of OpenStack projects. If you're planning to use a name like this, please discuss it with the Oslo team first - we aren't sure we like this name scheme and may suggest an alternative.
  3. something - Used for libraries that may be generally useful outside of OpenStack. Note: you should verify that the name you are proposing is not in use on the Python Package Index (https://pypi.python.org/pypi).

Graduating a Library from the Incubator

There are three sources for new Oslo libraries:

  1. Incubated code that is graduating.
  2. A brand new library.
  3. An existing library that we are adopting.

Most of this document covers incubated code. Brand new and existing libraries can skip ahead to "Importing the Repository into the CI System" below.

Graduating a library is a multi-step process. It isn't complicated, but there are a lot of details that are easy to miss. Please tread carefully.

The goal with incubated code is to extract it from the incubator with all of its history, and publish the results as a new library. This means that a new repository must be created outside of gerrit, and then imported, to avoid having to re-review all of the existing changesets. A lot of that process is automated, but there are inevitably some manual steps -- especially to ensure that the tests work properly.

Perform the steps in this section before following the directions in the Project Creator's Guide, then use the repository you create as the "upstream" value when adding the project to gerrit.

Updating the Status in the Incubator

Update oslo-incubator/MAINTAINERS

Set the status of the module(s) to "Graduating".

Verify all of the correct names and contact details are present.

If only some modules in a sub-package under openstack/common are being graduated, add comments to those files clearly indicating that they should not be modified during graduation.

Remove dependencies on oslo logging code

For the files that will be graduating, submit a change to oslo-incubator that switches them to use standard Python logging instead of any Oslo logging they may be using (either from incubator or oslo.log). The logging code is now able to add context information to all third-party library log messages, so it makes sense to have graduated Oslo libraries behave the same as any other third-party library.

We want this done in incubator before the new repo is created so we can review the changes, but don't have to pull in logging dependencies that will just be removed immediately. This shouldn't be a problem for anyone continuing to use the incubator versions of the files either.

Extracting a Clean Copy of the History

Start by checking out a copy of the oslo-incubator to use for running the tool:

 cd /tmp
 mkdir graduation
 cd graduation
 git clone git://git.openstack.org/openstack/oslo-incubator

Now clone a second copy of the repository to be edited:

 git clone git://git.openstack.org/openstack/oslo-incubator oslo.i18n

The graduation script works by editing the git history of the current repository, removing commits that do not touch the files you want to keep. It then rearranges the resulting files and sets up the oslo package namespace. To do these things, it needs to know the name of the new library being created and the names of the files you wish to keep.

 cd oslo.i18n
 ../oslo-incubator/tools/graduate.sh i18n openstack/common/gettextutils.py tests/unit/test_gettext.py

The script is fairly I/O intensive, and will take a while to run. It logs all of its output to /tmp/oslo-graduate.XXXX/output.log, and other scratch files are kept in the same directory.

Manual Fixes

It is not possible to completely automate the process of reorganizing the library code and ensuring that it will work, so there are several manual tasks to be performed next.

  1. Add missing dependencies
  2. Fix broken imports within the current repository resulting from files being renamed
  3. Make sure the tests run -- not just make sure tox passes, make sure it is actually running the tests (moving files around makes this break silently some times)
  4. Test the new library with some of the projects that use the incubator version now

At this point, make as few changes as possible, just to make the test jobs pass. Larger changes to the API should go through the normal review process.

Allowed changes include:

  • Broken imports
  • PEP8/test failures
  • Syncing dependencies from the incubator

Do not:

  • Enable new tests or PEP8 checks
  • Add new library dependencies that were not already in use by the incubated code

Sync Required Tools from Incubator

Some of the tools common to all Oslo libraries reside in the oslo-incubator. These need to be synced into the new repository before it is imported.

  1. clone a copy of the oslo-incubator repository
  2. Add appropriate modules and scripts to /path/to/newproject/openstack-common.conf
  3. In oslo-incubator, run: ./update.sh /path/to/newproject
  4. In /path/to/newproject, add the new files and commit any changes

Publish Your Results

When you have a clean version of the library, publish it to a public git repository (e.g., on github). This copy of the project history will be used in subsequent steps.

Receive Oslo Team Sign-off

Before proceeding to import the library into the OpenStack CI system, post to the openstack-dev mailing list asking for the Oslo team to review the public repository.

Adding the Repository to the CI System

Follow the steps in the Project Creator's Guide for importing your new repository, using the repository you have created as the "upstream" value when adding the project to gerrit.

Add the Project to the Oslo Library List

Edit Oslo wiki page to add the new project to the list of libraries we maintain. Include a brief description.

Verify That Gerrit and the Test Jobs are Working

The next step is to verify that you can submit a change request for the repository, the tests run successfully, you have permission to approve changes, and the release process works.

Prepare an Initial Release

Make Your Library Useful

Before going any farther, make the library do something useful.

If you are importing an existing library with features, you can go ahead.

If you are creating a brand new library, add some code and tests to provide some minimal functionality useful to another project.

Provide Basic Documentation

Update the README.rst file to include a paragraph describing the new library.

Update the rest of the documentation with information about the public API, tips on adopting it, etc.

Tagging a Release

To verify that the release machinery works, push a signed tag to the "gerrit" remote. Use the smallest version number possible. If this is the first release, use "0.1.0". If other releases of the library exist, choose an appropriate next version number using Oslo/VersioningPolicy.

Run:

 git tag -s -m "descriptive message" $version
 git push gerrit $version

Wait a little while for the pypi job to run and publish the release.

If you need to check the logs, you can use the "git-os-job" plugin (https://pypi.python.org/pypi/git-os-job):

 git os-job $version

Remove Incubated Code

After the first successful release of a newly graduated library, the module(s) should be removed from the incubator. During this phase, critical bug fixes will be merged to the previous release stable branch in the incubator version of the code. New features and minor bugs should be fixed in the released library, and effort should be spent focusing on having downstream projects consume the library.

Avoid Rewriting References in Incubator's update.py

The oslo-incubator/update.py script rewrites references to "oslo" with the name of the project receiving the copy of the code. Now that the code is no longer being consumed from the incubator, any code remaining in the incubator that refers to it needs to refer to the released library. Modify update.py so that it no longer rewrites references by adding it to the list of libraries known in the _copy_file() function. Look for "for oslo_module in ..." around line 158.

Allowing Other OpenStack Projects to Use Your Library

OpenStack projects share a common global requirements list so that all components can be installed together on the same system. To allow other projects to use your library, you need to update that requirements list.

Update the Global Requirements List

Check out the openstack/requirements git repository and modify global-requirements.txt to:

  1. add the new library
  2. add any of the library's direct dependencies that are not already listed

Document Migration Process

If the new library is graduating from code that was already being used through the incubator, create a brief description of the types of changes that are needed to adopt the library. For example, if the API has changed or if a common pattern is needed to create an integration module, describe that and provide an example. This should be a wiki page under Oslo with a name like Oslo/MigratingTo-project-name (keeping this in the wiki will make it easier for project liaisons to add tips and notes).

Link to the documentation under Oslo#Libraries.

Updating devstack to Enable Gate Testing

The devstack gate jobs install all OpenStack projects from source so that the appropriate git revisions (head, or revisions in the merge queue) are tested together. To include the new library in these tests, it needs to be included in the list of projects in the devstack gate wrapper script. For the same feature to work for developers outside of the gate, the project needs to be added to the Oslo portion of devstack.

Check out openstack-dev/devstack.

Edit lib/oslo to add a variable defining where the source should go:

 NEWPROJECT_DIR=$DEST/newproject

Edit the "install_oslo()" function in lib/oslo to add commands to check out the repository. The libraries need to be installed in order so that the lower-level packages are installed first (this avoids having a library installed from a package and then re-installed from source):

 function install_oslo() {
   ...
   git_clone $NEWPROJECT_REPO $NEWPROJECT_DIR $NEWPROJECT_BRANCH
   setup_develop $NEWPROJECT_DIR
   ...
 }

Edit stackrc to add the other variables needed for configuring the new library:

 # new-project
 NEWPROJECT_REPO=${NEWPROJECT_REPO:-${GIT_BASE}/openstack/new-project.git}
 NEWPROJECT_BRANCH=${NEWPROJECT_BRANCH:-master}

Wait for these changes to be accepted before proceeding.

Add Link to Your Developer Documentation

Update the http://docs.openstack.org/developer/openstack-projects.html page with a link to your documentation by checking out the openstack/openstack-manuals repository and edititing www/developer/openstack-projects.html.

Update Consuming Projects

Now that the library is fully integrated with the rest of the OpenStack test infrastructure, projects can be updated to use it safely. Send email to the openstack-dev mailing list notifying the community that the library is available and ready for use.

Coordinate with the development teams for each project to find a time in their dev cycle when they can work with you on the update and review the necessary changes. The liaisons can help with this.

Track adoption progress using the etherpad for the release cycle. For example, https://etherpad.openstack.org/p/juno-oslo-adoption-status

The PTLs of the other projects may also ask for blueprints or bugs to track the work of migrating within their projects. Use whatever process they are comfortable with.

Try to set the topic for changes in the other projects to refer to your graduation blueprint, either on the command line or by referencing the blueprint in the git commit message. This makes it easier to track all of the work for the blueprint, across all of our repositories.

Checklist

You can paste this task list into the "Work Items" section of the blueprint to make keeping track of the steps easier.

Create Launchpad project: TODO
Create Launchpad bug tracker: TODO
Create Launchpad blueprint tracker: TODO
Change owner of Launchpad project: TODO
Give openstackci Owner permissions on PyPI: TODO
Create Initial Repository: TODO
Update MAINTAINERS in incubator with status and name: TODO
Remove Oslo logging calls in incubator: TODO
Fix the output of graduate.sh: TODO
Sync tools from incubator: TODO
Publish git repo: TODO
Oslo team review new repository: TODO
openstack/governance reference/programs.yaml: TODO
openstack-infra/project-config - gerrit/projects.yaml: TODO
openstack-infra/project-config - gerrit/acls/openstack/project-name.config: TODO
openstack-infra/project-config - jenkins/jobs/projects.yaml: TODO
openstack-infra/project-config - zuul/layout.yaml: TODO
openstack-infra/project-config - gerritbot/channels.yaml: TODO
Update Gerrit Groups and ACLs: TODO
openstack-infra/devstack-gate - devstack-vm-gate-wrap.sh: TODO
openstack/requirements projects.txt: TODO
Update list of libraries on Oslo wiki page: TODO
Make the library do something: TODO
Update the README.rst: TODO
Tag a release: TODO
Remove graduated code from oslo-incubator: TODO
Update oslo-incubator/update.py to not rewrite references to the library: TODO
openstack/requirements - global-requirements.txt: TODO
Document Migration Process: TODO
openstack-dev/devstack - lib/oslo: TODO
openstack-dev/devstack - stackrc: TODO
Update project list on docs.openstack.org: TODO