Difference between revisions of "Oslo/CreatingANewLibrary"
(→Allowing Other OpenStack Projects to Use Your Library) |
(→Setting up Gate Testing) |
||
Line 233: | Line 233: | ||
== Setting up Gate Testing == | == Setting up Gate Testing == | ||
− | The devstack gate jobs install all OpenStack projects from source so that the appropriate git revisions (head, or revisions in the merge queue) are tested together. To include the new library in these tests, it needs to be included in the list of projects in the devstack gate wrapper script. | + | The devstack gate jobs install all OpenStack projects from source so that the appropriate git revisions (head, or revisions in the merge queue) are tested together. To include the new library in these tests, it needs to be included in the list of projects in the devstack gate wrapper script. For the same feature to work for developers outside of the gate, the project needs to be added to the Oslo portion of devstack. |
+ | |||
+ | === Updating devstack-vm-gate-wrap.sh === | ||
Check out openstack-infra/devstack-gate and edit devstack-vm-gate-wrap.sh to add the new project: | Check out openstack-infra/devstack-gate and edit devstack-vm-gate-wrap.sh to add the new project: | ||
Line 242: | Line 244: | ||
See https://review.openstack.org/#/c/72487/ for an example. | See https://review.openstack.org/#/c/72487/ for an example. | ||
+ | |||
+ | === Updating devstack === | ||
+ | |||
+ | Check out openstack-dev/devstack. | ||
+ | |||
+ | Edit lib/oslo to add a variable defining where the source should go: | ||
+ | |||
+ | NEWPROJECT_DIR=$DEST/newproject | ||
+ | |||
+ | Edit the "install_oslo()" function in lib/oslo to add commands to check out the repository: | ||
+ | |||
+ | function install_oslo() { | ||
+ | ... | ||
+ | git_clone $NEWPROJECT_REPO $NEWPROJECT_DIR $NEWPROJECT_BRANCH | ||
+ | setup_develop $NEWPROJECT_DIR | ||
+ | ... | ||
+ | } | ||
+ | |||
+ | Edit stackrc to add the other variables needed for configuring the new library: | ||
+ | |||
+ | # new-project | ||
+ | NEWPROJECT_REPO=${NEWPROJECT_REPO:-${GIT_BASE}/openstack/new-project.git} | ||
+ | NEWPROJECT_BRANCH=${NEWPROJECT_BRANCH:-master} | ||
+ | |||
+ | See https://review.openstack.org/#/c/72437/ for an example. | ||
== Marking Incubator Obsolete == | == Marking Incubator Obsolete == | ||
After the first release of a newly graduated library, the status of the module(s) should be updated to "Obsolete." During this phase, only critical bug fixes will be allowed in the incubator version of the code. New features and minor bugs should be fixed in the released library, and effort should be spent focusing on having downstream projects consume the library. | After the first release of a newly graduated library, the status of the module(s) should be updated to "Obsolete." During this phase, only critical bug fixes will be allowed in the incubator version of the code. New features and minor bugs should be fixed in the released library, and effort should be spent focusing on having downstream projects consume the library. |
Revision as of 23:35, 10 February 2014
Contents
- 1 Creating an Initial Repository for Import
- 2 Importing the Repository into the CI System
- 3 Verify That Gerrit is Working
- 4 Prepare an Initial Release
- 5 Allowing Other OpenStack Projects to Use Your Library
- 6 Setting up Gate Testing
- 7 Marking Incubator Obsolete
Creating an Initial Repository for Import
There are three sources for new Oslo libraries:
- Incubated code that is graduating.
- A brand new library.
- An existing library that we are adopting.
These subsections cover incubated code and new libraries. Existing libraries can skip ahead to "Importing the Repository into the CI System" below.
Choosing a Name
Oslo libraries are either named "oslo.something" or just "something". The oslo prefix is reserved for libraries which are expected to only be of use to OpenStack developers. Libraries that may be useful outside of OpenStack should have more generic names.
Graduating a Library from the Incubator
Graduating a library is a multi-step process. It isn't complicated, but there are a lot of details that are easy to miss. Please tread carefully.
Updating the Incubator
Update oslo-incubator/MAINTAINERS:
Set the status of the module(s) to "Graduating".
Verify all of the correct names and contact details are present.
Extracting a Clean Copy of the History
The oslo-incubator repository includes a script in tools/graduate.sh that can be used to
Importing the Repository into the CI System
openstack-infra/config
These instructions are based on the steps for creating a new Stackforge project with some variations, and apply to changes made in the openstack-infra/config git repository.
Refer to https://review.openstack.org/#/c/70435/ for an example of the changes described in this section.
Add the project to the master project list
Edit modules/openstack_project/files/review.projects.yaml to add a new section like:
- project: openstack/project-name description: Latest and greatest cloud stuff. upstream: git://github.com/awesumsauce/project-name.gi
The projects in the file need to be listed in alphabetical order.
Set the "upstream" URL to the repository created earlier. If the existing library being imported is already on stackforge, the import can be handled by renaming the repository instead of using the upstream URL, so leave that line out.
Add Gerrit permissions
Each project should have 2 groups. The first, "project-name-core", is the normal core group, with permission to +2 changes. The other, "project-name-milestone", is a (usually smaller) group able to push tags to trigger releases.
Create modules/openstack_project/files/gerrit/acls/stackforge/project-name.config:
[access "refs/heads/*"] label-Code-Review = -2..+2 group project-name-core label-Approved = +0..+1 group project-name-core workInProgress = group project-name-core [access "refs/tags/*"] create = group project-name-milestone pushTag = group project-name-milestone [receive] requireChangeId = true requireContributorAgreement = true [submit] mergeContent = true
Update the Gerrit Group Members
Ask the Infra team to add you to both groups in gerrit, and then you can add other members.
The "oslo-core" group should be added to every sub-project core list (as a group, not individually).
The project PTL and Oslo PTL, at least, should be added to "project-name-milestone", and other developers who understand the release process can volunteer to be added as well.
Add Basic Jenkins Jobs
Establish the standard Python jobs, including publishing releases to PyPI and pre-release tarballs to tarballs.openstack.org.
Edit modules/openstack_project/files/jenkins_job_builder/config/projects.yaml to add:
- project: name: project-name github-org: openstack node: precise tarball-site: tarballs.openstack.org doc-publisher-site: docs.openstack.org jobs: - python-jobs - openstack-publish-jobs - pypi-jobs
Configure Zuul to Run Jobs
Zuul is the gate keeper. It watches for changes in gerrit to trigger the appropriate jobs. To start, establish the rules for the basic jobs already configured, but not the full devstack-gate jobs.
Edit modules/openstack_project/files/zuul/layout.yaml to add a section like:
- name: openstack/project-name check: - gate-project-name-pep8 - gate-project-name-python26 - gate-project-name-python27 - gate-project-name-python33 - gate-project-name-pypy gate: - gate-project-name-pep8 - gate-project-name-python26 - gate-project-name-python27 - gate-project-name-python33 - gate-project-name-pypy post: - project-name-branch-tarball pre-release: - project-name-tarball release: - project-name-tarball: - project-name-pypi-upload: - post-mirror-python26 - post-mirror-python27 - post-mirror-python33
For projects with documentation hosted on readthedocs.org, include "hook-project-name-rtfd" under the post, pre-release, and release groups.
Wait Here
The rest of the process needs this initial import to finish, so coordinate with the Infra team, and read ahead, but don't do any of these other steps until the import is complete and the new repository is configured.
Add Project to the Requirements Mirror List
The global requirements repository (openstack/requirements) controls which packages are available in the PyPI mirror used for the CI system. It also automatically contributes updates to the requirements lists for OpenStack projects when the global requirements change.
Edit the projects.txt file to add the new library, adding "openstack/project-name" in the appropriate place in alphabetical order.
Refer to https://review.openstack.org/#/c/35845/ for an example.
Add Project to the Governance Repository
Each project managed by an official program in OpenStack needs to be listed in reference/programs.yaml in the openstack/governance repository to indicate who owns the project so we know where ATCs voting rights extend.
Find the "Common Libraries" section in reference/programs.yaml and add the new project to the list of "other" projects under Oslo. (No Oslo projects are designated as core right now.)
Common Libraries: codename: Oslo ptl: Doug Hellmann (dhellmann) mission: To produce a set of python libraries containing code shared by OpenStack projects. The APIs provided by these libraries should be high quality, stable, consistent, documented and generally applicable. url: https://wiki.openstack.org/wiki/Oslo projects: other: - openstack/oslo-incubator - openstack/oslo.config - openstack/oslo.messaging - openstack/oslo.rootwrap - openstack/oslo.sphinx - openstack/oslo.version - openstack-dev/cookiecutter - openstack-dev/hacking - openstack-dev/pbr
Verify That Gerrit is Working
The next step is to verify that you can submit a change request for the repository, the tests run successfully, you have permission to approve changes, and the release process works.
Add a .gitreview File to the Repository
The review plugin for git looks in the root of the repository to find a .gitreview file with location instructions for interacting with gerrit to submit change requests, pull down changes for review, etc. Adding this file to your repository now allows you to verify all of the review machinery works, without potentially having issues because of a more complicated changeset.
Check out a clean copy of your new repository and in the root directory of the checkout add a .gitreview file containing:
[gerrit] host=review.openstack.org port=29418 project=openstack/project-name.git
Prepare the changeset and submit it with "git review".
Wait for the various test suites to run and for Jenkins to verify the changeset.
Approve the change (or have another core reviewer do it).
Wait for the test and merge jobs to run successfully.
Prepare an Initial Release
Make Your Library Useful
Before going any farther, make the library do something useful.
If you are importing an existing library with features, you can go ahead.
If you are creating a brand new library, add some code and tests to provide some minimal functionality useful to another project.
Give OpenStack Permission to Publish Releases
Existing projects which are already registered on http://pypi.python.org need to be updated so the "openstackci" user has "Owner" permissions.
Visit https://pypi.python.org/pypi?:action=role_form&package_name=project-name and add "openstackci" in the "User Name" field, set the role to Owner, and click "Add Role".
New libraries without any releases do not need to follow this step -- when the project is released for the first time it will be registered correctly.
Tagging a Release
To verify that the release machinery works, push a signed tag to the "gerrit" remote. Use the smallest version number possible. If this is the first release, use 0.1. If other releases of the library exist, choose an appropriate next version number using the semantic versioning rules.
Run:
git tag -s -m "descriptive message" $version git push gerrit $version
Wait a little while for the pypi job to run and publish the release.
If you need to check the logs, you can use the "git-os-job" plugin (https://pypi.python.org/pypi/git-os-job):
git os-job $version
Allowing Other OpenStack Projects to Use Your Library
OpenStack projects share a common global requirements list so that all components can be installed together on the same system. To allow other projects to use your library, you need to update that requirements list.
Update the Global Requirements List
Check out the openstack/requirements git repository and modify global-requirements.txt to add the new library.
See https://review.openstack.org/#/c/35845/ for an example.
Setting up Gate Testing
The devstack gate jobs install all OpenStack projects from source so that the appropriate git revisions (head, or revisions in the merge queue) are tested together. To include the new library in these tests, it needs to be included in the list of projects in the devstack gate wrapper script. For the same feature to work for developers outside of the gate, the project needs to be added to the Oslo portion of devstack.
Updating devstack-vm-gate-wrap.sh
Check out openstack-infra/devstack-gate and edit devstack-vm-gate-wrap.sh to add the new project:
PROJECTS="openstack/project-name $PROJECTS"
Keep the list in alphabetical order.
See https://review.openstack.org/#/c/72487/ for an example.
Updating devstack
Check out openstack-dev/devstack.
Edit lib/oslo to add a variable defining where the source should go:
NEWPROJECT_DIR=$DEST/newproject
Edit the "install_oslo()" function in lib/oslo to add commands to check out the repository:
function install_oslo() { ... git_clone $NEWPROJECT_REPO $NEWPROJECT_DIR $NEWPROJECT_BRANCH setup_develop $NEWPROJECT_DIR ... }
Edit stackrc to add the other variables needed for configuring the new library:
# new-project NEWPROJECT_REPO=${NEWPROJECT_REPO:-${GIT_BASE}/openstack/new-project.git} NEWPROJECT_BRANCH=${NEWPROJECT_BRANCH:-master}
See https://review.openstack.org/#/c/72437/ for an example.
Marking Incubator Obsolete
After the first release of a newly graduated library, the status of the module(s) should be updated to "Obsolete." During this phase, only critical bug fixes will be allowed in the incubator version of the code. New features and minor bugs should be fixed in the released library, and effort should be spent focusing on having downstream projects consume the library.