- 1 Stable Branch
- 1.1 Overview
- 1.2 Stable branch policy
- 1.3 Processes
- 1.4 Team organization
The stable branch is intended to be a safe source of fixes for high impact bugs and security issues which have been fixed on master since a given release.
OpenStack's stable branch policy borrows much from prior art. There are many projects which maintain such a branch and it's possible to build a picture of a rough consensus across these projects on how a stable branch should operate. One example of another project's policy is the Linux kernel's stable_kernel_rules.txt.
Stable branch policy
Only a limited class of changes are appropriate for inclusion on the stable branch.
A number of factors must be weighed when considering a change:
- The risk of regression - even the tiniest changes carry some risk of breaking something and we really want to avoid regressions on the stable branch
- The user visible benefit - are we fixing something that users might actually notice and, if so, how important is it?
- How self-contained the fix is - if it fixes a significant issue but also refactors a lot of code, it's probably worth thinking about what a less risky fix might look like
- Whether the fix is already on master and all consequent stable branches - a change must be a backport of a change already merged onto master, unless the change simply does not make sense on master. Same applies to N-2 releases, where N is master, in which case both N-1 and N branches should have the patch merged. (Note: some patches may get exception from this specific rule. These are patches that do not touch production code, like test-only patches, or tox.ini changes that fix major gate breakage, etc.; or security patches that should not take much time to merge once the patches are published. In those cases, stable patches may be pushed into gate without waiting for all consequent branches to be fixed. Warning: in case review process reveals issues in master patch which require rework after stable patches are merged, it's expected that additional changes are merged into stable branches to avoid unneeded difference between branches. So use the exception with due care.)
The stable review teams need to balance the risk of any given patch with the value that it will provide to users of the stable branch. A large, risky patch for a major data corruption issue might make sense. As might a trivial fix for a fairly obscure error handling case.
Some types of changes are completely forbidden:
- New features
- Changes to the external HTTP APIs
- Changes to Nova's internal AMQP API
- Changes to the notification definitions
- DB schema changes
- Incompatible config file changes
Proposed backports breaking any of above guidelines can be discussed as exception requests on the openstack-dev list (prefix with [stable]) where stable-maint-core team will have the final say.
Stability is always a trade-off between "bug-free" and "slow-moving". In order to reach that stability, we define several support phases:
- Phase I (first 6 months): All bugfixes which are appropriate
- Phase II (6-12 months): Only critical bugfixes and security patches are acceptable
- Phase III (more than 12 months): Only security patches are acceptable
Given that releases are every 6 months, that means that at a given time, only one release is in Phase I support and only one release is in Phase II support. Depending on how long a release is supported, there may be more than one release in Phase III support.
Each backported commit proposed to gerrit should be reviewed and +2ed by two $PROJECT-stable-maint members before it is approved. Where a stable-maint member has backported a fix, a single other +2 is sufficient for approval.
If unsure about the technical details of a given fix, stable-maint members should consult with the appropriate $PROJECT-core developers for a more detailed technical review.
If unsure if a fix is appropriate for the stable branch, stable-maint members should seek stable-maint-core team members opinion.
Existing core developers are greatly encouraged to join the stable-maint team in order to help with reviewing backports, judging their appropriateness for the stable branch and approving them.
Fixes for embargoed security issues receive special treatment. These should be reviewed in advance of disclosure by core developers and stable-maint. At the time of coordinated public disclosure, the fix is proposed simultaneously to master and the stable branches and immediately approved.
Anyone can propose a cherry-pick to the stable-maint team.
One way is that if a bug in launchpad looks like a good candidate for backporting - e.g. if it's a significant bug with the previous release - then just nominating the bug for the icehouse series will bring it to the attention of the maintainers e.g. for nova.
If you don't have the appropriate permissions to nominate the bug, then tagging it with e.g. liberty-backport-potential is also sufficient e.g. for nova.
The best way to get the patch merged in timely manner is to send it backported by yourself. To do so, you may try to use "Cherry Pick To" button in Gerrit UI for the original patch in master. Gerrit will take care of creating a new review, modifying commit message to include 'cherry-picked from ...' line etc.
Note: unless there is a serious need for that (gate failure, test framework changed in master, ...), backport up for review must not differ from the master counterpart. If you get a suggestion to ‘enhance’ your backport in some way that is of no user visible influence, kindly suggest the reviewer to fix it in master and backport to stable branches, if they care that much. It’s not your job to fix it. The only exception is if by chance backport review reveals an actual bug in master code that would introduce a regression in stable branches, if backported. In that case, the backport should be blocked until master is fixed and the second backport is proposed; in that case, both backports should be up for review and ready to merge before the first one is pushed into gate (that’s to ensure we reduce chance of breaking someone who catches up with the branch in the middle).
Note: for code that touches code from oslo-incubator, special backporting rules apply. More details in Oslo policies.
If the patch you're proposing will not cherry-pick cleanly, you can help by resolving the conflicts yourself and proposing the resulting patch. Please keep Conflicts lines in the commit message to help reviewers! You can use git-review to propose a change to the stable branch with:
$> git checkout stable/icehouse $> git cherry-pick -x $master_commit_d $> git review stable/icehouse
Note: cherry-pick -x option includes 'cherry-picked from ...' line in the commit message which is required to avoid Gerrit bug
Failing all that, just ping one of the team and mention that you think the bug/commit is a good candidate.
When cherry-picking a commit, keep the original
Change-Id and gerrit will show a separate review for the stable branch while still allowing you to use the
Change-Id to see all the reviews associated with it. See this change as an example.
Hint: Change-Id line must be in the last paragraph. Conflicts in the backport add a new paragraph, creating a new Change-Id but you can avoid that by moving conflicts above the paragraph with Change-Id line or removing empty lines to make a single paragraph.
If you want to be notified of these patches you can create a watch on this screen: https://review.openstack.org/#settings,projects with the settings:
Project Name: All-Projects Only If: branch:stable/icehouse
Then check the "Email Notifications - New Changes" checkbox. That will cause gerrit to send an email whenever a matching change is proposed, and better yet, the change shows up in your 'watched changes' list in gerrit.
Bugs tagged with icehouse-backport-potential are bugs which apply to the stable release and may be suitable for backporting once fixed. Once the backport has been proposed, the tag should be removed.
Gerrit tags bugs with in-stable-icehouse when they are merged into the stable branch. The release manager later removes the tag when the bug is targeted to the appropriate series.
Each project with a stable branch will have a specific $PROJECT-stable-maint team, which would be in charge of reviewing backports for a given project, following the stable branch policy. Originally that group should be the project Stable Branch Cross-Project Liaison + the stable-maint-core team. Those groups are managed by stable-maint-core, names are added after the suggestion of the Stable Branch CPL.
Stable branch champions
Stable branches are less exercised than master branches, and they may get broken by external events. Stable Branch Champions are tasked with championing a specific stable branch support, making sure the branch stays in good shape and remains usable at all times. They monitor periodic jobs failures and enlist the help of others in order to fix the branches in case of breakage. They should also raise flags if for some reason they are blocked and don't receive enough support, in which case early abandon of the branch will be considered.
Current champions are: Ihar Hrachyshka (ihrachys at redhat dot com): Icehouse; Adam Gandelman (adamg at ubuntu dot com): Juno.
Stable release managers
The stable-maint and release teams work together to publish point releases from the stable branch. See StableBranchRelease for details about how these releases are prepared.
One week before the planned point release, stable release manager will announce that stable branch is frozen to allow testing before the release. Freeze exceptions can be proposed on openstack-dev list with [stable] tag in subject where stable-maint team will try to reach consensus.
When new OpenStack version is released it becomes "Current stable release, security-supported"" and previous version "Security-supported". "Current stable release" is the primary target for backports of bugfixes by the stable-maint team, other active branches may receive bugfix backports. "Security-supported" releases are target for the security fixes by the VMT OpenStack Havana stable release will reach EOL at Juno (N+2) milestone-3 and stable releases starting from Icehouse will reach EOL 15 months after their release (approximately N+3 milestone-2).
The stable-maint-core team is responsible for the enforcement of the Stable Branch policy. It is composed of Stable release managers, Stable branch champions and other cross-project stable branch custodians. It will be granting exceptions for all questionable backports raised by $PROJECT-stable-maint groups, providing backports reviews help everywhere, maintaining the stable branch policy (and make sure its rules are respected), educating proposed $PROJECT-stable-maint members on those rules and adding them to $PROJECT-stable-maint teams.
Joining the Team
We're really keen to add more folks to the stable-maint teams to help out with reviews.
All you really need is some time and the ability to apply the "safe source of high impact fixes" and "must be fixed on master first" policies. It mostly comes down to having a good sense of the risk vs benefit of applying a backport to the branch.
If you'd like to join the team, you can start by simply [+-]1ing stable branch reviews. It's best if you can add some brief thoughts to your review on why you think the fix is suitable for stable so that we know how you're applying the policy.
Same as in other OpenStack core teams, stable-maint membership will be periodically reviewed and inactive members removed.
Keeping the stable branches in good health in an ongoing effort. To see what bugs are currently causing gate failures and preventing code from merging into stable branches, please see the stable tracker etherpad, where we will track current bugs and in-flight fixes.
Scheduled test runs occur daily for each project's stable branch. If failures crop up, the bot will email the openstack-stable-maint mailing list . It is best to react quickly to these and get them resolved ASAP to prevent them from piling up. Please subscribe if you're interested in helping out.