Jump to: navigation, search

Difference between revisions of "StableBranch"

(added link to maintains group)
(Re-work things significantly to reflect the current state of the world)
Line 1: Line 1:
 
__NOTOC__
 
__NOTOC__
Starting with [[OpenStack]] Diablo (Sept, 2011) we've continued to developer maintain and support the stable release of [[OpenStack]] for security and high impact issues. The current stable release is Essex.
 
 
 
= Stable Branch =
 
= Stable Branch =
  
At the Essex Design Summit it was agreed that a working group would be established to maintain stable/backport/fixes branches.
+
The [https://launchpad.net/~openstack-stable-maint stable-maint team] manages the stable branch for [[OpenStack]] core projects with the support of the core developers of those projects and the release team.
 
 
With the Diablo stable branch, we aimed to establish:
 
 
 
* That the idea makes sense
 
* That there is enough people willing to help maintain it for it to be viable
 
* The policies for what belongs in such a stable branch
 
* How changes should be proposed to the branch
 
* The workflow for maintaining the branch
 
* Whether updates should be released from the branch
 
 
 
If these processes evolve into a successful model, they will be applied to all future releases.
 
  
 
<<[[TableOfContents]]()>>
 
<<[[TableOfContents]]()>>
Line 21: Line 8:
 
== Overview ==
 
== Overview ==
  
Each project will have a branch named after the previous release. For example, the stable branch for the diablo release was called 'stable/diablo'.
+
The stable branch is intended to be a safe source of fixes for high impact bugs and security issues which have been fixed on master since a given release.
  
The stable branch will only be maintained until the next release is out. This period may be extended if there are volunteers to maintain it beyond this point.
+
Official point releases for each project are published from the branch as sufficient fixes accumulate on the branch.
  
The [https://launchpad.net/~openstack-stable-maint maintainers], in conjunction with the release team, may choose to cut a new release from the branch if there are enough worthwhile fixes and the branch has been tested enough.
+
The Diablo branch (named stable/diablo) is currently passively maintained. Fixes proposed to the branch will be approved by stable-maint if the issue is particularly serious. Security fixes are a good example of the kind of fixes which are appropriate for the Diablo branch at this point. The Diablo branch will reach End Of Life when Folsom is released.
  
The branch maintainer(s) will monitor master for changes and periodically cherry-pick patches into the branch. They may choose to stop actively monitoring master after a period (e.g. 2 months) if they feel there are so few backport candidates that it is no longer worth the effort. If so, they will notify developers via the mailing list.
+
The Essex branch (named stable/essex) is currently actively maintained. The stable-maint team monitors the master branch for suitable fixes and backports them. The team also reviews and approves backports proposed by other developers.
  
All changes to the stable branch will go through gerrit. The branch maintainers and core teams of each project can +2 the changes.
+
[[OpenStack]]'s stable branch policy borrows much from prior art. There are many projects which maintain such a branch and it's possible to build a picture of a rough consensus across these projects on how a stable branch should operate. One example of another project's policy is [http://github.com/torvalds/linux/blob/master/Documentation/stable_kernel_rules.txt the Linux kernel's stable_kernel_rules.txt].
  
Every change on the stable branch must pass all the tests.
+
== Appropriate Fixes ==
  
Security patches will follow a different process, whereby the patch can be embargoed and pushed at the same time as the patch on master, once the issue has been made public.
+
Only a limited class of changes are appropriate for inclusion on the stable branch.
  
In Essex, the RBP will be when the RCs are considered "good enough". It is at that point the stable maintainers need to start monitoring master. For Diablo, the PTLs and release manager did this work. That implies that the PTLs and RM will initially take an active interest in the stable branch, but that interest will want after a time. Which makes perfect sense.
+
A number of factors must be weighed when considering a change:
  
== Policy ==
+
* The risk of regression - even the tiniest changes carry some risk of breaking something and we really want to avoid regressions on the stable branch
 +
* The user visible benefit - are we fixing something that users might actually notice and, if so, how important is it?
 +
* How self-contained the fix is - if it fixes a significant issue but also refactors a lot of code, it's probably worth thinking about what a less risky fix might look like
 +
* Whether the fix is already on master - a change must be a backport of a change already merged onto master, unless the change simply does not make sense on master
  
The most obvious question we need to answer is "What belongs in a stable branch?"
+
The stable-maint team need to balance the risk of any given patch with the value that it will provide to users of the stable branch. A large, risky patch for a major data corruption issue might make sense. As might a trivial fix for a fairly obscure error handling case.
  
However, we don't need to ignore prior art here. Plenty of projects maintain such a branch and there's a very rough consensus across projects on what the policy should be.
+
Some types of changes are completely forbidden:
  
As an example, [http://github.com/torvalds/linux/blob/master/Documentation/stable_kernel_rules.txt stable_kernel_rules.txt] says:
+
* New features
 +
* Changes to the external HTTP APIs
 +
* Changes to Nova's internal AMQP API
 +
* DB schema changes
 +
* Incompatible config file changes
  
* It must be obviously correct and tested.
+
== Gerrit ==
* It cannot be bigger than 100 lines, with context.
 
* It must fix only one thing.
 
* It must fix a real bug that bothers people (not a, "This could be a problem..." type thing).
 
* It must fix a problem that causes a build error [..], an oops, a hang, data corruption, a real security issue, or some "oh, that's not good" issue.  In short, something critical.
 
* No "theoretical race condition" issues, unless an explanation of how the race can be exploited is also provided.
 
* It cannot contain any "trivial" fixes in it (spelling changes, whitespace cleanups, etc).
 
* It or an equivalent fix must [...] (upstream).
 
  
We can also add some rules specific to [[OpenStack]]:
+
Each backported commit proposed to gerrit should be reviewed and +2ed by two stable-maint members before it is approved. Where a stable-maint member has backported a fix, a single other +2 is sufficient for approval.
  
* No changes to the external HTTP APIs
+
If unsure about a given fix, stable-maint members should consult with the appropriate core developers for a more detailed technical review.
* No changes to the internal AMQP API
 
* No DB schema changes
 
  
The branch is intended to be a safe source of fixes for significant issues which have been fixed on master since the release. New features, regressions, compatibility issues, subtle behavioural changes, etc. have no place here.
+
Existing core developers are greatly encouraged to join the stable-maint team in order to help with reviewing backports, judging their appropriateness for the stable branch and approving them.
  
The maintainers need to balance the risk of any given patch with the value that it will provide to users of the stable branch. A large,
+
== Security Fixes ==
risky patch for a major data corruption issue might make sense. As might a trivial fix for a fairly obscure error handling case.
 
  
== Reviews ==
+
Fixes for embargoed security issues receive special treatment. These should be reviewed in advance of disclosure by core developers and stable-maint. At the time of co-ordinated public disclosure, the fix is proposed simulataneously to master and the stable branches and immediately approved.
  
The review process is aligned with the same workflow as trunk, +/-1 reviews are welcomed from anybody.  Please follow the [https://review.openstack.org/#q,status:open+branch:stable/diablo,n,z queue] for outstanding merge proposals.
+
== Releases ==
  
A stable branch reviewer should only +2 if:
+
When a project's stable branch accumulates a sufficient number of fixes, the stable-maint and release teams will work together publish a point release. See [[StableBranchRelease]] for details about how the 2011.3.1 release was prepared.
  
# It fixes a significant issue, seen, or potentially seen, by someone during real life use
+
== Joining the Team ==
# The fix, or equivalent, must be in master already
 
# The fix was either a fairly trivial cherry-pick that looks equally correct for the stable branch, or that the fix has sufficient technical review (e.g. a +1 from another stable reviewer if it's fairly straightforward, or one or more +1s from folks on core it it's really gnarly)
 
# If this reviewer proposed the patch originally, another stable branch reviewer should have +1ed it
 
  
 
If you think you've got what it takes to apply these rules, do a few reviews and [https://launchpad.net/~openstack-stable-maint apply for membership]. All you need is an understanding of this policy and reasonable judgement, it's not rocket science!
 
If you think you've got what it takes to apply these rules, do a few reviews and [https://launchpad.net/~openstack-stable-maint apply for membership]. All you need is an understanding of this policy and reasonable judgement, it's not rocket science!
 
 
== Workflow ==
 
== Workflow ==
  

Revision as of 21:36, 12 April 2012

Stable Branch

The stable-maint team manages the stable branch for OpenStack core projects with the support of the core developers of those projects and the release team.

<<TableOfContents()>>

Overview

The stable branch is intended to be a safe source of fixes for high impact bugs and security issues which have been fixed on master since a given release.

Official point releases for each project are published from the branch as sufficient fixes accumulate on the branch.

The Diablo branch (named stable/diablo) is currently passively maintained. Fixes proposed to the branch will be approved by stable-maint if the issue is particularly serious. Security fixes are a good example of the kind of fixes which are appropriate for the Diablo branch at this point. The Diablo branch will reach End Of Life when Folsom is released.

The Essex branch (named stable/essex) is currently actively maintained. The stable-maint team monitors the master branch for suitable fixes and backports them. The team also reviews and approves backports proposed by other developers.

OpenStack's stable branch policy borrows much from prior art. There are many projects which maintain such a branch and it's possible to build a picture of a rough consensus across these projects on how a stable branch should operate. One example of another project's policy is the Linux kernel's stable_kernel_rules.txt.

Appropriate Fixes

Only a limited class of changes are appropriate for inclusion on the stable branch.

A number of factors must be weighed when considering a change:

  • The risk of regression - even the tiniest changes carry some risk of breaking something and we really want to avoid regressions on the stable branch
  • The user visible benefit - are we fixing something that users might actually notice and, if so, how important is it?
  • How self-contained the fix is - if it fixes a significant issue but also refactors a lot of code, it's probably worth thinking about what a less risky fix might look like
  • Whether the fix is already on master - a change must be a backport of a change already merged onto master, unless the change simply does not make sense on master

The stable-maint team need to balance the risk of any given patch with the value that it will provide to users of the stable branch. A large, risky patch for a major data corruption issue might make sense. As might a trivial fix for a fairly obscure error handling case.

Some types of changes are completely forbidden:

  • New features
  • Changes to the external HTTP APIs
  • Changes to Nova's internal AMQP API
  • DB schema changes
  • Incompatible config file changes

Gerrit

Each backported commit proposed to gerrit should be reviewed and +2ed by two stable-maint members before it is approved. Where a stable-maint member has backported a fix, a single other +2 is sufficient for approval.

If unsure about a given fix, stable-maint members should consult with the appropriate core developers for a more detailed technical review.

Existing core developers are greatly encouraged to join the stable-maint team in order to help with reviewing backports, judging their appropriateness for the stable branch and approving them.

Security Fixes

Fixes for embargoed security issues receive special treatment. These should be reviewed in advance of disclosure by core developers and stable-maint. At the time of co-ordinated public disclosure, the fix is proposed simulataneously to master and the stable branches and immediately approved.

Releases

When a project's stable branch accumulates a sufficient number of fixes, the stable-maint and release teams will work together publish a point release. See StableBranchRelease for details about how the 2011.3.1 release was prepared.

Joining the Team

If you think you've got what it takes to apply these rules, do a few reviews and apply for membership. All you need is an understanding of this policy and reasonable judgement, it's not rocket science!

Workflow

Proposing Fixes

Anyone can propose a cherry-pick to the maintainers. This helps ensure that the maintainers don't miss anything. To catch the maintainers attention, simply add


Cc: stable-maintainers


to the commit message. If the patch you're proposing will not cherry-pick cleanly, you can help by resolving the conflicts yourself and proposing the resulting patch.

You can use git-review to propose a change to the stable branch with:


git review stable/diablo


If a bug in launchpad looks like a good candidate for backporting - e.g. if it's a significant bug with the previous release - then just tagging it with 'diablo-backport' will bring it to the attention of the maintainers e.g. for nova and for glance. The tag will be removed when the fix has been applied to the stable branch, or rejected as unsuitable.

Email Notifications

If you want to be notified of these patches you can create a watch on this screen: https://review.openstack.org/#settings,projects with the settings:

Project Name: All-Projects
     Only If: message:"stable-maintainers"


And then check the "Email Notifications - New Changes" checkbox. That will cause gerrit to send an email whenever a matching change is proposed, and better yet, the change shows up in your 'watched changes' list in gerrit.

See the docs for Watched Projects configuration and the search syntax.

Change-Ids

When cherry-picking a commit, keep the original Change-Id and gerrit will show a separate review for the stable branch while still allowing you to use the Change-Id to see all the reviews associated with it. See this change as an example.

Stale .pyc Files From Migrations

Switching between master and diablo is a pain because you end up with .pyc files for migrations that exist only on master and diablo blows up when trying to run them. Delete them with:

 $> rm -f nova/db/sqlalchemy/migrate_repo/versions/*.pyc

Reviewing Changes On Master

Try e.g.


 $> git log --graph 2011.3..origin/master


If you're looking at a sequence of commits on a branch which got merged into master and you want to see exactly what changes were merged in, then look at the merge commit:


 commit f5f429bac6446db084ca3f5d86c64127a1e539f2
 Merge: e95e923 f3fb16a


and do:


 $> git diff e95e923..f5f429b


Keeping Notes

When reviewing the master branch for candidates to backport, you probably want to keep notes on which commits do and don't look good for cherry-picking. Even if it's just a case of keeping track of which was the latest commit on master you reviewed. You may also be curious about another maintainers opinion on a given patch.

git notes allows you to maintain notes on commits in the same git repository as the commits themselves. It's exactly what's needed here, but it can be a bit of a pain to use.

To fetch my notes on nova, do:


 $> git remote add markmc-notes git://github.com/markmc/nova.git
 $> git config remote.markmc-notes.fetch +refs/notes/diablo-notes:refs/notes/markmc/diablo-notes
 $> git fetch markmc-notes


You now have my notes in ref/notes/markmc/diablo-notes. To look at them, do either:


 $> git log --notes=markmc/diablo-notes


Or:


 $> export GIT_NOTES_DISPLAY_REF=refs/notes/markmc/diablo-notes
 $> git log


You'll see something like:


commit 1b7fba648aa3eb4cdda345237c9f77dc0b229329
Author: Naveed Massjouni [...]
Date:   Wed Oct 26 14:32:48 2011 -0400

    Adding support for retrying glance image downloads.
    
    Change-Id: Ifff40d90f7dc61a6d41ae2d6908d6e1e6f0aea7e

Notes (markmc/diablo-notes):
    New feature.
    
    Action: pass


(The values of this Action tag I'm using are: pass = "Don't cherry-pick", pick = "Do cherry-pick", proposed = "Already cherry-picked and proposed in gerrit", merged = "Already cherry-picked and merged into stable" and defer = "Consider cherry-picking later")

If you want to start maintaining your own notes, you can do e.g.


 $> export GIT_NOTES_REF=refs/notes/diablo-notes
 $> git notes add 261b4111d4
 $> git notes edit 261b4111d4
 $> git notes show 261b4111d4
 $> git show 261b4111d4


and to push them to a remote repo:


 $> git push ${myrepo} refs/notes/diablo-notes:refs/notes/diablo-notes