Jump to: navigation, search

Nova/Liberty Release Schedule

< Nova
Revision as of 11:55, 15 July 2015 by John Garbutt (talk | contribs) (→‎FAQs)

We are aligning with: Liberty_Release_Schedule But there are some extra deadlines around Priority features.

If you are new to Nova, please read this first: https://wiki.openstack.org/wiki/Nova/Mentoring


Dates overview

  • June 23-25: liberty-1 -- spec freeze for L, backlog stays open
  • July 16: non-priority feature proposal freeze
  • July 21-23: mid-cycle meetup
  • July 30: liberty-2 -- non-priority feature freeze
  • August 17-21: Feature Proposal Freeze
  • September 1-3: liberty-3 -- align with string freeze, etc, open specs for M


OpenStack wide tracking of current progress: http://status.openstack.org/release/

Special review days

To make the above gates go smoother, we have some big review pushes:


All of these involve hanging out in #openstack-nova and working together to try and improve the statistics mentioned above.

Feature Freeze

This effort is primarily to help the horizontal teams help prepare their items for release, while at the same time giving developers time to focus on stabilising what is currently in master, and encouraging users and packages to perform tests (automated, and manual) on the release, to spot any major bugs.

As such we have the following processes:


We align with this in Nova and the dates for this release are stated above.

As with all processes here, there are exceptions. But the exceptions at this stage need to be discussed with the horizontal teams that might be affected by changes beyond this point, and as such are discussed with one of the OpenStack release managers.

Spec and Blueprint Approval Freeze

This is a (mostly) nova specific process.

Why do we have a Spec Freeze:

  • specs take a long time to review, keeping it open distracts from code reviews
  • keeping them "open" and being slow at reviewing the specs (or just ignoring them) really annoys the spec submitters
  • we generally have more code submitted that we can review, this time bounding is a way to limit the number of submissions

By the freeze date, we expect this to also be the complete list of approved blueprints for liberty: https://blueprints.launchpad.net/nova/liberty

The date listed above is when we expect all specifications for Liberty to be merged and displayed here: http://specs.openstack.org/openstack/nova-specs/specs/liberty/approved/

New in Liberty, we will keep the backlog open for submission at all times. Note: the focus is on accepting and agreeing problem statements as being in scope, rather than queueing up work items for the M release. We still working on a new lightweight process to get our of the backlog and approved for a particular release. For more details on backlog specs, please see: http://specs.openstack.org/openstack/nova-specs/specs/backlog/index.html

There can be exceptions, usually its an urgent feature request that comes up after the initial deadline. These will generally be discussed at the weekly Nova meeting, by adding the spec or blueprint to discuss in the appropriate place in the meeting agenda here (ideally make yourself available to discuss the blueprint, or alternatively make your case on the ML before the meeting): https://wiki.openstack.org/wiki/Meetings/Nova

Non-priority Blueprint Feature Freeze

This is a nova specific process.

We currently have a very finite amount of review bandwidth. In order to make code review time for the agreed community wide priorities, we have to not do some other things. To this end, we are reserving liberty-3 for priority features and bug fixes. As such, we indent not to merge any non-priority things during liberty-3, so liberty-2 is the "Feature Freeze" for blueprints that are not a priority for liberty.

You can see the list of priorities for each release: http://specs.openstack.org/openstack/nova-specs/#priorities

For things that are very close to merging, its possible it might get an exception for one week after the freeze date, given the patches get enough +2s from the core team to get the code merged. But we expect this list to be zero, if everything goes to plan (no massive gate failures, etc). Exceptions will be discussed in the nova-meeting following the freeze date, after the procedural -2s will be applied to any outstanding non-priority blueprint patches. You can add your blueprint for discussion, by adding it to the list on this etherpad: https://etherpad.openstack.org/p/liberty-nova-priorities-tracking

Alternatives:

  • It was hoped to make this a continuous process using "slots" to control what gets reviewed, but this was rejected by the community when it was last discussed. There is hope this can be resurrected to avoid the "lumpy" nature of this process.

How do I get my code merged?

OK, so you are new to Nova, and you have been given a feature to implement. How do I make that happen?

So you can get most of your questions answered here:

But lets put a Nova specific twist on things...

Overview

Nova spec process.png

Where do you track bugs?

We track bugs here:

If you fix an issue, please raise a bug so others that spot that issue can find the fix you kindly created for them.

Also, before submitting your patch, its worth checking to see if someone has already fixed it for you (launchpad helps you with that, at little, when you create the bug report).

When do I need a blueprint vs a spec?

For more details see:

To understand this question, we need to understand why blueprints and specs are useful.

But here is a rough idea:

  • if it needs a spec, it will need a blueprint
  • if its an API change, it needs a spec
  • if its a single small patch that touches a small amount of code, with limited deployer and doc impact, it probably doesn't need a spec


If you are unsure, please ask johnthetubaguy on IRC, or one of the other nova-drivers.

How do I get my blueprint approved?

So you need your blueprint approved? Here is how:

Got any more questions? Contact johnthetubaguy on IRC for help, or one of the other nova-specs-core who are awake at the same time as you, or send him an email.

Why are the reviewers being mean to me?

They don't mean to be mean, honest! They are just trying to make sure you are solving the problem you intend, and that you are also not creating problems for other users at the same time, and making sure we keep the code maintainable.

TODO - need more details

My code reviews seems to have got stuck, what can I do?

So you have a -1 or a -2 but the reviewer is not responding (or the patch author is not responding) or you are failing to reach common ground, what can be done?

Well raise it on IRC or via email. If that doesn't work, add it into the agenda for the next nova meeting you are able to attend: https://wiki.openstack.org/wiki/Meetings/Nova

If you code is not getting any attention be sure that you have:

  • nova is a big project, be aware of the average wait times
  • put your blueprint in the commit message
    • ensure your blueprint is marked as NeedsCodeReview if you are finished
    • ensure the whiteboard on the blueprint is easy to understand in terms of what patches need reviewing (ideally all in a single gerrit topic)
  • or put a bug in your commit message
    • if its a bug, share the fact you have fixed an issue by raising a bug in launch pad
    • make sure it has the correct priority set and bug tag set
    • for more details see:
  • Is your patch for a priority, or could it get recommended by a subteam, or is it a trivial fix?
  • if non of the above apply, reach out on IRC for advice on how you get more attention for your patch

Nova Process Mission

This section takes a high level look at the guiding principles behind the Nova process.

Open

Our mission is to have:

  • Open Source
  • Open Design
  • Open Development
  • Open Community


We have to work out to keep communication open in all areas. We need to be welcoming and mentor new people, and make it easy for them to pickup the knowledge they need to get involved with OpenStack. For more info on Open, please see: https://wiki.openstack.org/wiki/Open

Interoperable API, supporting a vibrant ecosystem

An interoperable API that gives uses on demand access to compute resources is at the heart of Nova's mission: http://docs.openstack.org/developer/nova/project_scope.html#mission

Nova has a vibrant ecosystem of tools built on top of the current Nova API. All features should be added such that they can work with all technology combinations, so the feature can be adopted by our ecosystem. If a new feature is not adopted by the ecosystem, it will make it hard for your users to make use of those features, defeating most of the reason to add the feature in the first place. The microversion system allows users to isolate they selves

This is a very different aim to being "pluggable" or wanting to expose all capabilities to end users. At the same time, it is not just a "lowest common denominator" APIs. It should be discoverable what features are available, and while no implementation details should leak to the end users, purely admin concepts may need to understand technology specific details that back that implement the interoperable and more abstract concepts that are exposed to the end user. This is a hard goal, and one area we currently don't do well is isolating image creators from these technology specific details.

Smooth Upgrades

As part of our mission for a vibrant ecosystem around out APIs, we want to make it easy for those deploying Nova to upgrade with minimal impact to their users. Here is the scope of Nova's upgrade support:

  • upgrade from any commit, to any future commit, within the same major release
  • only support upgrades between N and N+1 major versions, to reduce technical debt relating to ugprades

Here are some of the things we require developers to do, to help with upgrades:

  • when replacing an existing feature of configuration option make it clear how to transition to any replacement
  • deprecate configuration options and features before removing them
    • i.e. continue support and test features for at least one release, before they are removed
    • gives time for operator feedback on any removals
  • End User API will always be kept backwards compatible

Interaction goals

When thinking about the importance of process, we should take a look at: http://agilemanifesto.org

With that in mind, lets look at how we want different members of the community to interact. Lets start with looking at issues we have tried to resolve in the past (currently in no particular order):

  • we want a way for everyone to review blueprints and designs, including allowing for input from operators and all types of users (keep it open)
  • we need to take care to not expand Nova's scope any more (unless absolutely necessary)
  • need to ensure we get sufficient focus on the core of Nova so that we can maintain or improve the stability and flexibility of overall the codebase
  • any API we release, we must support (approx.) for ever, and we currently release every commit, so we best get the API right first time
  • need to avoid low priority blueprints slowing work on high priority work, but ideally only ever blocked for a single cycle, and those blocked should be able to help unblock
  • focus on a consistent experience for our users, rather than ease of development
  • optimise for completed blueprints, rather than more half completed blueprints, so we get maximum value for our users out of our review bandwidth
  • focused core reviewers tend to be more productive, and get more things merged, so try to focus efforts on a subset of patches
  • ideally tell people sooner, rather than later, that their work is unlikely to get reviewed in a particular cycle, to avoid sitting in an expensive rebase loop
  • be aware of users that do not work on the project full time
  • be aware of users that are only able to work on the project at certain times that may not align with the overall idea for the community
  • in some cases its best to discuss the design before you start coding, or at least get some good feedback about your approach, and where to start

FAQs

Why bother with all this process?

We are a large community, spread across multiple timezones, working with several horizontal teams. Good communication is a challenge, and the processes we have are mostly there to try and help fix some communication challenges.

If you have a problem with a process, please engage with the community, discover the reasons behind our current process, and help fix the issues you are experiencing.

Why do you not remove old process?

We do removal old process. Please see how in liberty we stopped trying to predict the milestones when a feature will land.

As we evolve, it is important to unlearn new habits, explore if things get better if we choose to optimise for a different set of issues.

Why are specs useful?

Specs reviews allow anyone to step up an contribute to reviews, just like with code. Before we used gerrit, it was a very messy review process, that felt very "closed" to most people involved in that process.

As Nova has grown in size, it can be hard to work out how to modify Nova to meet your needs. Specs are a great way of having that discussion with the wider Nova community.

For Nova to be a success, we need to ensure we don't break our existing users. The spec template helps focus the mind on the impact your change might have on exisitng users, and gives an opportunity to discuss the best way to deal with those issues.

However, there are some pitfalls with the process. Here are some top tips to avoid them:

  • keep it simple, shorter, simpler decomposed specs are much quicker to review and merge much quicker (just like code patches)
  • specs can help with documentation, but they are only intended to document the design discussion, rather than document the final code
  • don't add details that are best reviewed in code, almost always best leaving those things for the code review

If we have specs, why still have blueprints?

We use specs record the design agreement, we use blueprints to track progress on the implementation of the spec.

Currently, in Nova, specs are only approved for one release, and must be re-submitted for each release you want to merge the spec, although that is currently under review.

Why do we have priorities?

To be clear, there is no "nova dev team manager", we are an open team of professional software developers, that all work for a variety of (mostly completing) companies that collaborate to ensure the Nova project is a success.

Over time, a lot of technical debt has accumulated, because there was a lack of collected ownership to solve those cross cutting concerns. Before the kilo release, it was noted that progress felt much slower, because we were unable to get appropriate attention on the architectural evolution of Nova. This was important, partly for major concerns like upgrades and stability. We agreed its something we all cared about, and needs to be given priority to ensure it these things get fixed.

The discussion at the summit, turns in to a spec review, which eventually means we get a list of priorities here: http://specs.openstack.org/openstack/nova-specs/#priorities

This does mean we need review bandwidth for all these efforts, which means the reviews needs to stop doing something else, to make time to review these priority items. This is mostly why we now have the non-priority Feature Freeze.

Blocking a priority effort is one of the few widely acceptable reasons to block someone adding a feature. One of the great advantages of being more explicit about that relationship, is that people can step up to help review and/or implement the work that is needed to unblock the feature they want to get landed. This is a key part of being an Open community.

Why is there a Feature Freeze (and String Freeze) in Nova?

The main reason Nova has a feature freeze, as it gives people working on docs and translations to sync up with the latest code. Traditionally this happens at the same time across multiple projects, so the docs are synced between what used to be called the "integrated release".

We also use this time period as an excuse to focus our development efforts on bug fixes, ideally lower risk bug fixes, and improving test coverage.

In theory, with a waterfall hat on, this would be a time for testing and stabilisation of the product. In Nova we have a much stronger focus on keeping every commit stable, by making use of extensive continuous testing. In reality, we frequently see the biggest influx of fixes in the few weeks after the release, as distributions do final testing of the released code.

It is hoped the work on Feature Classification will lead us to better understand the levels of testing of different Nova features, so we will be able to reduce and dependency between Feature Freeze and regression testing. It is also likely that the move away from "integrated" releases will help find a more developer friendly approach to keep the docs and translations in sync.

Why is there a non-priority Feature Freeze in Nova?

We have already discussed why we have priority features.

The Nova project, while growing at a surprisingly fast rate, still has a limited ability to merge code, due to the amount of people willing to commit time to review code. Given this fixed capacity, making something a priority, means not doing something else.

The simplest way to make space for the priority features is to stop reviewing and merging non-priority features for a whole milestone. The idea being developers should focus on bug fixes and priority features during that milestone, rather than working on non-priority features.

A known limitation of this approach is developer frustration. Many developers are not being given permission to review code, work on bug fixes or work on priority features, and so feel very unproductive upstream. An alternative approach of "slots" or "runways" has been considered, that uses a kanban style approach to regulate the influx of work onto the review queue. We are yet to get agreement on a more balanced approach, so the existing system is being continued to ensure priority items are more likely to get the attention they require.

Why do you still use launchpad?

We are actively looking for an alternative to launchpad's bugs and blueprints.

Originally the idea was to create storyboard. However the development has stalled. A more likely front runner is this: http://phabricator.org/applications/projects/

When should I submit my spec?

Ideally we want to get all specs for a release merged before the summit. For things that we can't get agreement on, we can then discuss those at the summit.

Process Evolution Ideas

We are always evolving our process, as we try to improve and adapt to the changing shape of the community. Here we discuss some of the ideas, along with their pros and cons.

Splitting out the virt drivers (or other bits of code)

Currently, Nova doesn't have strong enough interface to split out virt drivers, nor to split out the scheduler, or the REST API. This is seen as the key blocker. Lets look at both sides of the debate here.

Reasons for the split:

  • can have separate core teams for each repo
    • leader to quicker turn around times, largely due to focused teams
  • splitting out things from core means less knowledge required to become core in a specific area


Reasons against the split:

  • fear of fragmenting the nova community, leaving few to work on the core of the project
  • loss of interoperability between drivers
    • could enforce some behaviour with stronger interfaces and functional tests
  • new features often need changes in the API and virt driver anyway
    • the new "depends-on" can make this cross repo dependency easier
  • loss code style of consistency across the code base
  • could work in subteams within the main tree


TODO - need to complete analysis

Subteam recommendation as a +2

There are groups of people with great knowledge of particular bits of the code base. It may be a good idea to give their recommendation of a merge. In addition, having the subteam focus review efforts on a subset of patches should help concentrate the nova-core reviews they get, and increase the velocity of getting code merged.

The first part is for subgroups to show they can do a great job of recommending patches. This is starting in there: https://etherpad.openstack.org/p/liberty-nova-priorities-tracking

Ideally this would be done with gerrit user "tags" rather than an etherpad. There are some investigations by sdague in how feasible it would be to add tags to gerrit.

Stop having to submit a spec for each release

As mentioned above, we use blueprints for tracking, and specs to record design decisions. Targeting specs to a specific release is a very heavyweight solution, and blurs the lines between specs and blueprints. At the same time, we don't want to loose the opportunity to revise existing blueprints. Maybe there is a better balance?

What about this kind of process:

  • backlog has these folders:
    • backlog/incomplete - merge a partial spec
    • backlog/complete - merge complete specs (remove assignee part of the template)
    • backlog/expired - specs are moved here from incomplete or complete when no longer seem to be given attention (after 1 year, by default)
    • <release>/implemented - when a spec is complete it gets moved into the release directory, and possibly updated to reflect what actually happened
    • there will no longer be a per release approved spec list


To get your blueprint approved:

  • add it to the next nova meeting
    • if a spec is required, update the URL to point to the spec merged in a spec to the blueprint
    • ensure there is an assignee in the blueprint
  • a day before the meeting, a note is sent to the ML to review the list before the meeting
  • any final objections discuss in the nova-meeting
    • this may result in a request to refine the spec, if things have changed since it was merged
  • trivial cases can be approved in advance by a nova-driver, so not all folks need to go through the meeting


This still needs more thought, but should decouple the spec review from the release process. It is also more compatible with a runway style system, that might be less focused on milestones.

Runways

Runways are a form of Kanban, where we look at optimising the flow through the system, by ensure we focus our efforts on reviewing a specific subset of patches.

The idea goes something like this:

  • define some states, such as: design backlog, design review, code backlog, code review, test+doc backlog, complete
  • blueprints must be in one of the above state
    • large or high priority bugs may also occupy a code review slot
  • core reviewer member moves item between the slots
    • must not violate the rules on the number of items in each state
    • states have a limited number of slots, to ensure focus
    • certain percentage of slots are dedicated to priorities, depending on point in the cycle, and the type of the cycle, etc


Reasons for:

  • more focused review effort, get more things merged more quickly
  • more upfront about when your code is likely to get reviewed
  • smooth out current "lumpy" non-priority feature freeze system


Reasons against:

  • feels like more process overhead
  • control is too centralised

Replacing Milestones with SemVer Releases

You can deploy any commit of Nova and upgrade to a later commit in that same release. Making our milestones versioned more like an official release would help signal to our users that people can use the milestones in production, and get a level of upgrade support.

It could go something like this:

  • 14.0.0 is milestone 1
  • 14.0.1 is milestone 2 (maybe, because we add features, it should be 14.1.0?)
  • 14.0.2 is milestone 3
  • we might do other releases (once a critical bug is fixed?), as it makes sense, but we will always be the time bound ones
  • 14.0.3 two weeks after milestone 3, adds only bug fixes (and updates to RPC versions?)
    • maybe a stable branch is created at this point?
  • 14.1.0 adds updated translations and co-ordinated docs
    • this is released from the stable branch?
  • 15.0.0 is the next milestone, in the following cycle
    • not the bump of the major version to signal an upgrade incompatibility with 13.x


We are currently watching ironic to see how their use of semver goes, and see what lessons need to be learnt before we look to maybe apply this technique during M.

Retrieved from "https://wiki.openstack.org/w/index.php?title=Nova/Liberty_Release_Schedule&oldid=86057"