StarlingX/Security/Vulnerability Management

Vulnerability Management Process
The StarlingX Vulnerability Management Team (VMT) is responsible for coordinating the progressive disclosure of a vulnerability.

Members of the team are independent and security-minded folks who ensure that vulnerabilities are dealt with in a timely manner and that downstream stakeholders are notified in a coordinated and fair manner. Where a member of the team is employed by a downstream stakeholder, the member does not give their employer prior notice of any vulnerabilities. In order to reduce the disclosure of vulnerability in the early stages, membership of this team is intentionally limited to a small number of people.

Supported versions
The Vulnerability Management team coordinates patches fixing vulnerabilities in supported stable branches (corresponding to previous major releases) of StarlingX, in addition to the master branch (next version under development), for all security supported projects.

Process
Each security bug is assigned a VMT coordinator (member from the vulnerability management team) that will drive the fixing and disclosure process. Here are the steps we follow.



Reception
A report can be received either as a private encrypted email to one of the VMT members, or as a Launchpad security bug (check the box marked “this is a security issue”). Follow the instructions on how to create a security launchpad detailed at https://wiki.openstack.org/wiki/StarlingX/Security#How_to_report_security_issues_to_StarlingX

The first steps performed by the VMT are to confirm the validity of the report, create a Launchpad bug if necessary, prefix the description with an embargo reminder, add an stxsa bugtask and subscribe the project’s core security review team for confirmation of impact and determination of affected branches. Reports starting with an Incomplete stxsa bugtask should have a corresponding incomplete reception message added in a comment. Once the VMT confirms an STXSA is warranted, the stxsa bugtask status will be set to Confirmed. If the need for an STXSA is challenged, the stxsa bugtask status should be set back to Incomplete until that question is resolved.

For some lower-risk issues or problems which may only be easy to solve in future releases, the stxsa bugtask will be set to Opinion and the core security reviewers for the StarlingX Security team will be subscribed to determine whether they wish to issue an STXSN (these reports may still sometimes remain under embargo until the STXSN is issued). If no STXSA is warranted and there is no benefit to an STXSN then the stxsa bugtask will be set to Won’t Fix or Invalid (depending on the specific situation) and the bug state switched from Private Security to Public, optionally adding the security bug tag if the report concerns a potential security hardening opportunity. The specifics are indexed in the report taxonomy and task status tables.

Patch development
For a private report, the reporter (automatic if reported directly as a bug) and the affected projects’ core security review teams plus anyone they deem necessary to develop and validate a fix are added to the bug’s subscription list. A fix is proposed as a patch to the current master branch (as well as any affected supported branches) and attached to the private bug report, not sent to the public code review system.

For public reports, there is no need to directly subscribe anyone and patches can be submitted directly to the code review system instead of as bug attachments (though the bug should be referenced in any commit messages so it will be updated automatically).

If project-side delays are encountered at this or any subsequent stage of the process, the VMT and other interested parties may reach out to that project’s Vulnerability Management Liaison requesting more immediate attention to the issue.

Patch review
For a private report once the initial patch has been attached to the bug, core reviewers on the subscription list from the project in question should review it and suggest updates or pre-approve it for merging. Privately-developed patches need to be pre-approved so that they can be fast-tracked through public code review later at disclosure time.

For public reports, StarlingX's usual public code review and approval processes apply.

Draft impact description
In the mean time, the VMT coordinator prepares a vulnerability description that will be communicated to downstream stakeholders, and will serve as the basis for the Security Advisory that will be finally published.

The description should properly credit the reporter, specify affected versions (including unsupported ones) and accurately describe impact and mitigation mechanisms. The VMT coordinator should use the template below. Once the description is posted, the stxsa bugtask status should be switched to Triaged.

Review impact description
The description is validated by the reporter and the PTL.

Send CVE request
To ensure full traceability, we attempt to obtain a CVE assignment before the issue is communicated to a larger public. This is generally done as the patch gets nearer to final approval. The stxsa bugtask status is set to In progress and the approved impact description is submitted through MITRE’s CVE Request form. The request type is Request a CVE ID, the e-mail address should be that of the requester (generally the assigned VMT coordinator in the case of reports officially managed by the VMT), and for embargoed reports the coordinator’s OpenPGP key should be pasted into the field provided.

In the required section set the checkboxes indicating the product is not CNA-covered and that no prior CVE ID has been assigned, select an appropriate vulnerability type (using Other or Unknown to enter a freeform type if there is nothing relevant on the drop-down), set the vendor to StarlingX, and the product and version fields to match the $PROJECTS and $AFFECTED_VERSIONS from the impact description. In the optional section set the radio button for confirmed/acknowledged to Yes, choose an appropriate attack type in the drop-down (often this is Context-dependent for our cases), check the relevant impact checkboxes, attempt to fill in the affected components and attack vector fields if possible, paste in the suggested description from the prose of the impact description (usually omitting the first sentence as it’s redundant with other fields), put the $CREDIT details in the discoverer/credits field, and the bug URL (along with Gerrit URLs for patches if already public) in the references field. If the report is still private, note that in the additional information field like This report is currently under embargo and no disclosure date has been scheduled at this time.

At the bottom of the page, fill in the security code and click the submit request button. If some fields contain invalid data they will be highlighted red; correct these, update the security code and submit request again until you get a confirmation page.

Get assigned CVE
MITRE returns the assigned CVE. It is added to the Launchpad bug (see “link to CVE” at the top-right), and the bug is retitled to “$TITLE ($CVE)”.

Embargoed disclosure
Once the patches are approved and the CVE is assigned, a signed email with the vulnerability description is sent to the downstream stakeholders. The disclosure date is set to 3-5 business days, excluding Monday/Friday and holiday periods, at 1500 UTC. No stakeholder is supposed to deploy public patches before disclosure date.

Once the email is sent, the stxsa bugtask status should be set to Fix committed. At that point we can also add downstream stakeholders to the Launchpad bug, if they use Launchpad for security patches. This means adding ~canonical-security to the bug subscribers.

For non-embargoed, public vulnerabilities no separate downstream advance notification is sent. Instead the stxsa bugtask is set to fix committed status once the CVE assignment is received STXSA is drafting begins immediately.

Open bug, Push patch
In preparation for this, make sure you have a core reviewer and a stable maintainer available to help pushing the fix at disclosure time.

On the disclosure hour, open bug, push patches to Gerrit for review on master and supported stable branches, fast-track approvals (referencing the bug).

Update the Launchpad bug title to “[STXSA-$NUM] $TITLE”.

Embargo reminder can be removed at that point.

MITRE’s CVE Request form should be used again at this point, but instead select a request type of Notify CVE about a publication and fill in the coordinator’s e-mail address, provide a link to the advisory (the URL to it on https://security.openstack.org/ if this was an official STXSA), the CVE IDs covered, and the date published. Once more, fill in the security code at the bottom of the page and submit request.

Publish STXSA
Shortly after pushing the patches (potentially waiting for the first test runs to complete), publish the advisory to the StarlingX ML. Wait until all patches merged to supported branches before setting the stxsa bugtask status to Fix released.

All patches merged
Patches approved in code review do not necessarily merge immediately, but should be tracked closely until they do (if the bug number is correctly identified in commit messages then it will be automatically updated to reflect this as well). Subsequent security point releases of affected software may then be tagged if warranted.

Incident Report Taxonomy
The VMT is now using this classification list in order to assist vulnerability report triage, especially whenever a bug does not warrant an advisory.

STXSA Task status
Here is a summary of the different STXSA task status meanings:

Extent of Disclosure
The science of vulnerability management is somewhere around being able to assess impact and severity of a report, being able to design security patches, being an obsessive process-following perfectionist and respecting the rule of lesser disclosure.

Lesser disclosure is about disclosing the vulnerability details to an increasing number of people over time, but only to the people that are necessary to reach the next step. The diagram above shows “disclosure extent” across the various steps of the process.

Vulnerability reporters retain final control over the disclosure of their findings. If for some reason they are uncomfortable with our process, their choice of disclosure terms prevails.

Embargo exceptions
To keep the embargo period short and effective, the VMT may choose to open bug reports. Issues that take too much time to be fixed (e.g., more than 2 weeks) or issues that require a complex patch are usually better solved in the open.

Downstream stakeholders
StarlingX as an upstream project is used in a number of distributions, products, private and public service offerings that are negatively affected by vulnerabilities. In the spirit of responsible disclosure, this ecosystem, collectively known as the downstream stakeholders, needs to be warned in advance to be able to prepare patches and roll them out in a coordinated fashion on disclosure day. The embargo period is kept voluntarily small (3-5 business days), as a middle ground between keeping the vulnerability under cover for too long and not giving a chance to downstream stakeholders to react.

If you’re currently not a referenced stakeholder and think you should definitely be included on that email distribution list, please submit an email with a rationale to member(s) of the VMT.

Reception incomplete message (unconfirmed issues)
Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

Reception embargo reminder (private issues)
This issue is being treated as a potential security risk under embargo. Please do not make any public mention of embargoed (private) security vulnerabilities before their coordinated publication by the OpenStack Vulnerability Management Team in the form of an official OpenStack Security Advisory. This includes discussion of the bug or associated fixes in public forums such as mailing lists, code review systems and bug trackers. Please also avoid private disclosure to other individuals not already approved for access to this information, and provide this same reminder to those who are made aware of the issue prior to publication. All discussion should remain confined to this private bug report, and any proposed fixes should be added to the bug as attachments.

Impact description ($DESCRIPTION)
The AFFECTED_VERSIONS needs to stay valid after the fix is released. For example, when kilo, liberty and mitaka are still security supported, the AFFECTED_VERSIONS of keystone should read like this: Once kilo reaches end of life, that line becomes: If the oldest version affected is not easily identified, leave it open-ended:

Downstream stakeholders notification email (private issues)
We send two separate emails, to avoid off-topic replies to linux-distros:
 * To: embargo-notice@lists.openstack.org
 * To: linux-distros@vs.openwall.org

Subject and content for both emails is identical: Proposed patches are attached, email must be GPG-signed. Use something unique and descriptive for the patch attachment file names, for example cve-2013-4183-master-havana.patch or cve-2013-4183-stable-grizzly.patch.

OpenStack security advisories (STXSA)
The document is first submitted as a yaml description to the ossa project using this template:

Once approved, use the ‘Show Source’ button from the gate-ossa-docs output to get the generated RST document. We send two separate emails, to avoid off-topic replies to oss-security list: Subject and content for both emails is identical: Notes:
 * To: openstack-announce@lists.openstack.org, openstack@lists.openstack.org
 * To: oss-security@lists.openwall.com
 * Subject: [STXSA-$NUM] $TITLE ($CVE)
 * Body: The generated RST document
 * Email must be GPG-signed.
 * $CVE must always be of the form CVE-YYYY-XXXX
 * $NUM is of the form YYYY-XX