[[HypervisorSupportMatrix]] | [[NovaVMware]]

== Current Service Status ==
[[File:Sweeper-ok.png]] No known issues at this time
[[File:Sweeper-notok.png]] CI is offline or running with limited capacity

== Blocking Bugs ==
These bugs may be causing Minesweeper to fail in the form of failing tests. If there are fixes available, they need to be merged soon for Minesweeper to run correctly.

{| border="1" cellpadding="10"
|-
! Severity
! Bug
! Fix (needs review attention)
|-
|}

== Service Disruption History ==
All times are in PST.

{| border="1" cellpadding="10"
|-
! Event Time
! Description
|-
| Sep 16, 2:54 PM
| Minesweeper is commenting but not voting at the moment. We are in the process of addressing.
|-
| Aug 1, 3:06 PM
| Minesweeper is back up and running after cloud issues were resolved.
|-
| July 29, 4:07 PM
| Our cloud is down for maintenance. This page will be updated once maintenance is complete.
|-
| July 31, 11:03 PM
| We are experiencing networking issues after the upgrade and are currently working to get these issues fixed so we can resume the CI. Please stay tuned.
|-
| July 28, 9:54 PM
| Our cloud team was able to fix infrastructures and Minesweeper is back online. Reviews will be slow as we are trying to recover from the large queue that has accumulated so please bear with us.
|-
| July 17, 10:34 PM
| Our test infrastructure is experiencing major issues and is causing most builds to fail and/or queue up. We are temporarily pausing the Minesweeper and are working to find and fix the problems.
|-
| June 25, 9:45 PM
| Recheck functionality is currently not available due to problems after upgrading to the latest Gerrit Trigger. Working on a fix and recheck should work again by tomorrow.
|-
| June 25, 9:40 PM
| After wrangling with a few infrastructure issues and another Tempest [https://bugs.launchpad.net/glance/+bug/1328738 bug], Minesweeper is back up and running.
|-
| June 24, 3:34 PM
| SqlAlchemy commit has broken tempest. Waiting on [https://review.openstack.org/#/c/102130/ Updated from global requirements] to re-enable Minesweeper.
|-
| Apr 28, 9:00 AM
| We lost access to review server after the gerrit upgrade. We are currently working on restoring access to the server.
|-
| Apr 24, 3:53 PM
| HV issues have been fixed and networking issues seen this morning have been resolved. Builds are back online.
|-
| Apr 23, 5:05 PM
| We have been seeing intermittent HV issues since yesterday. We are investigating.
|-
| Apr 18, 8:00 PM.
| All issues related to infra are fixed and minesweeper is up and running.
|-
| Mar 28, 01:30 AM
| Upgrade done but we are experiencing neutron timeout issues in the cloud. Our engineering team is looking into it and CI will be back online as soon as the issue is fixed.
|-
| Mar 27, 04:00 PM
| Upgrading our cloud to havana. CI will be back online as soon as we are done with the upgrade
|-
| Mar 24, 08:00 AM
| Minesweeper recheck is fixed.
|-
| Mar 20, 12:10 PM
| Minesweeper recheck is not working at the moment. We are working on fixing this.
|-
| Mar 18, 11:48 AM
| Infrastructure issues are preventing us from spawning testbeds. CI is down while we investigate.
|-
| Mar 14, 3:31 PM
| Network issues are preventing us from reaching git.openstack.org. CI is down currently as we investigate.
|-
| Mar 6, 12:29 AM
| CI was back online as of 9:38 AM.
|-
| Mar 6, 9:45 PM
| Quota issues affecting CI. Builds overnight may have failed. We are fixing the quota issue and CI should be online soon.
|-
| Mar 5, 1:12 PM
| Physical switch issues resolved. CI is back up and we are continuing to monitor the builds.
|-
| Mar 5, 12:02 PM
| Physical switch had failed. CI will be back up as soon as this is fixed.
|-
| Mar 5, 11:30 AM
| Services are down due to infrastructure issues. We are investigating.
|-
| Mar 5, 10:56 AM
| Services have been restored and CI is back online.
|-
| Mar 5, 10:54 AM
| Physical networking hardware has failed. CI is paused currently as we work ASAP to fix issues.
|-
| Mar 4, 4:26 PM
| We have resolved the networking issues. CI is back online.
|-
| Mar 4, 2:47 PM
| An infrastructure issue is causing networking problems so CI is paused currently. Investigating.
|-
| Feb 28, 3:48 PM
| CI is down for infra maintenance and upgrade. We expect the upgrade to last 4 hours.
|-
| Feb 2:17 PM
| We have rolled back our Devstack to an earlier commit while we fix the exit issue. Meanwhile, a new problem has surfaced - Devstack is seeing random failures in connecting to git.openstack.org. Investigating.
|-
| Feb 27, 11:54 AM
| A Devstack [https://github.com/openstack-dev/devstack/commit/0ed4af02da0bd4a0f757dd8c2156913e6c7a724c commit] is breaking test bed provisioning. We are working on a patch to resolve the issue.
|-
| Feb 26, 4:00 PM
| GPG issue has been resolved.
|-
| Feb 26, 2:06 PM
| GPG issue issue was not resolved. Root cause has been identified and builds should be back soon.
|-
| Feb 26, 1:17 PM
| A GPG key issue caused a few builds to fail in the last hour. Builds are back online.
|-
| Feb 26, 11:59 AM
| Cause was due to failures in RAID adapter. Builds should be back online now.
|-
| Feb 26, 11:46 AM
| We are investigating possible disk failures causing failed builds.
|-
| Feb 20, 2:46 PM
| A [https://github.com/openstack/tempest/commit/74b08ad1e1ff7b738b38c03fc125206ce438ec4e commit] that added a test for cinder backup caused some tests to fail. These tests are not supported and has been added to the exclude list.
|-
| Feb 18, 11:10 AM
| A [https://github.com/openstack/tempest/commit/805efcf6fb9bbf986ccb8c546c1ba59c2ddc12ee commit] that added 2 new tests in Tempest caused build failures yesterday. These tests are not supported by the VMware Cinder driver and have been added to the exclude list.
|-
| Feb 14, 10:38 AM
| Last night, Minesweeper ran out of floating IPs causing builds to fail. Back online but running with only 2 executors.
|-
| Feb 10, 8:48 PM
| [https://git.openstack.org/cgit/openstack/nova/commit/?id=9bb9a77e41b009cb940e98feda666dd8d806c862 Rescue bug] has been merged. Infrastructure issues are still being addressed. CI is currently offline.
|-
| Feb 10, 2:58 PM
| One infra issue resolved having to do with proxy failure. Minesweeper is still running on 1 executor while other infrastructure issues are being addressed.
|-
| Feb 10, 2:33 PM
| Rescue patch is in gate now, however, infra issues are still being worked on.
|-
| Feb 9, 10:47 AM
| Infra issues are being worked on. Minesweeper is also hitting a [https://bugs.launchpad.net/nova/+bug/1278149 Rescue bug] causing some tests to fail.
|-
| Feb 7, 2:46 PM
| Keystone requirement problem has been resolved as per [https://code.google.com/p/ipaddr-py/issues/detail?id=96#c7 this bug]. CI remains offline due to infrastructure issues.
|-
| Feb 7, 1:30 PM
| Infrastructure issues are still affecting builds. Builds have also started to become affected by keystone requirements [https://bugs.launchpad.net/devstack/+bug/1277507 bug]. Minesweeper will be taken offline until these issues are resolved.
|-
| Feb 6, 10:53 AM
| Hypervisor crash contributed to build failures, currently fixing. Quota database problem still affecting builds, looking to manually fix entries in the database.
|-
| Feb 5, 4:07 PM
| Identified [https://bugs.launchpad.net/nova/+bug/1250173 Concurrent requests to quotas reserve sometimes lock db table] as a possible cause for infra issues. Still investigating.
|-
| Feb 5, 1:47 AM
| Small percentage of builds failing due to infra issues. Investigating.
|-
| Feb 4, 5:00 PM
| Infrastructure issue have been resolved.
|-
| Feb 3, 8:29 AM
| Infrastructure issue involving testbed provision. Small percentage of builds may be failing.
|}

ThirdPartySystems/VMware CI

2014-09-16T21:50:16Z

Rhsu:

ThirdPartySystems/VMware CI

2014-09-16T21:50:03Z

Rhsu:

NovaVMware/Minesweeper

2014-09-11T22:05:02Z

Rhsu: /* Triggered Paths */

[[HypervisorSupportMatrix]] | [[NovaVMware]]

== Minesweeper CI ==
Minesweeper is continuous integration for Openstack running on VMware environments. In a nutshell, Minesweeper listens to newly created patchsets on review.openstack.org and runs the Tempest test suite on the patch. Currently Minesweeper is listening to Nova and Neutron projects.

=== Current Status ===
Please see [[NovaVMware/Minesweeper/Status|here]] for current status and a history of service disruptions.

=== Triggered Paths ===
Minesweeper does not trigger on every single patchset because it sometimes it doesn't need to. Additionally, it is very costly for the Minesweeper CI to be running on non-crucial patches since 3 VMs need to be spun up everytime! For example, it doesn't trigger on unit tests because unit tests do not affect the outcome of the Tempest runs. We also don't trigger on doc changes or 3rd party driver changes. We match on the following patterns (if there are multiple modified files, only one of the them needs to match):

<pre>
^nova\/(?!virt\/vmware).*$
^nova\/virt\/(vmwareapi\/.+)$
</pre>

=== Excluded Tests ===
A small handful of Tempest tests are excluded from run because there are known issues or a specific feature may not be implemented. Information about these tests can be found [[NovaVMware/Minesweeper/ExcludedTests|here]].

=== What to do when a build fails ===
Minesweeper votes a +1 for your patch when it passes. In the case of a failure, a +0 will be posted with a message that the build failed. When this happens, the best way to proceed is to follow these steps:

1. The build failure comment will be accompanied by a link to the build logs. Click to follow the link.

2. Open "testr_results.html.gz" and note the tests that failed. If the file is not present, then open "console.txt" and scroll to the bottom. If the message "Build was unsuccessful due to Infra-related issues" appears, go directly to step 4 (and optionally check the [[NovaVMware/Minesweeper/Status|status]] page to see if Minesweeper is operational). Otherwise continue to step 3.

3. Navigate back to the logs directory and try to identify the cause of the failures using the screen logs. The best way to do this is go through the relevant logs and look for strings such as ERROR, CRITICAL, or Traceback.

4. If you are confident that your patch did not contribute to the error, then the last step is to issue a recheck. A recheck will essentially run your patch through the CI again in case the failure was caused by a transient bug, race condition, or [http://lambda-diode.com/opinion/ecc-memory cosmic rays]. To issue a recheck, submit a comment with the following text: <code>vmware-recheck-patch</code>

NovaVMware/Minesweeper

2014-09-11T21:54:35Z

Rhsu:

[[HypervisorSupportMatrix]] | [[NovaVMware]]

== Minesweeper CI ==
Minesweeper is continuous integration for Openstack running on VMware environments. In a nutshell, Minesweeper listens to newly created patchsets on review.openstack.org and runs the Tempest test suite on the patch. Currently Minesweeper is listening to Nova and Neutron projects.

=== Current Status ===
Please see [[NovaVMware/Minesweeper/Status|here]] for current status and a history of service disruptions.

=== Triggered Paths ===
Minesweeper does not trigger on every single patchset because sometimes it doesn't need to. For example, we don't trigger on unit tests because they do not affect the outcome of the Tempest runs. We also don't trigger on doc changes or 3rd party driver changes. We match on the following patterns (if there are multiple modified files, only one of the them needs to match):

<pre>
^nova\/(?!virt\/vmware).*$
^nova\/virt\/(vmwareapi\/.+)$
</pre>

=== Excluded Tests ===
A small handful of Tempest tests are excluded from run because there are known issues or a specific feature may not be implemented. Information about these tests can be found [[NovaVMware/Minesweeper/ExcludedTests|here]].

=== What to do when a build fails ===
Minesweeper votes a +1 for your patch when it passes. In the case of a failure, a +0 will be posted with a message that the build failed. When this happens, the best way to proceed is to follow these steps:

1. The build failure comment will be accompanied by a link to the build logs. Click to follow the link.

2. Open "testr_results.html.gz" and note the tests that failed. If the file is not present, then open "console.txt" and scroll to the bottom. If the message "Build was unsuccessful due to Infra-related issues" appears, go directly to step 4 (and optionally check the [[NovaVMware/Minesweeper/Status|status]] page to see if Minesweeper is operational). Otherwise continue to step 3.

3. Navigate back to the logs directory and try to identify the cause of the failures using the screen logs. The best way to do this is go through the relevant logs and look for strings such as ERROR, CRITICAL, or Traceback.

4. If you are confident that your patch did not contribute to the error, then the last step is to issue a recheck. A recheck will essentially run your patch through the CI again in case the failure was caused by a transient bug, race condition, or [http://lambda-diode.com/opinion/ecc-memory cosmic rays]. To issue a recheck, submit a comment with the following text: <code>vmware-recheck-patch</code>

NovaVMware/Minesweeper

2014-09-10T21:32:21Z

Rhsu: /* What to do when a build fails */

[[HypervisorSupportMatrix]] | [[NovaVMware]]

== Minesweeper CI ==
Minesweeper is continuous integration for Openstack running on VMware environments. In a nutshell, Minesweeper listens to newly created patchsets on review.openstack.org and runs the Tempest test suite on the patch. Currently Minesweeper is listening to Nova and Neutron projects.

=== Current Status ===
Please see [[NovaVMware/Minesweeper/Status|here]] for current status and a history of service disruptions.

=== Excluded Tests ===
A small handful of Tempest tests are excluded from run because there are known issues or a specific feature may not be implemented. Information about these tests can be found [[NovaVMware/Minesweeper/ExcludedTests|here]].

=== What to do when a build fails ===
Minesweeper votes a +1 for your patch when it passes. In the case of a failure, a +0 will be posted with a message that the build failed. When this happens, the best way to proceed is to follow these steps:

1. The build failure comment will be accompanied by a link to the build logs. Click to follow the link.

2. Open "testr_results.html.gz" and note the tests that failed. If the file is not present, then open "console.txt" and scroll to the bottom. If the message "Build was unsuccessful due to Infra-related issues" appears, go directly to step 4 (and optionally check the [[NovaVMware/Minesweeper/Status|status]] page to see if Minesweeper is operational). Otherwise continue to step 3.

3. Navigate back to the logs directory and try to identify the cause of the failures using the screen logs. The best way to do this is go through the relevant logs and look for strings such as ERROR, CRITICAL, or Traceback.

4. If you are confident that your patch did not contribute to the error, then the last step is to issue a recheck. A recheck will essentially run your patch through the CI again in case the failure was caused by a transient bug, race condition, or [http://lambda-diode.com/opinion/ecc-memory cosmic rays]. To issue a recheck, submit a comment with the following text: <code>vmware-recheck-patch</code>

2014-03-18T18:49:04Z

2014-03-05T00:27:02Z

Rhsu: