Jump to: navigation, search

NovaVMware/Minesweeper/Status

< NovaVMware‎ | Minesweeper
Revision as of 17:46, 27 March 2014 by Sreeram (talk | contribs) (Service Disruption History)

HypervisorSupportMatrix | NovaVMware

Current Service Status

Sweeper-ok.png No known issues at this time Sweeper-notok.png CI is offline or running with limited capacity

Blocking Bugs

These bugs may be causing Minesweeper to fail in the form of failing tests. If there are fixes available, they need to be merged soon for Minesweeper to run correctly.

Severity Bug Fix (needs review attention)
Low VMware: attaching a volume to the VM failed VMware: Support hot-adding for VMs with IDE controllers

Service Disruption History

All times are in PST.

Event Time Description
Mar 28, 01:30 AM Upgrade done but we are experiencing neutron timeout issues in the cloud. Our engineering team is looking into it and CI will be back online as soon as the issue is fixed.
Mar 26, 04:00 PM Upgrading our cloud to havana. CI will be back online as soon as we are done with the upgrade
Mar 24, 08:00 AM Minesweeper recheck is fixed.
Mar 20, 12:10 PM Minesweeper recheck is not working at the moment. We are working on fixing this.
Mar 18, 11:48 AM Infrastructure issues are preventing us from spawning testbeds. CI is down while we investigate.
Mar 14, 3:31 PM Network issues are preventing us from reaching git.openstack.org. CI is down currently as we investigate.
Mar 6, 12:29 AM CI was back online as of 9:38 AM.
Mar 6, 9:45 PM Quota issues affecting CI. Builds overnight may have failed. We are fixing the quota issue and CI should be online soon.
Mar 5, 1:12 PM Physical switch issues resolved. CI is back up and we are continuing to monitor the builds.
Mar 5, 12:02 PM Physical switch had failed. CI will be back up as soon as this is fixed.
Mar 5, 11:30 AM Services are down due to infrastructure issues. We are investigating.
Mar 5, 10:56 AM Services have been restored and CI is back online.
Mar 5, 10:54 AM Physical networking hardware has failed. CI is paused currently as we work ASAP to fix issues.
Mar 4, 4:26 PM We have resolved the networking issues. CI is back online.
Mar 4, 2:47 PM An infrastructure issue is causing networking problems so CI is paused currently. Investigating.
Feb 28, 3:48 PM CI is down for infra maintenance and upgrade. We expect the upgrade to last 4 hours.
Feb 2:17 PM We have rolled back our Devstack to an earlier commit while we fix the exit issue. Meanwhile, a new problem has surfaced - Devstack is seeing random failures in connecting to git.openstack.org. Investigating.
Feb 27, 11:54 AM A Devstack commit is breaking test bed provisioning. We are working on a patch to resolve the issue.
Feb 26, 4:00 PM GPG issue has been resolved.
Feb 26, 2:06 PM GPG issue issue was not resolved. Root cause has been identified and builds should be back soon.
Feb 26, 1:17 PM A GPG key issue caused a few builds to fail in the last hour. Builds are back online.
Feb 26, 11:59 AM Cause was due to failures in RAID adapter. Builds should be back online now.
Feb 26, 11:46 AM We are investigating possible disk failures causing failed builds.
Feb 20, 2:46 PM A commit that added a test for cinder backup caused some tests to fail. These tests are not supported and has been added to the exclude list.
Feb 18, 11:10 AM A commit that added 2 new tests in Tempest caused build failures yesterday. These tests are not supported by the VMware Cinder driver and have been added to the exclude list.
Feb 14, 10:38 AM Last night, Minesweeper ran out of floating IPs causing builds to fail. Back online but running with only 2 executors.
Feb 10, 8:48 PM Rescue bug has been merged. Infrastructure issues are still being addressed. CI is currently offline.
Feb 10, 2:58 PM One infra issue resolved having to do with proxy failure. Minesweeper is still running on 1 executor while other infrastructure issues are being addressed.
Feb 10, 2:33 PM Rescue patch is in gate now, however, infra issues are still being worked on.
Feb 9, 10:47 AM Infra issues are being worked on. Minesweeper is also hitting a Rescue bug causing some tests to fail.
Feb 7, 2:46 PM Keystone requirement problem has been resolved as per this bug. CI remains offline due to infrastructure issues.
Feb 7, 1:30 PM Infrastructure issues are still affecting builds. Builds have also started to become affected by keystone requirements bug. Minesweeper will be taken offline until these issues are resolved.
Feb 6, 10:53 AM Hypervisor crash contributed to build failures, currently fixing. Quota database problem still affecting builds, looking to manually fix entries in the database.
Feb 5, 4:07 PM Identified Concurrent requests to quotas reserve sometimes lock db table as a possible cause for infra issues. Still investigating.
Feb 5, 1:47 AM Small percentage of builds failing due to infra issues. Investigating.
Feb 4, 5:00 PM Infrastructure issue have been resolved.
Feb 3, 8:29 AM Infrastructure issue involving testbed provision. Small percentage of builds may be failing.