Jump to: navigation, search

Difference between revisions of "PowerKVM"

(Created page with "This is the page of the team working with PowerKVM and OpenStack = Actual projects being tested = == Production == Production means we are testing and reporting back to com...")
 
(→‎Contact)
 
(168 intermediate revisions by 7 users not shown)
Line 1: Line 1:
−
This is the page of the team working with PowerKVM and OpenStack
+
= Overview =
 +
This is the IBM KVM on Power CI status page.
  
−
= Actual projects being tested =
+
The KVM on Power CI performs tests using Ubuntu Xenial single use VMs. The VMs are spun up on OpenStack cloud, currently Mitaka release. CirrOS image with kernel >= 4.8.0-22 had to be built to support second level KVM acceleration on ppc64le (see [https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1634129 bug]).
  
−
== Production ==
+
Logs are uploaded to IBM Softlayer's Swift [http://dal05.objectstorage.softlayer.net/v1/AUTH_3d8e6ecb-f597-448c-8ec2-164e9f710dd6/pkvmci/index.html container] and expire in 1 month. The test jobs are using [http://github.com/openstack-infra/devstack-gate devstack-gate] to setup devstack and run the Tempest test suite.
−
Production means we are testing and reporting back to community. Right now we have the following projects being tested:
 
−
* Nova
 
  
−
== Development ==
+
Checks for Ironic project utilize bare metal hardware (ppc64le architecture).
−
Development means we are testing internally only, after the project became mature will be added in production and will report back to community
 
−
* Tempest
 
−
* devstack
 
  
−
= Debugging PowerKVM Failures =
+
== Contact ==
 +
IRC: mjturek on OpenStack Freenode channels
 +
 
 +
= Current Status =
 +
* nova, cinder, devstack, os-brick,  diskimage-builder reporting '''ENABLED''', voting only on success due to instability
 +
* ironic reporting '''ENABLED'''
 +
 
 +
{| class="wikitable"
 +
|+ Outage history
 +
|-
 +
! Start !! End !! Description
 +
|-
 +
| 2018-11-06 20:00 UTC  || 2018-11-07 16:00 UTC || Cloud provider down
 +
|-
 +
| 2018-09-28 18:00 UTC  || 2018-10-01 15:00 UTC || Scheduled lab maintenance
 +
|-
 +
| 2018-04-04 03:00 UTC  || 2018-04-04 18:00 UTC || Cloud provider down
 +
|-
 +
| 2018-03-21 22:00 UTC  || 2018-03-22 17:00 UTC || Network down after an upgrade
 +
|-
 +
| 2018-02-28 14:00 UTC  || 2018-02-28 21:00 UTC || Network outage
 +
|-
 +
| 2017-06-21 19:02 UTC  || 2017-06-21 19:58 UTC || Network
 +
|-
 +
| 2017-06-17 06:00 UTC  || 2017-06-19 12:00 UTC || Zuul failed
 +
|-
 +
| 2017-05-23 12:00 UTC  || 2017-05-24 20:00 UTC || Too many tempest failures due to breaking changes in glance, os-brick, nova
 +
|-
 +
| 2017-04-15 04:00 UTC  || 2017-04-17 16:00 UTC || Zuul failure due to dependency update (voluptuous)
 +
|-
 +
| 2017-04-11 11:30 UTC  || 2017-04-11 14:00 UTC || Network
 +
|-
 +
| 2017-04-07 19:30 UTC  || 2017-04-08 03:00 UTC || Network
 +
|-
 +
| 2017-03-16 15:00 UTC  || 2017-03-16 18:00 UTC || Zuul fail
 +
|-
 +
| 2017-03-07 15:00 UTC  || 2017-03-07 17:00 UTC || apt cache server is down
 +
|-
 +
| 2017-02-13 18:50 UTC  || 2017-02-14 01:00 UTC || Cloud down after a power outage
 +
|-
 +
| 2017-02-07 16:30 UTC  || 2017-02-07 18:00 UTC || Network outage
 +
|-
 +
| 2016-12-08 12:00 UTC  || 2016-12-08 17:00 UTC || pypi mirror is down
 +
|-
 +
| 2016-11-15 14:00 UTC  || 2016-11-17 15:00 UTC || Packet loss on route to git.o.o
 +
|-
 +
| 2016-11-02 13:00 UTC  || 2016-11-02 18:00 UTC || Network outage for maintenance
 +
|-
 +
| 2016-10-04 09:00 UTC  || 2016-10-05 13:00 UTC || devstack-vivid failure due pyeclib requirement updated to 1.3.1
 +
|-
 +
| 2016-07-13 20:00 UTC  || 2016-07-14 18:00 UTC || Internal pypi mirror is down
 +
|-
 +
| 2016-06-26 05:00 UTC  || 2016-06-27 22:00 UTC || Timeout while installing python XStatic-Angular-Bootstrap
 +
|-
 +
| 2016-06-24 02:20 UTC  || 2016-06-24 16:00 UTC || Pypi mirror server down, and pypi.python.org is too weak to handle the load
 +
|-
 +
| 2016-06-21 21:00 UTC  || 2016-06-23 02:00 UTC || Underlying cloud is down: no patches were missed, long delay in posting results
 +
|-
 +
| 2016-05-25 00:00 UTC  || 2016-05-26 16:00 UTC || devstack setup failure due to our Ubuntu images using upstart, infrastracture problems
 +
|-
 +
| 2016-05-23 08:00 UTC  || 2016-05-23 11:00 UTC || Zuul failed, zuul-merger hanging
 +
|-
 +
| 2016-05-12 12:30 UTC  || 2016-05-12 14:30 UTC || Zuul failed
 +
|-
 +
| 2016-05-08 08:00 UTC  || 2016-05-09 11:00 UTC || Zuul lost connection to gerrit.o.o
 +
|-
 +
| 2016-04-29 05:30 UTC  || 2016-04-29 16:30 UTC || Jobs failing due to missing dev dependency
 +
|-
 +
| 2016-04-19 12:30 UTC  || 2016-04-19 19:00 UTC || Network outage
 +
|-
 +
| 2016-04-10 11:00 UTC  || 2016-04-11 19:00 UTC ||
 +
|-
 +
| 2016-03-15 17:30 UTC  || 2016-03-16 12:05 UTC || Zuul down
 +
|-
 +
| 2016-03-03 14:00 UTC  || 2016-03-03 18:00 UTC || Zuul fails to listen gerrit.o.o events
 +
|-
 +
| 2016-02-13 13:00 UTC  || 2016-02-15 14:00 UTC || Zuul reconnect failure after gerrit.o.o restart
 +
|-
 +
| 2016-02-12 10:00 UTC  || 2016-02-12 19:00 UTC || Network loss
 +
|-
 +
| 2016-01-26 19:00 UTC  || 2016-01-27 15:00 UTC || Network loss
 +
|-
 +
| 2016-01-19 10:00 UTC  || 2016-01-19 18:00 UTC || Loss of Jenkins master
 +
|-
 +
| 2015-12-21 12:15 UTC  || 2015-12-22 12:45 UTC || Jobs failing due to launchpad bug #1447704
 +
|-
 +
| 2015-12-04 20:00 UTC  || 2015-12-07 18:00 UTC || Jobs are unstable
 +
|-
 +
| 2015-11-30 04:00 UTC  || 2015-11-30 13:00 UTC || nodepool failing
 +
|-
 +
| 2015-10-19 09:00 UTC  || 2015-10-19 17:00 UTC || reporting failures due to outdated pypi mirror
 +
|-
 +
| 2015-08-24 00:00 UTC  || 2015-08-24 13:00 UTC || bad zuul configuration
 +
|-
 +
| 2015-08-08 06:00 UTC  || 2015-08-10 12:50 UTC || zuul failed to update configuration
 +
|-
 +
| 2015-07-30 13:00 UTC  || 2015-07-31 21:00 UTC || Numerous failures due to upper_constraints, disabled constraints for the gate
 +
|-
 +
| 2015-07-27 12:00 UTC  || 2015-07-27 16:00 UTC || zuul/jenkins failure
 +
|-
 +
| 2015-07-03 13:00 UTC  || 2015-07-03 19:00 UTC ||  Gate was failing due to devstack-gate patch that enabled pypi package upper constraints
 +
|-
 +
| 2015-07-01 00:00 UTC  || 2015-07-01 14:00 UTC || Jenkins crashed (leap second suspected as the cause)
 +
|-
 +
| 2015-06-30 01:00 UTC  || 2015-06-30 13:00 UTC || Zuul did not reconnect to gerrit after connectivity loss
 +
|-
 +
| 2015-06-16 16:00 UTC  || 2015-06-16 19:00 UTC || Could not install python-libguestfs on Fedora 21 guests, yum cache problems
 +
|-
 +
| 2015-05-16 17:30 UTC  || 2015-05-18 02:00 UTC || Zuul service down
 +
|-
 +
| 2015-05-04 16:00 UTC  || 2015-05-04 17:00 UTC || Maintenance
 +
|-
 +
| 2015-04-25 19:00 UTC  || 2015-04-26 16:00 UTC || Zuul VM down
 +
|-
 +
| 2015-04-11 08:00 UTC  || 2015-04-11 17:00 UTC || Internal network down
 +
|-
 +
| 2015-04-10 14:00 UTC  || 2015-04-10 19:00 UTC || Internal network down
 +
|-
 +
|}
 +
 
 +
= Projects Tested =
 +
Current test status for all tested projects can be seen at
 +
[http://dal05.objectstorage.softlayer.net/v1/AUTH_3d8e6ecb-f597-448c-8ec2-164e9f710dd6/pkvmci/index.html Zuul Testing Report].
 +
 
 +
The report is compiled form zuul logs and is updated periodically. It might include aborted or otherwise terminated
 +
builds.
 +
 
 +
== Reporting ==
 +
 
 +
* Devstack
 +
* Ironic
 +
* Nova
 +
* Cinder
 +
* os-brick
 +
* diskimage-builder
 +
 
 +
 
 +
== Non-reporting ==
 +
 
 +
* Devstack-gate
 +
* Neutron
 +
* Glance
 +
* Keystone
 +
* Nova Unit tests
 +
* Requirements
 +
* Swift
 +
* Tempest
 +
 
 +
= Skipped Tests =
 +
 
 +
There are currently a few skipped tests, see test run logs for details.
 +
 
 +
== Known Issues ==
 +
* Missing PCI hotplug support in CirrOS ppc64le
 +
  * Neutron (3) failures to missing interface attach support (PCI hotplug) in CirrOS
 +
 
 +
== PowerKVM CI related blogs ==
 +
*  [https://developer.ibm.com/opentech/2016/04/21/kvm-on-power-ci-increasing-the-test-coverage/ KVM on Power CI: Increasing the test coverage]
 +
*  [https://developer.ibm.com/opentech/2016/04/20/ironic-continuous-integration-testing-with-hardware-pool-management/ Ironic CI with physical Power target nodes]
 +
*  [https://www.ibm.com/developerworks/community/blogs/fe313521-2e95-46f2-817d-44a4f27eba32/entry/building_a_tiny_ppc_cloud_image_for_ci_testing?lang=en CirrOS]
 +
*  [https://www.ibm.com/developerworks/community/blogs/fe313521-2e95-46f2-817d-44a4f27eba32/entry/using_ibmveth_and_ibmvscsi_on_openstack?lang=en Using ibmveth and ibmvscsi]
 +
*  [https://www.ibm.com/developerworks/community/blogs/fe313521-2e95-46f2-817d-44a4f27eba32/entry/spawning_a_little_endian_instance_in_openstack?lang=en LE guest]
 +
*  [https://www.ibm.com/developerworks/community/blogs/fe313521-2e95-46f2-817d-44a4f27eba32/entry/powerkvm_openstack_ci_a_custom_swift_upload_script?lang=en Swift script]
 +
*  [https://www.ibm.com/developerworks/community/blogs/fe313521-2e95-46f2-817d-44a4f27eba32/entry/building_your_openstack_3rd_party_ci_system1?lang=en Continuous Integration]
 +
*  [https://www.ibm.com/developerworks/community/blogs/fe313521-2e95-46f2-817d-44a4f27eba32/entry/openstack_nova_scheduler_disable_ram_weigher?lang=en RAM Weigher]
 +
*  [https://www.ibm.com/developerworks/community/blogs/fe313521-2e95-46f2-817d-44a4f27eba32/entry/setup_ibm_storwize_for_openstack_cinder_and_glance?lang=en Storwize]
 +
*  [http://www.kurttaylor.com/blog/2014/11/third-party-ci-work-group-at-kilo-summit/ Third Party CI Working Group]
 +
*  [http://www.kurttaylor.com/blog/2015/03/clean-swift-object-store/ Cleaning up Swift]
 +
*  [http://www.kurttaylor.com/blog/2015/05/ci-tools-repo/ Third Party CI Working Group Tools Repository]

Latest revision as of 02:59, 17 February 2019

Overview

This is the IBM KVM on Power CI status page.

The KVM on Power CI performs tests using Ubuntu Xenial single use VMs. The VMs are spun up on OpenStack cloud, currently Mitaka release. CirrOS image with kernel >= 4.8.0-22 had to be built to support second level KVM acceleration on ppc64le (see bug).

Logs are uploaded to IBM Softlayer's Swift container and expire in 1 month. The test jobs are using devstack-gate to setup devstack and run the Tempest test suite.

Checks for Ironic project utilize bare metal hardware (ppc64le architecture).

Contact

IRC: mjturek on OpenStack Freenode channels

Current Status

  • nova, cinder, devstack, os-brick, diskimage-builder reporting ENABLED, voting only on success due to instability
  • ironic reporting ENABLED
Outage history
Start End Description
2018-11-06 20:00 UTC 2018-11-07 16:00 UTC Cloud provider down
2018-09-28 18:00 UTC 2018-10-01 15:00 UTC Scheduled lab maintenance
2018-04-04 03:00 UTC 2018-04-04 18:00 UTC Cloud provider down
2018-03-21 22:00 UTC 2018-03-22 17:00 UTC Network down after an upgrade
2018-02-28 14:00 UTC 2018-02-28 21:00 UTC Network outage
2017-06-21 19:02 UTC 2017-06-21 19:58 UTC Network
2017-06-17 06:00 UTC 2017-06-19 12:00 UTC Zuul failed
2017-05-23 12:00 UTC 2017-05-24 20:00 UTC Too many tempest failures due to breaking changes in glance, os-brick, nova
2017-04-15 04:00 UTC 2017-04-17 16:00 UTC Zuul failure due to dependency update (voluptuous)
2017-04-11 11:30 UTC 2017-04-11 14:00 UTC Network
2017-04-07 19:30 UTC 2017-04-08 03:00 UTC Network
2017-03-16 15:00 UTC 2017-03-16 18:00 UTC Zuul fail
2017-03-07 15:00 UTC 2017-03-07 17:00 UTC apt cache server is down
2017-02-13 18:50 UTC 2017-02-14 01:00 UTC Cloud down after a power outage
2017-02-07 16:30 UTC 2017-02-07 18:00 UTC Network outage
2016-12-08 12:00 UTC 2016-12-08 17:00 UTC pypi mirror is down
2016-11-15 14:00 UTC 2016-11-17 15:00 UTC Packet loss on route to git.o.o
2016-11-02 13:00 UTC 2016-11-02 18:00 UTC Network outage for maintenance
2016-10-04 09:00 UTC 2016-10-05 13:00 UTC devstack-vivid failure due pyeclib requirement updated to 1.3.1
2016-07-13 20:00 UTC 2016-07-14 18:00 UTC Internal pypi mirror is down
2016-06-26 05:00 UTC 2016-06-27 22:00 UTC Timeout while installing python XStatic-Angular-Bootstrap
2016-06-24 02:20 UTC 2016-06-24 16:00 UTC Pypi mirror server down, and pypi.python.org is too weak to handle the load
2016-06-21 21:00 UTC 2016-06-23 02:00 UTC Underlying cloud is down: no patches were missed, long delay in posting results
2016-05-25 00:00 UTC 2016-05-26 16:00 UTC devstack setup failure due to our Ubuntu images using upstart, infrastracture problems
2016-05-23 08:00 UTC 2016-05-23 11:00 UTC Zuul failed, zuul-merger hanging
2016-05-12 12:30 UTC 2016-05-12 14:30 UTC Zuul failed
2016-05-08 08:00 UTC 2016-05-09 11:00 UTC Zuul lost connection to gerrit.o.o
2016-04-29 05:30 UTC 2016-04-29 16:30 UTC Jobs failing due to missing dev dependency
2016-04-19 12:30 UTC 2016-04-19 19:00 UTC Network outage
2016-04-10 11:00 UTC 2016-04-11 19:00 UTC
2016-03-15 17:30 UTC 2016-03-16 12:05 UTC Zuul down
2016-03-03 14:00 UTC 2016-03-03 18:00 UTC Zuul fails to listen gerrit.o.o events
2016-02-13 13:00 UTC 2016-02-15 14:00 UTC Zuul reconnect failure after gerrit.o.o restart
2016-02-12 10:00 UTC 2016-02-12 19:00 UTC Network loss
2016-01-26 19:00 UTC 2016-01-27 15:00 UTC Network loss
2016-01-19 10:00 UTC 2016-01-19 18:00 UTC Loss of Jenkins master
2015-12-21 12:15 UTC 2015-12-22 12:45 UTC Jobs failing due to launchpad bug #1447704
2015-12-04 20:00 UTC 2015-12-07 18:00 UTC Jobs are unstable
2015-11-30 04:00 UTC 2015-11-30 13:00 UTC nodepool failing
2015-10-19 09:00 UTC 2015-10-19 17:00 UTC reporting failures due to outdated pypi mirror
2015-08-24 00:00 UTC 2015-08-24 13:00 UTC bad zuul configuration
2015-08-08 06:00 UTC 2015-08-10 12:50 UTC zuul failed to update configuration
2015-07-30 13:00 UTC 2015-07-31 21:00 UTC Numerous failures due to upper_constraints, disabled constraints for the gate
2015-07-27 12:00 UTC 2015-07-27 16:00 UTC zuul/jenkins failure
2015-07-03 13:00 UTC 2015-07-03 19:00 UTC Gate was failing due to devstack-gate patch that enabled pypi package upper constraints
2015-07-01 00:00 UTC 2015-07-01 14:00 UTC Jenkins crashed (leap second suspected as the cause)
2015-06-30 01:00 UTC 2015-06-30 13:00 UTC Zuul did not reconnect to gerrit after connectivity loss
2015-06-16 16:00 UTC 2015-06-16 19:00 UTC Could not install python-libguestfs on Fedora 21 guests, yum cache problems
2015-05-16 17:30 UTC 2015-05-18 02:00 UTC Zuul service down
2015-05-04 16:00 UTC 2015-05-04 17:00 UTC Maintenance
2015-04-25 19:00 UTC 2015-04-26 16:00 UTC Zuul VM down
2015-04-11 08:00 UTC 2015-04-11 17:00 UTC Internal network down
2015-04-10 14:00 UTC 2015-04-10 19:00 UTC Internal network down

Projects Tested

Current test status for all tested projects can be seen at Zuul Testing Report.

The report is compiled form zuul logs and is updated periodically. It might include aborted or otherwise terminated builds.

Reporting

  • Devstack
  • Ironic
  • Nova
  • Cinder
  • os-brick
  • diskimage-builder


Non-reporting

  • Devstack-gate
  • Neutron
  • Glance
  • Keystone
  • Nova Unit tests
  • Requirements
  • Swift
  • Tempest

Skipped Tests

There are currently a few skipped tests, see test run logs for details.

Known Issues

  • Missing PCI hotplug support in CirrOS ppc64le
 * Neutron (3) failures to missing interface attach support (PCI hotplug) in CirrOS

PowerKVM CI related blogs

Retrieved from "https://wiki.openstack.org/w/index.php?title=PowerKVM&oldid=168409"