Revision as of 22:08, 15 September 2025

Weekly Project Infrastructure team meeting

The OpenDev Team holds public weekly meetings in #opendev-meeting on OFTC, Tuesdays at 1900 UTC. Everyone interested in infrastructure and process surrounding automated testing and deployment is encouraged to attend.

Please feel free to add agenda items (and your IRC nick in parenthesis).

Agenda for next meeting

• Announcements

• Actions from last meeting

• Specs Review

• Topics
  • Moving OpenDev's python-base/python-builder/uwsgi-base Images to Quay (clarkb 20250812)
    • The images have moved and we're no longer publishing to docker hub
    • https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/thread/HO6Z66QIMDIDY7CCVAREDOPSYZYNKIT3/
    • I believe that all images within OpenDev that consume these base images have been updated to pull from the new location (including Gerrit)
    • If you find images that need updates please push a change and let us know.
  • Gerrit 3.11 Upgrade Planning (clarkb 20250401)
    • https://www.gerritcodereview.com/3.11.html
    • Please check this for any concerns with the way we use Gerrit.
    • Working on getting two new held nodes after images were rebuilt for 3.10.8.
    • There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us.
      • Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems
    • Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup.
    • Gerrit shutdown remain plagued by large h2 cache files. Seems removing our long timeout was not enough to prevent shutdowns timing out.
    • https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade
    • Will need to select a date once we know more about the upgrade itself.
  • Upgrading Old Servers (clarkb 20230627)
    • https://etherpad.opendev.org/p/opendev-bionic-server-upgrades
    • https://etherpad.opendev.org/p/opendev-focal-server-upgrades
    • https://etherpad.opendev.org/p/opendev-server-replacement-sprint
      • wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade
      • tonyb looking at cacti after wiki
    • The entire openafs and kerberos system has been upgraded to Noble except for afs02.dfw.openstack.org. Waiting on RW volume moves to complete before performing this last upgrade
      • Will need to remember to remove all of the hosts from the emergency file when done.
    • Next on the list are graphite and backup servers
    • Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic
  • AFS mirror content cleanup (clarkb 20250916)
    • Fungi identified a number of AFS mirror cleanups that can be performed once server upgrades are complete
    • OpenEuler mirror content is no longer needed as we don't have any test images
    • Debian stretch and bullseye backports are no longer needed as they are both old enough to have aged out
    • For the reprepro content we should also look into automatically clearing content if the cost to do so is actually a noop when we haven't removed releases from the mirror list.
  • Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520)
    • The spec (954826) has merged.
    • clarkb will start working on this as soon as time permits.
  • Pre PTG Planning (clarkb 20250624)
    • https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document
    • Clarkb has added a number of topic Ideas. Please add more.
    • Times: Tuesday October 7 1800-2000 UTC, Wednesday October 8 1500-1700 UTC, Thursday October 9 1500-1700
    • The Tuesday time block would effectively replace our weekly IRC meeting.
  • Etherpad 2.5.0 Upgrade (clarkb 20250805)
    • https://github.com/ether/etherpad-lite/blob/v2.5.0/CHANGELOG.md
    • This release is reported to have fixed our css problems.
    • https://review.opendev.org/c/opendev/system-config/+/956593/
    • 104.130.127.119 is a held node for testing. You need to edit /etc/hosts to point etherpad.opendev.org at that IP.
  • Lists Server Slowness (clarkb 20250909)
    • We think we have tracked this down to the different disk_io_index values between Performance and Standard flavors on rax classic
    • Specifically the value is 2 on Standard (lists) and 40 on mirror and fio measures about a 20x differences in iops during randread testing to match
    • We will need to migrate the mailman3 installation off of the server disk and onto a cinder volume to improve the number of iops.

• Open discussion

Upcoming Project Renames

(any additions should mention original->new full names and link to the corresponding project-config rename change in Gerrit) Changes should have their topic set to project-rename.

• Rename example/foo -> example/bar: https://review.opendev.org/c/openstack/project-config/+/123456

Previous meetings

Previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/infra/ and earlier at http://eavesdrop.openstack.org/meetings/ci/