Revision as of 16:48, 11 November 2025

Weekly Project Infrastructure team meeting

The OpenDev Team holds public weekly meetings in #opendev-meeting on OFTC, Tuesdays at 1900 UTC. Everyone interested in infrastructure and process surrounding automated testing and deployment is encouraged to attend.

Please feel free to add agenda items (and your IRC nick in parenthesis).

Agenda for next meeting

• Announcements

• Actions from last meeting

• Specs Review

• Topics
  • Gerrit 3.11 Upgrade Planning (clarkb 20250401)
    • Gerrit 3.13 has been released so we should really try to upgrade to 3.11 and then 3.12 soon.
    • https://www.gerritcodereview.com/3.11.html
    • Please check this for any concerns with the way we use Gerrit.
    • https://review.opendev.org/c/opendev/system-config/+/966083 Fix container bind mounts for Gerrit
    • https://review.opendev.org/c/opendev/system-config/+/966084 Update to Gerrit 3.10.9 and 3.11.7
    • There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us.
      • Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems
    • Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup.
    • Gerrit shutdown remain plagued by large h2 cache files. Seems removing our long timeout was not enough to prevent shutdowns timing out.
    • https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade
    • Will need to select a date once we know more about the upgrade itself.
  • Upgrading Old Servers (clarkb 20230627)
    • https://etherpad.opendev.org/p/opendev-bionic-server-upgrades
    • https://etherpad.opendev.org/p/opendev-focal-server-upgrades
    • https://etherpad.opendev.org/p/opendev-server-replacement-sprint
      • wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade
      • tonyb looking at cacti after wiki
    • Next on the list are graphite and backup servers
    • Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups.
    • Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic
  • Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520)
    • The spec (954826) has merged.
    • clarkb will start working on this as soon as time permits.
  • Upgrade Zuul Zookeeper Cluster to 3.9 (clarkb 20251111)
    • https://review.opendev.org/c/opendev/system-config/+/966612
    • This will add some new features Zuul can take advantage of
  • Gitea 1.25.1 Upgrade (clarkb 20251028)
    • https://review.opendev.org/c/opendev/system-config/+/965960 Upgrade Gitea to 1.25.1
  • Gitea Performance (clarkb 20251028)
    • All traffic should be through the load balancer now
    • Memcached size limits have been increased
    • We should monitor and determine if these changes have improved performance
  • Raxflex DFW3 Disabled (clarkb 20251104)
    • We discovered the raxflex dfw3 mirror had afs cache errors. fs flushall appeared to get stuck and the kernel oopsed. After forcing a shutdown via openstack the server refuses to boot
    • We've asked dan_with what our next steps should be via IRC
    • Worst case we rebuild the mirror with a new cinder volume and replace the whole thing.

• Open discussion

Upcoming Project Renames

(any additions should mention original->new full names and link to the corresponding project-config rename change in Gerrit) Changes should have their topic set to project-rename.

• Rename example/foo -> example/bar: https://review.opendev.org/c/openstack/project-config/+/123456

Previous meetings

Previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/infra/ and earlier at http://eavesdrop.openstack.org/meetings/ci/