Revision as of 04:45, 3 June 2025

Weekly Project Infrastructure team meeting

The OpenDev Team holds public weekly meetings in #opendev-meeting on OFTC, Tuesdays at 1900 UTC. Everyone interested in infrastructure and process surrounding automated testing and deployment is encouraged to attend.

Please feel free to add agenda items (and your IRC nick in parenthesis).

Agenda for next meeting

• Announcements

• Actions from last meeting

• Specs Review

• Topics
  • Zuul-launcher image builds (corvus 20240910)
  • Gerrit shutdown problems (clarkb 20250527)
    • After successfully using SIGINT to shutdown gerrit a few weeks ago the recent restart last week failed to shutdown and timed out to SIGKILL
    • https://review.opendev.org/c/opendev/system-config/+/950595 One theory is that h2 compaction time may be slowing down shutdown enough to time out
    • Another theory is that SIGINT just doesn't work as expected and we need to use SIGHUP (which podman blocks by default)
    • Proposal: When convenient time for a restart we land 950595. Since that won't apply until the next restart we manually issue a kill -HUP against the gerrit process to see if that works better. Then once h2 compaction cleanup is in place the restart after can try SIGINT again.
  • Gerrit 3.11 Upgrade Planning (clarkb 20250401)
    • https://www.gerritcodereview.com/3.11.html
    • Please check this for any concerns with the way we use Gerrit.
    • Automated testing is currently testing this generally works for us.
    • 104.130.253.194 is a held Gerrit 3.11 node for testing purposes.
    • https://review.opendev.org/c/opendev/system-config/+/949778 Update Gerrit images to 3.10.6 and 3.11.3
    • https://review.opendev.org/c/opendev/system-config/+/882900 Host Gerrit images on quay.io
    • Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup.
    • https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade
    • Will need to select a date once we know more about the upgrade itself.
  • Upgrading Old Servers (clarkb 20230627)
    • https://etherpad.opendev.org/p/opendev-bionic-server-upgrades
    • https://etherpad.opendev.org/p/opendev-focal-server-upgrades
    • https://etherpad.opendev.org/p/opendev-server-replacement-sprint
      • wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade
      • tonyb looking at cacti after wiki
    • On the easy list we've got refstack, mirror-update, eavesdrop, zuul schedulers, and zookeeper servers remaining.
  • Working through our TODO list (clarkb 20250107)
    • https://etherpad.opendev.org/p/opendev-january-2025-meetup
  • OFTC Matrix bridge no longer supporting new users (clarkb 20250520)
    • https://github.com/matrix-org/matrix-appservice-irc/issues/1851
    • This means that matrix bridge users don't see messages from newer IRC nicks/users
    • Options:
      • Declare bankruptcy: Matrix needs a Matrix client and IRC needs an IRC client
      • Pay EMS for a bridge. This is apparently billed per active IRC bridge user and could be a large increase on current EMS hosting cost
      • Host our own bridge. https://postmarketos.org/blog/2025/03/31/matrix-bridge-migration/ has useful related info
      • Embrace Matrix and move there. This represents a compromise between those who want IRC and those who want something like Slack/Discord. But for both sides the important functionality/needs seems to be preserved. (IRC users continue to get to use open source tooling with federated servers. Slack/Discord users get a system that runs over HTTPS using clients with modern bells and whistles).
  • Enabling hashtags globally (corvus 20250523)
    • Agreed to apply this the day of the meeting (June 3, 2025).
    • Announced at https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/thread/SVXT6X4WRYE6XQPB5PTWFUKUATICLO56/
  • Adding CentOS 10 Stream Support to Glean, DIB, and Nodepool (clarkb 20250527)
    • https://review.opendev.org/c/opendev/glean/+/941672 Glean NetworkManager Keyfile support
    • https://review.opendev.org/c/openstack/diskimage-builder/+/934045 DIB support for CentOS 10 Stream
    • There have been a few struggles
      • CentOS 10 Stream requires x86-64-v3 which clouds other than rax classic can provide (or about only 40% of our capacity).
      • CentOS 10 Stream upstream images label the / partition as /boot with a partition uuid. This seems like an upstream bug. centos-minimal element mitigates this.

• Open discussion

Upcoming Project Renames

(any additions should mention original->new full names and link to the corresponding project-config rename change in Gerrit) Changes should have their topic set to project-rename.

• Rename example/foo -> example/bar: https://review.opendev.org/c/openstack/project-config/+/123456

Previous meetings

Previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/infra/ and earlier at http://eavesdrop.openstack.org/meetings/ci/