Latest revision as of 01:52, 7 July 2026

Weekly Project Infrastructure team meeting

The OpenDev Team holds public weekly meetings in #opendev-meeting on OFTC, Tuesdays at 1900 UTC. Everyone interested in infrastructure and process surrounding automated testing and deployment is encouraged to attend.

Please feel free to add agenda items (and your IRC nick in parenthesis).

Agenda for next meeting

Announcements

Actions from last meeting

Specs Review

Topics
- Upgrading Old Servers (clarkb 20230627)
  - https://etherpad.opendev.org/p/opendev-server-upgrade-planning Central tracking document which may link to more host specific documents
  - Next on the list are graphite and backup servers
  - backup03.ca-ymq-1.vexxhost.opendev.org has been launched and is being backed up too
    - https://review.opendev.org/c/opendev/system-config/+/995420 Starting backup02 removal here
  - Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic
- Deploying a Prometheus for Server Metrics (clarkb 20260331)
  - https://review.opendev.org/c/opendev/system-config/+/980840
  - This change and its child deploy prometheus with node exporter to collect server metrics
  - These two changes simplify the setup and testing of prometheus and node exporter
    - https://review.opendev.org/c/zuul/zuul-jobs/+/994564 manage /etc/hosts with public IPs
    - https://review.opendev.org/c/opendev/system-config/+/994565 Use public IPs in system-config-run jobs
- Dealing with alien zuul config errors in the openstack tenant (frickler 20260617)
  - clarkb reached out to starlingx and airship about this
    - Someone from airship is in the opendev matrix room now and their zuul tenant config errors appear to be gone
    - https://lists.starlingx.io/archives/list/starlingx-discuss@lists.starlingx.io/thread/YQVACUR4OCX74ZULHAJ4AD44MHGY37YI/
- Gitea 1.26.4 Upgrade (clarkb 20260622)
  - https://review.opendev.org/c/opendev/system-config/+/994326 Upgrade Gitea to 1.26.4
  - Its time to upgrade to the next Gitea bugfix release
- Bump Anubis difficult to 5 (clarkb 20260630)
  - There is some evidence that bots are regularly solving the Anubis challenge
  - The challenges are slowing them down enough that services continue to be mostly responsive
  - Should we increase the difficulty one level to slow them down even futher?
  - This will impact regular users too which is likely the primary consideration we should make.
  - https://review.opendev.org/c/opendev/system-config/+/995096
- Planning Gerrit Project Renames (clarkb 20260622)
  - We have a request to rename x/cursive to openstack/cursive
  - Any concern with project ownership doing that? The current group membership includes people from Johns Hopkins University and OpenStack Barbican
  - Aiming for July 9 at ~2100 UTC
    - Clarkb is unlikely to make this time
  - https://etherpad.opendev.org/p/opendev-project-renames-20260709
- Matrix Meetbot Porting (clarkb 20260707)
  - Anil has started a port of the IRC meetbot to Matrix
  - https://review.opendev.org/c/opendev/system-config/+/995755 stack starts here
- OpenMetal Cloud Offline (clarkb 20260707)
  - Zuul had errors talking to this cloud so the cloud was disabled
  - `openstack server list` reports HTTP 500 errors

Open discussion

Upcoming Project Renames

(any additions should mention original->new full names and link to the corresponding project-config rename change in Gerrit) Changes should have their topic set to project-rename.

Rename x/cursive -> openstack/cursive: https://review.opendev.org/c/openstack/project-config/+/990122 (stephenfin, fungi)

Previous meetings

Previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/infra/ and earlier at http://eavesdrop.openstack.org/meetings/ci/

@@ Line 16: / Line 16: @@
 * Topics
-** Gerrit 3.11 Upgrade Planning (clarkb 20250401)
-*** https://www.gerritcodereview.com/3.11.html
-*** Please check this for any concerns with the way we use Gerrit.
-*** Held nodes for these two builds:
-**** https://zuul.opendev.org/t/openstack/build/f1ca0d1f2e054829a4506ececb58bed3
-**** https://zuul.opendev.org/t/openstack/build/588723b923e94901af3065143d9df818
-*** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us.
-**** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems
-*** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup.
-*** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade
-*** Will need to select a date once we know more about the upgrade itself.
 ** Upgrading Old Servers (clarkb 20230627)
-*** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades
+*** https://etherpad.opendev.org/p/opendev-server-upgrade-planning Central tracking document which may link to more host specific documents
-*** https://etherpad.opendev.org/p/opendev-focal-server-upgrades
-*** https://etherpad.opendev.org/p/opendev-server-replacement-sprint
-**** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade
-**** tonyb looking at cacti after wiki
-*** Kerberos and OpenAFS servers all updated to Jammy. afs01.dfw.openstack.org was subsequently updated to Noble and then began to unping
-**** Seems likely that Noble is simply not honoring the old /etc/network/interfaces scripts and needs netplan or similar. We should be able to rescue the node and set that directly or set a password for login via the console to do it interactively.
-**** I've also discovered that all but afs02.dfw.openstack.org and kdc04.opendev.org are running on an older xen version. In CI we've seen Noble boot with a single vcpu on the older hypervisors. We may also need to request these hosts get migrated to new hypervisors before updating. We can see what afs01.dfw.openstack.org looks like when networking is happy again.
 *** Next on the list are graphite and backup servers
+*** backup03.ca-ymq-1.vexxhost.opendev.org has been launched and is being backed up too
+**** https://review.opendev.org/c/opendev/system-config/+/995420 Starting backup02 removal here
 *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic
-** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520)
+** Deploying a Prometheus for Server Metrics (clarkb 20260331)
-*** https://review.opendev.org/c/opendev/infra-specs/+/954826 Spec outlining the motivation and plan for Matrix trialing
+*** https://review.opendev.org/c/opendev/system-config/+/980840
-** Pre PTG Planning (clarkb 20250624)
+*** This change and its child deploy prometheus with node exporter to collect server metrics
-*** https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document
+*** These two changes simplify the setup and testing of prometheus and node exporter
-*** Clarkb has added a number of topic Ideas. Please add more.
+**** https://review.opendev.org/c/zuul/zuul-jobs/+/994564 manage /etc/hosts with public IPs
-*** Times: Tuesday October 7 1800-2000 UTC, Wednesday October 8 1500-1700 UTC, Thursday October 9 1500-1700
+**** https://review.opendev.org/c/opendev/system-config/+/994565 Use public IPs in system-config-run jobs
-*** The Tuesday time block would effectively replace our weekly IRC meeting.
+** Dealing with alien zuul config errors in the openstack tenant (frickler 20260617)
-** Loss of upstream Debian bullseye-backports mirror (fungi 20250805)
+*** clarkb reached out to starlingx and airship about this
-*** https://review.opendev.org/956497 temporarily ignore undefinedtarget
+**** Someone from airship is in the opendev matrix room now and their zuul tenant config errors appear to be gone
-*** Zuul-jobs will no longer enable debian backports by default on September 9 (day of this meeting)
+**** https://lists.starlingx.io/archives/list/starlingx-discuss@lists.starlingx.io/thread/YQVACUR4OCX74ZULHAJ4AD44MHGY37YI/
-**** https://lists.zuul-ci.org/archives/list/zuul-announce@lists.zuul-ci.org/thread/NZ54HYFHIYW3OILYYIQ72L7WAVNSODMR/
+** Gitea 1.26.4 Upgrade (clarkb 20260622)
-*** Once zuul-jobs is updated we can delete the content from our mirrors and drop the workaround
+*** https://review.opendev.org/c/opendev/system-config/+/994326 Upgrade Gitea to 1.26.4
-** Etherpad 2.5.0 Upgrade (clarkb 20250805)
+*** Its time to upgrade to the next Gitea bugfix release
-*** https://github.com/ether/etherpad-lite/blob/v2.5.0/CHANGELOG.md
+** Bump Anubis difficult to 5 (clarkb 20260630)
-*** This release is reported to have fixed our css problems.
+*** There is some evidence that bots are regularly solving the Anubis challenge
-*** https://review.opendev.org/c/opendev/system-config/+/956593/
+*** The challenges are slowing them down enough that services continue to be mostly responsive
-*** 104.130.127.119 is a held node for testing. You need to edit /etc/hosts to point etherpad.opendev.org at that IP.
+*** Should we increase the difficulty one level to slow them down even futher?
-** Moving OpenDev's python-base/python-builder/uwsgi-base Images to Quay (clarkb 20250812)
+*** This will impact regular users too which is likely the primary consideration we should make.
-*** All of our containers that are build from these base images are now running on Noble nodes with podman
+*** https://review.opendev.org/c/opendev/system-config/+/995096
-*** A zuul user reported problems with the move to always using buildx to build images. This has been corrected by setting network=host on the build itself (not just the custom builder)
+** Planning Gerrit Project Renames (clarkb 20260622)
-*** If there are no other complaints by the time of this meeting I think we are in good shape to actually proceed with the container image move on our side.
+*** We have a request to rename x/cursive to openstack/cursive
-** Adding Debian Trixie Base Python Container Images (clarkb 20250826)
+*** Any concern with project ownership doing that? The current group membership includes people from Johns Hopkins University and OpenStack Barbican
-*** https://review.opendev.org/c/opendev/system-config/+/958480
+*** Aiming for July 9 at ~2100 UTC
-*** Trixie is here. Let's bootstrap some container images so that we can migrate over to the new release in our containers.
+**** Clarkb is unlikely to make this time
-** Dropping Ubuntu Bionic Test Nodes (clarkb 20250826)
+*** https://etherpad.opendev.org/p/opendev-project-renames-20260709
-*** Zuul will be dropping Ansible 9 support soon. When that happens Ubuntu Bionic test nodes will stop working in Zuul as Python on them is too old for Ansible 11
+** Matrix Meetbot Porting (clarkb 20260707)
-*** Decided that since Bionic is well past its EOL date we don't need to make a big announcement. At least not yet. But cleanup is a good thing.
+*** Anil has started a port of the IRC meetbot to Matrix
-*** I wrote a number of changes to OpenDev tools like glean, bindep, and git-review to remove Bionic based test jobs
+*** https://review.opendev.org/c/opendev/system-config/+/995755 stack starts here
-*** Please call out others that you notice or push up changes for them too.
+** OpenMetal Cloud Offline (clarkb 20260707)
-*** It should still be possible to run python2.7 jobs if we need them on Ubuntu Jammy. Note that tox 4 is not compatible with python2.7 so we need to pin tox in the jobs to <4
+*** Zuul had errors talking to this cloud so the cloud was disabled
-** Lists Server Slowness (clarkb 20250909)
+*** `openstack server list` reports HTTP 500 errors
-*** Like every other service mailman is getting bombarded by web crawlers
-*** User agent lists have been updated to try and cut down on some of the impacts
-*** Service is extremely slow at times. When this occurs top reports significant iowait
-*** Thoughts on further improvements that could be made?
 * Open discussion
@@ Line 80: / Line 60: @@
 Changes should have their topic set to project-rename.
-* Rename example/foo -> example/bar: https://review.opendev.org/c/openstack/project-config/+/123456
+* Rename x/cursive -> openstack/cursive: https://review.opendev.org/c/openstack/project-config/+/990122 (stephenfin, fungi)
 == Previous meetings ==
 Previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/infra/ and earlier at http://eavesdrop.openstack.org/meetings/ci/

Difference between revisions of "Meetings/InfraTeamMeeting"

Latest revision as of 01:52, 7 July 2026

Contents

Weekly Project Infrastructure team meeting

Agenda for next meeting

Upcoming Project Renames

Previous meetings