Latest revision as of 01:52, 7 July 2026

Weekly Project Infrastructure team meeting

The OpenDev Team holds public weekly meetings in #opendev-meeting on OFTC, Tuesdays at 1900 UTC. Everyone interested in infrastructure and process surrounding automated testing and deployment is encouraged to attend.

Please feel free to add agenda items (and your IRC nick in parenthesis).

Agenda for next meeting

Announcements

Actions from last meeting

Specs Review

Topics
- Upgrading Old Servers (clarkb 20230627)
  - https://etherpad.opendev.org/p/opendev-server-upgrade-planning Central tracking document which may link to more host specific documents
  - Next on the list are graphite and backup servers
  - backup03.ca-ymq-1.vexxhost.opendev.org has been launched and is being backed up too
    - https://review.opendev.org/c/opendev/system-config/+/995420 Starting backup02 removal here
  - Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic
- Deploying a Prometheus for Server Metrics (clarkb 20260331)
  - https://review.opendev.org/c/opendev/system-config/+/980840
  - This change and its child deploy prometheus with node exporter to collect server metrics
  - These two changes simplify the setup and testing of prometheus and node exporter
    - https://review.opendev.org/c/zuul/zuul-jobs/+/994564 manage /etc/hosts with public IPs
    - https://review.opendev.org/c/opendev/system-config/+/994565 Use public IPs in system-config-run jobs
- Dealing with alien zuul config errors in the openstack tenant (frickler 20260617)
  - clarkb reached out to starlingx and airship about this
    - Someone from airship is in the opendev matrix room now and their zuul tenant config errors appear to be gone
    - https://lists.starlingx.io/archives/list/starlingx-discuss@lists.starlingx.io/thread/YQVACUR4OCX74ZULHAJ4AD44MHGY37YI/
- Gitea 1.26.4 Upgrade (clarkb 20260622)
  - https://review.opendev.org/c/opendev/system-config/+/994326 Upgrade Gitea to 1.26.4
  - Its time to upgrade to the next Gitea bugfix release
- Bump Anubis difficult to 5 (clarkb 20260630)
  - There is some evidence that bots are regularly solving the Anubis challenge
  - The challenges are slowing them down enough that services continue to be mostly responsive
  - Should we increase the difficulty one level to slow them down even futher?
  - This will impact regular users too which is likely the primary consideration we should make.
  - https://review.opendev.org/c/opendev/system-config/+/995096
- Planning Gerrit Project Renames (clarkb 20260622)
  - We have a request to rename x/cursive to openstack/cursive
  - Any concern with project ownership doing that? The current group membership includes people from Johns Hopkins University and OpenStack Barbican
  - Aiming for July 9 at ~2100 UTC
    - Clarkb is unlikely to make this time
  - https://etherpad.opendev.org/p/opendev-project-renames-20260709
- Matrix Meetbot Porting (clarkb 20260707)
  - Anil has started a port of the IRC meetbot to Matrix
  - https://review.opendev.org/c/opendev/system-config/+/995755 stack starts here
- OpenMetal Cloud Offline (clarkb 20260707)
  - Zuul had errors talking to this cloud so the cloud was disabled
  - `openstack server list` reports HTTP 500 errors

Open discussion

Upcoming Project Renames

(any additions should mention original->new full names and link to the corresponding project-config rename change in Gerrit) Changes should have their topic set to project-rename.

Rename x/cursive -> openstack/cursive: https://review.opendev.org/c/openstack/project-config/+/990122 (stephenfin, fungi)

Previous meetings

Previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/infra/ and earlier at http://eavesdrop.openstack.org/meetings/ci/

@@ Line 16: / Line 16: @@
 * Topics
-** Moving OpenDev's python-base/python-builder/uwsgi-base Images to Quay (clarkb 20250812)
-*** The images have moved and we're no longer publishing to docker hub
-*** https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/thread/HO6Z66QIMDIDY7CCVAREDOPSYZYNKIT3/
-*** I believe that all images within OpenDev that consume these base images have been updated to pull from the new location (including Gerrit)
-*** If you find images that need updates please push a change and let us know.
-** Gerrit 3.11 Upgrade Planning (clarkb 20250401)
-*** https://www.gerritcodereview.com/3.11.html
-*** Please check this for any concerns with the way we use Gerrit.
-*** Working on getting two new held nodes after images were rebuilt for 3.10.8.
-*** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us.
-**** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems
-*** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup.
-*** Gerrit shutdown remain plagued by large h2 cache files. Seems removing our long timeout was not enough to prevent shutdowns timing out.
-*** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade
-*** Will need to select a date once we know more about the upgrade itself.
 ** Upgrading Old Servers (clarkb 20230627)
-*** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades
+*** https://etherpad.opendev.org/p/opendev-server-upgrade-planning Central tracking document which may link to more host specific documents
-*** https://etherpad.opendev.org/p/opendev-focal-server-upgrades
-*** https://etherpad.opendev.org/p/opendev-server-replacement-sprint
-**** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade
-**** tonyb looking at cacti after wiki
-*** The entire openafs and kerberos system has been upgraded to Noble except for afs02.dfw.openstack.org. Waiting on RW volume moves to complete before performing this last upgrade
-**** Will need to remember to remove all of the hosts from the emergency file when done.
 *** Next on the list are graphite and backup servers
+*** backup03.ca-ymq-1.vexxhost.opendev.org has been launched and is being backed up too
+**** https://review.opendev.org/c/opendev/system-config/+/995420 Starting backup02 removal here
 *** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic
-** AFS mirror content cleanup (clarkb 20250916)
+** Deploying a Prometheus for Server Metrics (clarkb 20260331)
-*** Fungi identified a number of AFS mirror cleanups that can be performed once server upgrades are complete
+*** https://review.opendev.org/c/opendev/system-config/+/980840
-*** OpenEuler mirror content is no longer needed as we don't have any test images
+*** This change and its child deploy prometheus with node exporter to collect server metrics
-*** Debian stretch and bullseye backports are no longer needed as they are both old enough to have aged out
+*** These two changes simplify the setup and testing of prometheus and node exporter
-*** For the reprepro content we should also look into automatically clearing content if the cost to do so is actually a noop when we haven't removed releases from the mirror list.
+**** https://review.opendev.org/c/zuul/zuul-jobs/+/994564 manage /etc/hosts with public IPs
-** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520)
+**** https://review.opendev.org/c/opendev/system-config/+/994565 Use public IPs in system-config-run jobs
-*** The spec (954826) has merged.
+** Dealing with alien zuul config errors in the openstack tenant (frickler 20260617)
-*** clarkb will start working on this as soon as time permits.
+*** clarkb reached out to starlingx and airship about this
-** Pre PTG Planning (clarkb 20250624)
+**** Someone from airship is in the opendev matrix room now and their zuul tenant config errors appear to be gone
-*** https://etherpad.opendev.org/p/opendev-preptg-october-2025 Planning happening in this document
+**** https://lists.starlingx.io/archives/list/starlingx-discuss@lists.starlingx.io/thread/YQVACUR4OCX74ZULHAJ4AD44MHGY37YI/
-*** Clarkb has added a number of topic Ideas. Please add more.
+** Gitea 1.26.4 Upgrade (clarkb 20260622)
-*** Times: Tuesday October 7 1800-2000 UTC, Wednesday October 8 1500-1700 UTC, Thursday October 9 1500-1700
+*** https://review.opendev.org/c/opendev/system-config/+/994326 Upgrade Gitea to 1.26.4
-*** The Tuesday time block would effectively replace our weekly IRC meeting.
+*** Its time to upgrade to the next Gitea bugfix release
-** Etherpad 2.5.0 Upgrade (clarkb 20250805)
+** Bump Anubis difficult to 5 (clarkb 20260630)
-*** https://github.com/ether/etherpad-lite/blob/v2.5.0/CHANGELOG.md
+*** There is some evidence that bots are regularly solving the Anubis challenge
-*** This release is reported to have fixed our css problems.
+*** The challenges are slowing them down enough that services continue to be mostly responsive
-*** https://review.opendev.org/c/opendev/system-config/+/956593/
+*** Should we increase the difficulty one level to slow them down even futher?
-*** 104.130.127.119 is a held node for testing. You need to edit /etc/hosts to point etherpad.opendev.org at that IP.
+*** This will impact regular users too which is likely the primary consideration we should make.
-** Lists Server Slowness (clarkb 20250909)
+*** https://review.opendev.org/c/opendev/system-config/+/995096
-*** We think we have tracked this down to the different disk_io_index values between Performance and Standard flavors on rax classic
+** Planning Gerrit Project Renames (clarkb 20260622)
-*** Specifically the value is 2 on Standard (lists) and 40 on mirror and fio measures about a 20x differences in iops during randread testing to match
+*** We have a request to rename x/cursive to openstack/cursive
-*** We will need to migrate the mailman3 installation off of the server disk and onto a cinder volume to improve the number of iops.
+*** Any concern with project ownership doing that? The current group membership includes people from Johns Hopkins University and OpenStack Barbican
+*** Aiming for July 9 at ~2100 UTC
+**** Clarkb is unlikely to make this time
+*** https://etherpad.opendev.org/p/opendev-project-renames-20260709
+** Matrix Meetbot Porting (clarkb 20260707)
+*** Anil has started a port of the IRC meetbot to Matrix
+*** https://review.opendev.org/c/opendev/system-config/+/995755 stack starts here
+** OpenMetal Cloud Offline (clarkb 20260707)
+*** Zuul had errors talking to this cloud so the cloud was disabled
+*** `openstack server list` reports HTTP 500 errors
 * Open discussion
@@ Line 70: / Line 60: @@
 Changes should have their topic set to project-rename.
-* Rename example/foo -> example/bar: https://review.opendev.org/c/openstack/project-config/+/123456
+* Rename x/cursive -> openstack/cursive: https://review.opendev.org/c/openstack/project-config/+/990122 (stephenfin, fungi)
 == Previous meetings ==
 Previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/infra/ and earlier at http://eavesdrop.openstack.org/meetings/ci/

Difference between revisions of "Meetings/InfraTeamMeeting"

Latest revision as of 01:52, 7 July 2026

Contents

Weekly Project Infrastructure team meeting

Agenda for next meeting

Upcoming Project Renames

Previous meetings