Jump to: navigation, search

Difference between revisions of "Meetings/InfraTeamMeeting"

(→‎Agenda for next meeting)
(→‎Agenda for next meeting)
 
(60 intermediate revisions by 5 users not shown)
Line 16: Line 16:
  
 
* Topics
 
* Topics
−
** Gerrit 3.11 Upgrade Planning (clarkb 20250401)
 
−
*** https://www.gerritcodereview.com/3.11.html
 
−
*** Please check this for any concerns with the way we use Gerrit.
 
−
*** Need to hold new test nodes after some Zuul launcher updates cleared out the old ones.
 
−
*** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us.
 
−
**** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems
 
−
*** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup.
 
−
*** Gerrit shutdown remain plagued by large h2 cache files. Seems removing our long timeout was not enough to prevent shutdowns timing out.
 
−
*** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade
 
−
*** Will need to select a date once we know more about the upgrade itself.
 
−
** Gerrit Spontaneous Shutdown During Summit (clarkb 20251028)
 
−
*** The Gerrit server was discovered to be in a shutdown state with no recorded errors
 
−
*** The Nova team indicates this could happen if the hypervisor is unable to allocate memory. The host would be shutdown and Nova would notice but not record an error
 
−
*** Its possible the underlying cause is something else; either way we need to work with Vexxhost to identify the source of the problem and mitigate it
 
 
** Upgrading Old Servers (clarkb 20230627)
 
** Upgrading Old Servers (clarkb 20230627)
−
*** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades
+
*** https://etherpad.opendev.org/p/opendev-server-upgrade-planning Central tracking document which may link to more host specific documents
−
*** https://etherpad.opendev.org/p/opendev-focal-server-upgrades
 
−
*** https://etherpad.opendev.org/p/opendev-server-replacement-sprint
 
−
**** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade
 
−
**** tonyb looking at cacti after wiki
 
 
*** Next on the list are graphite and backup servers
 
*** Next on the list are graphite and backup servers
−
*** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups.
+
*** backup03.ca-ymq-1.vexxhost.opendev.org has been launched and is being backed up too
 +
**** https://review.opendev.org/c/opendev/system-config/+/995420 Starting backup02 removal here
 
*** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic
 
*** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic
−
** AFS mirror content cleanup (clarkb 20250916)
+
** Deploying a Prometheus for Server Metrics (clarkb 20260331)
−
*** Can we automate cleanups for reprepro content? In other words are the steps cheap enough during the noop case to run with our regular reprepro update script?
+
*** https://review.opendev.org/c/opendev/system-config/+/980840
−
*** Additional volumes to look into cleanups for include ceph, docker, and puppetlabs package mirrors as well as our python wheel caches
+
*** This change and its child deploy prometheus with node exporter to collect server metrics
−
*** CentOS 9 Stream grows at a rate that requires regular quota bumps. Is there anything we can do about that?
+
*** These two changes simplify the setup and testing of prometheus and node exporter
−
*** With these cleanups and server upgrades completed we can probably start to consider adding new mirrors again.
+
**** https://review.opendev.org/c/zuul/zuul-jobs/+/994564 manage /etc/hosts with public IPs
−
**** Trixie, CentOS 10 Stream, Rocky Linux ?
+
**** https://review.opendev.org/c/opendev/system-config/+/994565 Use public IPs in system-config-run jobs
−
** Zuul Launcher Updates (clarkb 20250923)
+
** Dealing with alien zuul config errors in the openstack tenant (frickler 20260617)
−
*** NODE_FAILURES created by a bug in zuul-launcher
+
*** clarkb reached out to starlingx and airship about this
−
**** https://review.opendev.org/c/zuul/zuul/+/964893 is the fix which we should roll out as soon as it has merged
+
**** Someone from airship is in the opendev matrix room now and their zuul tenant config errors appear to be gone
−
** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520)
+
**** https://lists.starlingx.io/archives/list/starlingx-discuss@lists.starlingx.io/thread/YQVACUR4OCX74ZULHAJ4AD44MHGY37YI/
−
*** The spec (954826) has merged.
+
** Gitea 1.26.4 Upgrade (clarkb 20260622)
−
*** clarkb will start working on this as soon as time permits.
+
*** https://review.opendev.org/c/opendev/system-config/+/994326 Upgrade Gitea to 1.26.4
−
** Etherpad 2.5.1 Upgrade (clarkb 20250805)
+
*** Its time to upgrade to the next Gitea bugfix release
−
*** https://github.com/ether/etherpad-lite/blob/v2.5.1/CHANGELOG.md
+
** Bump Anubis difficult to 5 (clarkb 20260630)
−
*** There is an even newer 2.5.1 release. We'll need to recycle the test nodes and check if CSS has improved
+
*** There is some evidence that bots are regularly solving the Anubis challenge
−
*** Note we shouldn't upgrade until after the PTG as the event relies on etherpad.
+
*** The challenges are slowing them down enough that services continue to be mostly responsive
−
** Gitea 1.24.7 Upgrade (clarkb 20251028)
+
*** Should we increase the difficulty one level to slow them down even futher?
−
*** https://review.opendev.org/c/opendev/system-config/+/964899/ Upgrade Gitea to 1.24.7
+
*** This will impact regular users too which is likely the primary consideration we should make.
−
** Gitea Performance (clarkb 20251028)
+
*** https://review.opendev.org/c/opendev/system-config/+/995096
−
*** https://review.opendev.org/c/opendev/system-config/+/964728 Don't allow direct backend access
+
** Planning Gerrit Project Renames (clarkb 20260622)
−
**** Some performance issues seem related to crawlers hitting backends directly so they aren't balanced properly
+
*** We have a request to rename x/cursive to openstack/cursive
−
**** Most recent issue was via traffic to the load balancer frontend so this won't fix everything
+
*** Any concern with project ownership doing that? The current group membership includes people from Johns Hopkins University and OpenStack Barbican
−
**** Means we'll have to take extra steps to test backends directly (ssh port forwards)
+
*** Aiming for July 9 at ~2100 UTC
−
** PBR Updates to Handle Setuptools Deprecations (clarkb 20251028)
+
**** Clarkb is unlikely to make this time
−
*** Heads up setuptools has an October 31 deadline for no longer relying on easy_install tooling
+
*** https://etherpad.opendev.org/p/opendev-project-renames-20260709
−
*** https://review.opendev.org/c/openstack/pbr/+/964712/ and children aim to address this
+
** Matrix Meetbot Porting (clarkb 20260707)
 +
*** Anil has started a port of the IRC meetbot to Matrix
 +
*** https://review.opendev.org/c/opendev/system-config/+/995755 stack starts here
 +
** OpenMetal Cloud Offline (clarkb 20260707)
 +
*** Zuul had errors talking to this cloud so the cloud was disabled
 +
*** `openstack server list` reports HTTP 500 errors
  
 
* Open discussion
 
* Open discussion
Line 72: Line 60:
 
Changes should have their topic set to project-rename.
 
Changes should have their topic set to project-rename.
  
−
* Rename example/foo -> example/bar: https://review.opendev.org/c/openstack/project-config/+/123456
+
* Rename x/cursive -> openstack/cursive: https://review.opendev.org/c/openstack/project-config/+/990122 (stephenfin, fungi)
  
 
== Previous meetings ==
 
== Previous meetings ==
 
Previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/infra/ and earlier at http://eavesdrop.openstack.org/meetings/ci/
 
Previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/infra/ and earlier at http://eavesdrop.openstack.org/meetings/ci/

Latest revision as of 01:52, 7 July 2026

Weekly Project Infrastructure team meeting

The OpenDev Team holds public weekly meetings in #opendev-meeting on OFTC, Tuesdays at 1900 UTC. Everyone interested in infrastructure and process surrounding automated testing and deployment is encouraged to attend.

Please feel free to add agenda items (and your IRC nick in parenthesis).

Agenda for next meeting

  • Announcements
  • Actions from last meeting
  • Specs Review
  • Open discussion

Upcoming Project Renames

(any additions should mention original->new full names and link to the corresponding project-config rename change in Gerrit) Changes should have their topic set to project-rename.

Previous meetings

Previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/infra/ and earlier at http://eavesdrop.openstack.org/meetings/ci/

Retrieved from "https://wiki.openstack.org/w/index.php?title=Meetings/InfraTeamMeeting&oldid=187739"