Jump to: navigation, search

Difference between revisions of "Meetings/InfraTeamMeeting"

(→‎Agenda for next meeting)
(→‎Agenda for next meeting)
 
(59 intermediate revisions by 5 users not shown)
Line 10: Line 10:
  
 
* Announcements
 
* Announcements
−
** Clarkb out early Friday and all of Monday. Expect a meeting Tuesday with a possibly delayed agenda.
 
  
 
* Actions from last meeting
 
* Actions from last meeting
Line 17: Line 16:
  
 
* Topics
 
* Topics
−
** Gerrit 3.11 Upgrade Planning (clarkb 20250401)
 
−
*** https://www.gerritcodereview.com/3.11.html
 
−
*** Please check this for any concerns with the way we use Gerrit.
 
−
*** Need to hold new test nodes after some Zuul launcher updates cleared out the old ones.
 
−
*** There is discussion about replication and reindexing bugs in 3.11.4/3.12.1 that don't affect us yet due to being on 3.10. We should try to test these problems to see if they affect us.
 
−
**** Clarkb tested this with held notes on admittedly trivial test data sets but was unable to reproduce the problems
 
−
*** Shutting down Gerrit can race the indexing of new changes. If this happens we can have new changes created that are not indexed leading to multiple changes with the same changeid later. To mitigate this we can/should reindex changes on Gerrit startup.
 
−
*** Gerrit shutdown remain plagued by large h2 cache files. Seems removing our long timeout was not enough to prevent shutdowns timing out.
 
−
*** https://etherpad.opendev.org/p/gerrit-upgrade-3.11 Planning Document for the eventual Upgrade
 
−
*** Will need to select a date once we know more about the upgrade itself.
 
−
** Gerrit Spontaneous Shutdown During Summit (clarkb 20251028)
 
−
*** The Gerrit server was discovered to be in a shutdown state with no recorded errors
 
−
*** The Nova team indicates this could happen if the hypervisor is unable to allocate memory. The host would be shutdown and Nova would notice but not record an error
 
−
*** Its possible the underlying cause is something else; either way we need to work with Vexxhost to identify the source of the problem and mitigate it
 
 
** Upgrading Old Servers (clarkb 20230627)
 
** Upgrading Old Servers (clarkb 20230627)
−
*** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades
+
*** https://etherpad.opendev.org/p/opendev-server-upgrade-planning Central tracking document which may link to more host specific documents
−
*** https://etherpad.opendev.org/p/opendev-focal-server-upgrades
 
−
*** https://etherpad.opendev.org/p/opendev-server-replacement-sprint
 
−
**** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade
 
−
**** tonyb looking at cacti after wiki
 
 
*** Next on the list are graphite and backup servers
 
*** Next on the list are graphite and backup servers
−
*** Can probably spin up new backup servers alongside the old ones then migrate the old volumes off the old servers to the new ones and finally delete the old servers. Just need to double check borg version support matrix details and also what adding new backup servers will do to our cron job setups for backups.
+
*** backup03.ca-ymq-1.vexxhost.opendev.org has been launched and is being backed up too
 +
**** https://review.opendev.org/c/opendev/system-config/+/995420 Starting backup02 removal here
 
*** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic
 
*** Remember to use launch-node's --config-drive flag when booting new Noble nodes in Rax Classic
−
** AFS mirror content updates (clarkb 20250916)
+
** Deploying a Prometheus for Server Metrics (clarkb 20260331)
−
*** https://review.opendev.org/c/opendev/system-config/+/965334 Mirror trixie packages
+
*** https://review.opendev.org/c/opendev/system-config/+/980840
−
*** Rocky Linux didn't hit these mirror issues because we don't mirror any rocky linux releases. Debian expects mirror content for all releases.
+
*** This change and its child deploy prometheus with node exporter to collect server metrics
−
** Zuul Launcher Updates (clarkb 20250923)
+
*** These two changes simplify the setup and testing of prometheus and node exporter
−
*** Nodes can be from mixed providers again due to a bug when reassigning unassigned nodes.
+
**** https://review.opendev.org/c/zuul/zuul-jobs/+/994564 manage /etc/hosts with public IPs
−
*** https://review.opendev.org/c/zuul/zuul/+/965954 Fix assignment of unassigned nodes.
+
**** https://review.opendev.org/c/opendev/system-config/+/994565 Use public IPs in system-config-run jobs
−
** Moving OpenDev Synchronous Communication to Matrix (clarkb 20250520)
+
** Dealing with alien zuul config errors in the openstack tenant (frickler 20260617)
−
*** The spec (954826) has merged.
+
*** clarkb reached out to starlingx and airship about this
−
*** clarkb will start working on this as soon as time permits.
+
**** Someone from airship is in the opendev matrix room now and their zuul tenant config errors appear to be gone
−
** Etherpad 2.5.2 Upgrade (clarkb 20250805)
+
**** https://lists.starlingx.io/archives/list/starlingx-discuss@lists.starlingx.io/thread/YQVACUR4OCX74ZULHAJ4AD44MHGY37YI/
−
*** https://github.com/ether/etherpad-lite/blob/v2.5.2/CHANGELOG.md
+
** Gitea 1.26.4 Upgrade (clarkb 20260622)
−
*** https://review.opendev.org/c/opendev/system-config/+/956593
+
*** https://review.opendev.org/c/opendev/system-config/+/994326 Upgrade Gitea to 1.26.4
−
*** This fixes our css issues and clarkb believes we are ready to upgrade.
+
*** Its time to upgrade to the next Gitea bugfix release
−
** Gitea 1.25.0 Upgrade (clarkb 20251028)
+
** Bump Anubis difficult to 5 (clarkb 20260630)
−
*** https://review.opendev.org/c/opendev/system-config/+/965960 Upgrade Gitea to 1.25.0
+
*** There is some evidence that bots are regularly solving the Anubis challenge
−
** Gitea Performance (clarkb 20251028)
+
*** The challenges are slowing them down enough that services continue to be mostly responsive
−
*** https://review.opendev.org/c/opendev/system-config/+/964728 Don't allow direct backend access
+
*** Should we increase the difficulty one level to slow them down even futher?
−
**** Some performance issues seem related to crawlers hitting backends directly so they aren't balanced properly
+
*** This will impact regular users too which is likely the primary consideration we should make.
−
**** Most recent issue was via traffic to the load balancer frontend so this won't fix everything
+
*** https://review.opendev.org/c/opendev/system-config/+/995096
−
**** Means we'll have to take extra steps to test backends directly (ssh port forwards)
+
** Planning Gerrit Project Renames (clarkb 20260622)
−
*** https://review.opendev.org/c/opendev/system-config/+/965420 Increase memcached cache size to mitigate effect of crawlers poisoning the cache
+
*** We have a request to rename x/cursive to openstack/cursive
 +
*** Any concern with project ownership doing that? The current group membership includes people from Johns Hopkins University and OpenStack Barbican
 +
*** Aiming for July 9 at ~2100 UTC
 +
**** Clarkb is unlikely to make this time
 +
*** https://etherpad.opendev.org/p/opendev-project-renames-20260709
 +
** Matrix Meetbot Porting (clarkb 20260707)
 +
*** Anil has started a port of the IRC meetbot to Matrix
 +
*** https://review.opendev.org/c/opendev/system-config/+/995755 stack starts here
 +
** OpenMetal Cloud Offline (clarkb 20260707)
 +
*** Zuul had errors talking to this cloud so the cloud was disabled
 +
*** `openstack server list` reports HTTP 500 errors
  
 
* Open discussion
 
* Open discussion
Line 68: Line 60:
 
Changes should have their topic set to project-rename.
 
Changes should have their topic set to project-rename.
  
−
* Rename example/foo -> example/bar: https://review.opendev.org/c/openstack/project-config/+/123456
+
* Rename x/cursive -> openstack/cursive: https://review.opendev.org/c/openstack/project-config/+/990122 (stephenfin, fungi)
  
 
== Previous meetings ==
 
== Previous meetings ==
 
Previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/infra/ and earlier at http://eavesdrop.openstack.org/meetings/ci/
 
Previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/infra/ and earlier at http://eavesdrop.openstack.org/meetings/ci/

Latest revision as of 01:52, 7 July 2026

Weekly Project Infrastructure team meeting

The OpenDev Team holds public weekly meetings in #opendev-meeting on OFTC, Tuesdays at 1900 UTC. Everyone interested in infrastructure and process surrounding automated testing and deployment is encouraged to attend.

Please feel free to add agenda items (and your IRC nick in parenthesis).

Agenda for next meeting

  • Announcements
  • Actions from last meeting
  • Specs Review
  • Open discussion

Upcoming Project Renames

(any additions should mention original->new full names and link to the corresponding project-config rename change in Gerrit) Changes should have their topic set to project-rename.

Previous meetings

Previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/infra/ and earlier at http://eavesdrop.openstack.org/meetings/ci/

Retrieved from "https://wiki.openstack.org/w/index.php?title=Meetings/InfraTeamMeeting&oldid=187739"