Jump to: navigation, search

Difference between revisions of "Meetings/InfraTeamMeeting"

(Agenda for next meeting)
 
(364 intermediate revisions by 15 users not shown)
Line 3: Line 3:
 
= Weekly Project Infrastructure team meeting =
 
= Weekly Project Infrastructure team meeting =
  
The OpenDev Team holds public weekly meetings in <code><nowiki>#opendev-meeting</nowiki></code>, Tuesdays at 1900 UTC. Everyone interested in infrastructure and process surrounding automated testing and deployment is encouraged to attend.
+
The OpenDev Team holds public weekly meetings in <code><nowiki>#opendev-meeting</nowiki></code> on OFTC, Tuesdays at 1900 UTC. Everyone interested in infrastructure and process surrounding automated testing and deployment is encouraged to attend.
  
 
Please feel free to add agenda items (and your IRC nick in parenthesis).
 
Please feel free to add agenda items (and your IRC nick in parenthesis).
Line 13: Line 13:
 
* Actions from last meeting
 
* Actions from last meeting
  
* Specs approval
+
* Specs Review
  
* Priority Efforts (Standing meeting agenda items. Please expand if you have subtopics.)
+
* Topics
** [http://specs.openstack.org/openstack-infra/infra-specs/specs/update-config-management.html Update Config Management]
+
** Upgrading Old Servers (clarkb 20230627)
*** topic:update-cfg-mgmt
+
*** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades
*** Zuul as CD engine
+
**** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade
** OpenDev
+
**** tonyb looking at cacti after wiki
*** Gerrit account and group inconsistencies
+
*** https://etherpad.opendev.org/p/opendev-focal-server-upgrades
**** https://etherpad.opendev.org/p/gerrit-user-consistency-2021 High level notes.
+
**** tonyb expects to try a simple focal -> noble upgrade this week
**** Group problems and 92 accounts with preferred emails missing external ids have been fixed.
+
** AFS Mirror cleanups (clarkb 20240220)
**** We have 17 accounts with preferred email addresses that don't have a matching external id
+
*** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial
**** We have ~642 accounts with conflicting emails in their external ids. This needs more investigating to better understand the fix for.
+
*** Can followup with webserver log processing to determine which other mirrors may be dead.
**** Need to correct the ~642 external id issues before we can push updates to refs/meta/external-ids with Gerrit online.
+
** Gerrit 3.9 Upgrade Planning (clarkb 20240423)
**** Workaround is we can stop Gerrit, push to external ids directly, reindex accounts (and groups?), start gerrit, then clear accounts caches (and groups caches?)
+
*** Gerrit has been upgraded
**** Next steps
+
*** https://review.opendev.org/c/opendev/system-config/+/920938 is ready when we think we won't revert to 3.8
***** Classify users further into situation groups
+
**** Followup changes to ^ add 3.10 image builds and testing
***** Decide on next steps for users depending on their situation group.
+
** Gitea 1.22 Upgrade Planning (clarkb 20240528)
***** Fix the preferred email issue if possible as this can be done with gerrit online
+
*** There is a Gitea 1.22.0 release now. Once we have the general upgrade working we can test the doctor tool to fixup the DB case sensitivity.
***** Start a refs/meta/external-ids checkout in a shared location and begin committing fixes to it. If we can't push all the fixes as separate commits we can squash them together and then push.
+
*** https://review.opendev.org/c/opendev/system-config/+/920580
*** Gerrit 3.3.1 includes the fix for Zuul and Zuul has the fixes too.
+
** Fixing Cloud Launcher Ansible (clarkb 20240604)
**** https://review.opendev.org/c/opendev/system-config/+/765021 Build 3.3 images, currently appears to need some work.
+
*** Git updates broke our cloud launcher playbook even after Frickler addressed security group redundancy errors
*** Configuration tuning
+
*** https://review.opendev.org/c/opendev/system-config/+/921061 workaround the Git issue
**** Using strong refs for jgit caches
+
** Increase Mailman 3 Out Runner Count (clarkb 20240604)
**** Batch user groups and threads
+
*** We've seen outbound mail delivery for large lists take noticeable amounts of time
*** Gitea OOMs
+
*** https://review.opendev.org/c/opendev/system-config/+/920765 Increase the number of out runners to speed up outbound mail delivery
**** https://review.opendev.org/c/opendev/system-config/+/774023 Rate limiting framework change for haproxy.
+
** OpenMetal Cloud Rebuild (clarkb 20240604)
 
+
*** Yuriys has been in touch via email indicating things are ready on their side to rebuild this cloud on their new platform
* General topics
+
*** We need to gracefully shutdown and remove the existing cloud resources from Nodepool
** OpenAFS cluster status (clarkb 20210209)
+
*** OpenMetal will work with us to build the new cloud, then we can add a mirror node and add the new cloud to Nodepool
*** What is server cluster status? Have they all been upgraded to 1.8.6?
+
*** https://review.opendev.org/c/openstack/project-config/+/921072 Nodepool cleanups
**** Upgrading servers to bionic then focal in place is next?
+
*** https://review.opendev.org/c/opendev/system-config/+/921075 System-config cleanups
** Bup and Borg Backups (clarkb 20210209)
+
** Testing Rackspace's New Cloud Offering (clarkb 20240604)
*** wiki backup status
+
*** Rax reached out via a ticket to our nodepool account indicating we can test out their new offering. They are specifically looking for feedback
*** borg disk consumption workarounds
+
*** Probably need to reach out to rax and find out a bit more information before determining if we are the target audience, but if so this is a good opportunity for more test nodes and improved clouds.
** Picking up steam on Puppet -> Ansible rewrites (clarkb 20210209)
 
*** Enable Xenial -> Bionic/Focal system upgrades
 
*** https://etherpad.opendev.org/p/infra-puppet-conversions-and-xenial-upgrades Start capturing TODO list here
 
** Deploy a new refstack.openstack.org server (kopecmartin 20210209)
 
*** https://review.opendev.org/c/opendev/system-config/+/705258
 
*** ianw offered to help with the deployment side
 
*** kopecmartin will help with testing the new instance
 
** Meetpad was not functional late last week. Seems fine now (clarkb 2020210209)
 
** opendev.org not reachable via IPv6 from some ISPs (frickler 20210215)
 
*** Caused by missing/inconsistent IRR records for the networks that vexxhost uses
 
*** Pinged mnaser a couple of times but no progress so far
 
*** Added here for increased visibility and tracking
 
  
 
* Open discussion
 
* Open discussion
Line 64: Line 52:
 
== Upcoming Project Renames ==
 
== Upcoming Project Renames ==
 
(any additions should mention original->new full names and link to the corresponding project-config rename change in Gerrit)
 
(any additions should mention original->new full names and link to the corresponding project-config rename change in Gerrit)
 +
Changes should have their topic set to project-rename.
  
* foo/example -> bar/example: https://review.opendev.org/#/c/123456
+
* Rename example/foo -> example/bar: https://review.opendev.org/c/openstack/project-config/+/123456
  
 
== Previous meetings ==
 
== Previous meetings ==
 
Previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/infra/ and earlier at http://eavesdrop.openstack.org/meetings/ci/
 
Previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/infra/ and earlier at http://eavesdrop.openstack.org/meetings/ci/

Latest revision as of 20:56, 3 June 2024

Weekly Project Infrastructure team meeting

The OpenDev Team holds public weekly meetings in #opendev-meeting on OFTC, Tuesdays at 1900 UTC. Everyone interested in infrastructure and process surrounding automated testing and deployment is encouraged to attend.

Please feel free to add agenda items (and your IRC nick in parenthesis).

Agenda for next meeting

  • Announcements
  • Actions from last meeting
  • Specs Review
  • Open discussion

Upcoming Project Renames

(any additions should mention original->new full names and link to the corresponding project-config rename change in Gerrit) Changes should have their topic set to project-rename.

Previous meetings

Previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/infra/ and earlier at http://eavesdrop.openstack.org/meetings/ci/