Jump to: navigation, search

Difference between revisions of "Meetings/InfraTeamMeeting"

m
(Agenda for next meeting)
 
(95 intermediate revisions by 7 users not shown)
Line 10: Line 10:
  
 
* Announcements
 
* Announcements
 +
** clarkb out June 14-19. Will miss the June 18 meeting if we have one.
  
 
* Actions from last meeting
 
* Actions from last meeting
Line 16: Line 17:
  
 
* Topics
 
* Topics
** Service Coordinator Election (clarkb 20230822)
+
** Upgrading Old Servers (clarkb 20230627)
*** Nomination period ends today (the day of the meeting)
 
** Bastion host (ianw 20230627)
 
*** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge
 
*** https://review.opendev.org/q/topic:bridge-backups
 
** Mailman 3 (clarkb 20230627)
 
*** https://etherpad.opendev.org/p/mm3migration
 
*** Need to change site domains
 
** Gerrit updates (clarkb 20230718)
 
*** Replication task file leaks.
 
**** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk.
 
**** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867
 
**** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868
 
**** https://review.opendev.org/c/opendev/system-config/+/884779 Revert bind mounts for Gerrit plugin data
 
** Upgrading Bionic servers to Focal/Jammy (clarkb 20230627)
 
 
*** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades
 
*** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades
*** Next up mirrors and meetpad.
+
**** wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade
*** Clarkb needs to look into cleaning up the old CI registry node.
+
**** tonyb looking at cacti after wiki
** Rax IAD image upload struggles (clarkb 20230822)
+
*** https://etherpad.opendev.org/p/opendev-focal-server-upgrades
*** The Rax IAD region has struggled to upload images within the image upload timeout. This seems to get worse the more images you upload
+
**** tonyb expects to try a simple focal -> noble upgrade this week
**** Almost like images are processed serially and we create a queue that gets backed up.
+
** AFS Mirror cleanups (clarkb 20240220)
*** No issue has been filed with rax yet, but we're testing incremental unpausing of images to get more data.
+
*** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial
** Fedora cleanup (clarkb 20230627)
+
*** Can followup with webserver log processing to determine which other mirrors may be dead.
*** Need changes to remove the label and nodeset from Zuul/Nodepool
+
** Gitea 1.22 Upgrade Planning (clarkb 20240528)
** Gitea 1.20 (clarkb 20230627)
+
*** There is a Gitea 1.22.0 release now. Once we have the general upgrade working we can test the doctor tool to fixup the DB case sensitivity.
*** https://review.opendev.org/c/opendev/system-config/+/886993
+
*** https://review.opendev.org/c/opendev/system-config/+/920580
**** Ready to go if we are happy with the resulting behaviors.
+
** Fixing Cloud Launcher Ansible (clarkb 20240604)
*** We disable oauth2 but need to configure multiple oauth2 jwt settings in order to avoid startup errors.
+
*** Git updates broke our cloud launcher playbook even after Frickler addressed security group redundancy errors
**** Need to update prod secrets with the oauth2 jwt settings before merging.
+
*** https://review.opendev.org/c/opendev/system-config/+/921685 workaround the Git issue
*** https://github.com/go-gitea/gitea/issues/25377
+
*** We should know on the day of the meeting if this is working again.
** Zuul changes and updates (clarkb 20230822)
+
** Improving mailman email throughput (clarkb 20240604)
*** Ansible 8 now availalble. We should convert the OpenDev tenant. Notify openstack tenant that we will switch them after the upcoming release
+
*** We've noticed slowness delivering emails for openstack-discuss
*** Stdout and stderr to be uncombined in Zuul Ansible. Currently this is a job setting for testing purposes. We'll need to start looking at how this impacts us.
+
*** Increasing the number of out runners hasn't alleviated this issue (it should prevent openstack-discuss from holding up other lists though)
*** Early failure detection regexes should arrive soon. This is potentially useful for our users?
+
*** Next we need to modify the mailman email batch count and exim verp queue sizes.
** Python container updates (tonyb 20230718)
+
** OpenMetal Cloud Rebuild (clarkb 20240604)
*** Some moderate progress in the first round
+
*** The old cloud is gone and the new cloud is getting configured.
*** https://review.opendev.org/q/topic:bookworm-python and https://review.opendev.org/q/topic:force-base-image-build
+
*** https://etherpad.opendev.org/p/openmetal-cloud-bootstrapping captures out todo list for bootstrapping the new cloud
*** I noticed that https://opendev.org/openstack/python-openstackclient/src/branch/master/Dockerfile is using a 3.9 tag which is buster
+
** Testing Rackspace's New Cloud Offering (clarkb 20240604)
**** We should stick to the explicit debian release + explicit python release versions for things. We can delete the old tags to force errors that indicate people should update the tags they use.
+
*** Rax reached out via a ticket to our nodepool account indicating we can test out their new offering. They are specifically looking for feedback
*** We should write a tool to find and clean leaked tags on docker hub (and quay.io?)
+
*** Probably need to reach out to rax and find out a bit more information before determining if we are the target audience, but if so this is a good opportunity for more test nodes and improved clouds.
  
 
* Open discussion
 
* Open discussion
Line 63: Line 50:
 
Changes should have their topic set to project-rename.
 
Changes should have their topic set to project-rename.
  
* Rename foo/example -> bar/example: https://review.opendev.org/123456
+
* Rename example/foo -> example/bar: https://review.opendev.org/c/openstack/project-config/+/123456
  
 
== Previous meetings ==
 
== Previous meetings ==
 
Previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/infra/ and earlier at http://eavesdrop.openstack.org/meetings/ci/
 
Previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/infra/ and earlier at http://eavesdrop.openstack.org/meetings/ci/

Latest revision as of 21:03, 10 June 2024

Weekly Project Infrastructure team meeting

The OpenDev Team holds public weekly meetings in #opendev-meeting on OFTC, Tuesdays at 1900 UTC. Everyone interested in infrastructure and process surrounding automated testing and deployment is encouraged to attend.

Please feel free to add agenda items (and your IRC nick in parenthesis).

Agenda for next meeting

  • Announcements
    • clarkb out June 14-19. Will miss the June 18 meeting if we have one.
  • Actions from last meeting
  • Specs Review
  • Topics
    • Upgrading Old Servers (clarkb 20230627)
    • AFS Mirror cleanups (clarkb 20240220)
      • Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial
      • Can followup with webserver log processing to determine which other mirrors may be dead.
    • Gitea 1.22 Upgrade Planning (clarkb 20240528)
    • Fixing Cloud Launcher Ansible (clarkb 20240604)
    • Improving mailman email throughput (clarkb 20240604)
      • We've noticed slowness delivering emails for openstack-discuss
      • Increasing the number of out runners hasn't alleviated this issue (it should prevent openstack-discuss from holding up other lists though)
      • Next we need to modify the mailman email batch count and exim verp queue sizes.
    • OpenMetal Cloud Rebuild (clarkb 20240604)
    • Testing Rackspace's New Cloud Offering (clarkb 20240604)
      • Rax reached out via a ticket to our nodepool account indicating we can test out their new offering. They are specifically looking for feedback
      • Probably need to reach out to rax and find out a bit more information before determining if we are the target audience, but if so this is a good opportunity for more test nodes and improved clouds.
  • Open discussion

Upcoming Project Renames

(any additions should mention original->new full names and link to the corresponding project-config rename change in Gerrit) Changes should have their topic set to project-rename.

Previous meetings

Previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/infra/ and earlier at http://eavesdrop.openstack.org/meetings/ci/