Meetings/InfraTeamMeeting

Weekly Project Infrastructure team meeting

The OpenDev Team holds public weekly meetings in #opendev-meeting on OFTC, Tuesdays at 1900 UTC. Everyone interested in infrastructure and process surrounding automated testing and deployment is encouraged to attend.

Please feel free to add agenda items (and your IRC nick in parenthesis).

Agenda for next meeting

• Announcements
  • Clarkb will miss the next meeting (September 3, 2024) due to the Summit.

• Actions from last meeting

• Specs Review

• Topics
  • Upgrading Old Servers (clarkb 20230627)
    • https://etherpad.opendev.org/p/opendev-bionic-server-upgrades
      • wiki.openstack.org: https://etherpad.opendev.org/p/opendev-mediawiki-upgrade
      • tonyb looking at cacti after wiki
    • https://etherpad.opendev.org/p/opendev-focal-server-upgrades
  • AFS Mirror cleanups (clarkb 20240220)
    • Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial
    • CentOS 8 Stream EOLd and jobs can no longer successfully run there. Cleanup is happening under topic:drop-centos-8-stream
    • Can followup with webserver log processing to determine which other mirrors may be dead.
  • Rackspace's Flex Cloud (clarkb 20240604)
    • Fungi and Clarkb spoke to rax about this and tl;dr is it is an entirely new cloud generation and we can use our existing accounts
    • Fungi got clouds.yaml updated and is able to talk to the new cloud region.
    • Need to enroll the cloud into our cloud launcher to configure networks, subnets, routers, security groups, and ssh keys.
      • https://review.opendev.org/c/opendev/system-config/+/927214
    • Need to figure out flavors and quotas.
    • Optionally upload the upstream Ubuntu Noble Cloud Image
  • Etherpad 2.2.2 Upgrade (clarkb 20240820)
    • Etherpad 2.2.2 has been released which breaks compatibility with our ep_headings plugin.
    • fungi tested etherpad 2.2.2 + ep_headings2 against our prod database and found that it seems to work. We can probably go ahead and upgrade but maybe be extra cautious and do a db backup before the upgrade?
    • https://review.opendev.org/c/opendev/system-config/+/926078
  • Service Coordinator Election (clarkb 20240820)
    • clarkb was the only nomination. Is Service Coordinator for six more months by default
    • If a nomination was missed now is the time to bring that up.
  • OSUOSL ARM Cloud Issues (clarkb 20240827)
    • nb04 ran out disk on /opt which broke image builds
    • Cleaning up /opt/dib_tmp didn't produce working builds because losetup was also out of device space
    • Current status is server has been rebooted to clear out stale losetup loopback devices and leaked mounts with another round of disk cleaning in progress.
    • Separately Kolla has discovered that jobs take significantly more time on OSUOSL nodes than they did on linaro and seem to fail more than they succeed. Other projects are having fewer issues.
      • Not sure yet what the underlying issue is but hunches are that it is related to IO (since Neutron unittests don't seem super bothered but Kolla container builds are).
  • Updating ansible+ansible-lint versions in our repos (clarkb 20240827)
    • After updating our default nodeset to Noble we discovered that ansible-lint with all of our pins cannot run on Noble due to incompatibilities with python 3.12.
    • Updating to latest ansible-lint and the current version of Zuul's Ansible install (>=8,<9) does work but introduces new rules and lint errors
    • https://review.opendev.org/c/openstack/project-config/+/926848
    • https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/926970
      • This change fails due to an inability to build openafs packages on CentOS 9 Stream arm64 images due to out of date kernels. Hopefully fixing those image builds will correct this problem.

• Open discussion

Upcoming Project Renames

(any additions should mention original->new full names and link to the corresponding project-config rename change in Gerrit) Changes should have their topic set to project-rename.

• Rename example/foo -> example/bar: https://review.opendev.org/c/openstack/project-config/+/123456

Previous meetings

Previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/infra/ and earlier at http://eavesdrop.openstack.org/meetings/ci/