Latest revision as of 22:53, 22 May 2023

Weekly Project Infrastructure team meeting

The OpenDev Team holds public weekly meetings in #opendev-meeting on OFTC, Tuesdays at 1900 UTC. Everyone interested in infrastructure and process surrounding automated testing and deployment is encouraged to attend.

Please feel free to add agenda items (and your IRC nick in parenthesis).

Agenda for next meeting

Announcements

Actions from last meeting

Specs Review

Topics
- Migrating to Quay (clarkb 20230523)
  - https://etherpad.opendev.org/p/opendev-quay-migration-2023 Plan/TODO list
  - Discovered a deficiency in Docker impacting speculative testing of container images
    - https://etherpad.opendev.org/p/3anTDDTht91wLwohumzW
    - A number of issues have been found when setting up services with podman
      - No reliable source of packages for Ubuntu older than Jammy
      - No syslog support
    - Migration to podman still a reasonable goal but a longer term one. In the interim suggestion is we revert quay.io image moves for opendev back to docker hub to keep speculative image testing. Then work towards a future where we use not docker and can switch to not docker.io more easily.
- Bastion host (ianw 20230523)
  - https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge
  - https://review.opendev.org/q/topic:bridge-backups
- Mailman 3 (clarkb 20230523)
  - https://etherpad.opendev.org/p/mm3migration
  - Need to change site domains
- Gerrit replication leaked tasks (clarkb 20230523)
  - https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk.
    - https://bugs.chromium.org/p/gerrit/issues/detail?id=16867
    - https://bugs.chromium.org/p/gerrit/issues/detail?id=16868
- Upgrading Bionic servers to Focal/Jammy (clarkb 20230523)
  - https://etherpad.opendev.org/p/opendev-bionic-server-upgrades
  - Next up mirrors, meetpad, and ci registry.
  - Please grab a node or two.
- Fedora cleanup (clarkb 20230523)
  - Removal of Fedora 37 mirror impacted DIB jobs. DIB can fetch from upstream.
  - Proposed the removal of Fedora mirrors and/or images https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/IOYIYWGTZW3TM4TR2N47XY6X7EB2W2A6/
    - Proceed with removal as it seems no one is asking for Fedora specifically.
- Quo vadis storyboard (frickler 20230523)
  - ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html
  - Some projects decided to move back to launchpad at the most recent PTG

Open discussion

Upcoming Project Renames

(any additions should mention original->new full names and link to the corresponding project-config rename change in Gerrit) Changes should have their topic set to project-rename.

Rename foo/example -> bar/example: https://review.opendev.org/123456

Previous meetings

Previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/infra/ and earlier at http://eavesdrop.openstack.org/meetings/ci/

@@ Line 10: / Line 10: @@
 * Announcements
-** Gerrit User Summit details arriving soon. I've been told they would be interested to hear from us on how we do automated testing and management of Gerrit.
 * Actions from last meeting
 * Specs Review
-** Mailman 3 spec https://review.opendev.org/810990
 * Topics
-** Improving OpenDev's CD throughput (clarkb 20211102)
+** Migrating to Quay (clarkb 20230523)
-*** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies
+*** https://etherpad.opendev.org/p/opendev-quay-migration-2023 Plan/TODO list
-**** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs.
+*** Discovered a deficiency in Docker impacting speculative testing of container images
-***** Example 1: Combine service-gitea-lb and service-gitea jobs.
+**** https://etherpad.opendev.org/p/3anTDDTht91wLwohumzW
-***** Example 2: Combine letsencrypt and nameserver jobs
+**** A number of issues have been found when setting up services with podman
-***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job
+***** No reliable source of packages for Ubuntu older than Jammy
-**** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs.
+***** No syslog support
-**** https://review.opendev.org/c/opendev/system-config/+/807672
+**** Migration to podman still a reasonable goal but a longer term one. In the interim suggestion is we revert quay.io image moves for opendev back to docker hub to keep speculative image testing. Then work towards a future where we use not docker and can switch to not docker.io more easily.
-***** should list dependencies for all jobs
+** Bastion host (ianw 20230523)
-***** zuul doesn't trigger on this?  not sure on best approach to make it mergable
+*** https://review.opendev.org/q/topic:prod-bastion-group Remaining changes are part of parallel ansible runs on bridge
-**** https://review.opendev.org/c/opendev/base-jobs/+/807807
+*** https://review.opendev.org/q/topic:bridge-backups
-***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks
+** Mailman 3 (clarkb 20230523)
-***** this change makes split jobs to do this.  however, production remains the same as both are called.
+*** https://etherpad.opendev.org/p/mm3migration
-**** https://review.opendev.org/c/opendev/system-config/+/807808
+*** Need to change site domains
-***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning.
+** Gerrit replication leaked tasks (clarkb 20230523)
-***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run
+*** https://review.opendev.org/c/opendev/system-config/+/880672 Workaround "leaked" replication tasks on disk.
-** Gerrit Account cleanups (clarkb 20211102)
+**** https://bugs.chromium.org/p/gerrit/issues/detail?id=16867
-*** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml
+**** https://bugs.chromium.org/p/gerrit/issues/detail?id=16868
-** Fedora 34 test node booting problems (clarkb 20211102)
+** Upgrading Bionic servers to Focal/Jammy (clarkb 20230523)
-*** Changes to Fedora's kernel packaging broke Xen
+*** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades
-*** Not yet sure if that may have also somehow broken OVH and iweb.
+*** Next up mirrors, meetpad, and ci registry.
-** Zuul multi scheduler setup (clarkb 20211102)
+*** Please grab a node or two.
-*** Zuul ran with two schedulers for the first time over the last weekend.
+** Fedora cleanup (clarkb 20230523)
-*** First jobs started by one scheduler and completed by another ran.
+*** Removal of Fedora 37 mirror impacted DIB jobs. DIB can fetch from upstream.
-*** Had to revert due to bugs in caching.
+*** Proposed the removal of Fedora mirrors and/or images https://lists.opendev.org/archives/list/service-discuss@lists.opendev.org/thread/IOYIYWGTZW3TM4TR2N47XY6X7EB2W2A6/
-*** Expect Zuul restarts as the scale out scheduler work in Zuul progresses.
+**** Proceed with removal as it seems no one is asking for Fedora specifically.
-** FIPS testing in our CI system (clarkb 20211102)
+** Quo vadis storyboard (frickler 20230523)
-*** There is interest in testing various pieces of software against FIPS enabled systems in our Zuul.
+*** ML thread https://lists.opendev.org/pipermail/service-discuss/2022-October/000370.html
-*** We are not building special FIPS images instead a Zuul role exists to update supported platforms and boot them into FIPS mode.
+*** Some projects decided to move back to launchpad at the most recent PTG
-*** Some tests have problems with ephemeral state being lost if the FIPS role runs too late. Then tests fail for unexpected reasons. Worth checking on reboot ordering relative to other test setup if there are problems.
 * Open discussion
@@ Line 53: / Line 50: @@
 == Upcoming Project Renames ==
 (any additions should mention original->new full names and link to the corresponding project-config rename change in Gerrit)
+Changes should have their topic set to project-rename.
 * Rename foo/example -> bar/example: https://review.opendev.org/123456

Difference between revisions of "Meetings/InfraTeamMeeting"

Latest revision as of 22:53, 22 May 2023

Contents

Weekly Project Infrastructure team meeting

Agenda for next meeting

Upcoming Project Renames

Previous meetings