Revision as of 18:00, 13 May 2024

Weekly Project Infrastructure team meeting

The OpenDev Team holds public weekly meetings in #opendev-meeting on OFTC, Tuesdays at 1900 UTC. Everyone interested in infrastructure and process surrounding automated testing and deployment is encouraged to attend.

Please feel free to add agenda items (and your IRC nick in parenthesis).

Agenda for next meeting

Announcements
- Clarkb out 5/17 and 5/20

Actions from last meeting

Specs Review

Topics
- Upgrading Old Servers (clarkb 20230627)
  - https://etherpad.opendev.org/p/opendev-bionic-server-upgrades
  - https://etherpad.opendev.org/p/opendev-focal-server-upgrades
  - Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58
  - Meetpad is done. Just need to clean up old servers when we feel confident we won't rollback.
  - What is next?
- AFS Mirror cleanups (clarkb 20240220)
  - Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial
  - Can followup with webserver log processing to determine which other mirrors may be dead.
- Building Ubuntu Noble Nodes (clarkb 20240416)
  - Next step is to add Noble to our mirrors
- Gerrit 3.9 Upgrade Planning (clarkb 20240423)
  - Upgrade prep notes are going in https://etherpad.opendev.org/p/gerrit-upgrade-3.9
  - https://www.gerritcodereview.com/3.9.html Release Notes
  - Downgrade is possible with an offline reindex using the old war.
  - Things to think about:
    - Do we want to enable diff3 diffs on merge changes?
    - Change topic limit seems to only apply to open changes. Being over the limit doesn't seem to impact the upgrade process
    - https://gerrit-review.googlesource.com/c/gerrit/+/424404 Upstream change to have release war build target without webfonts
    - There is a new batched ref updated event that collects multiple ref updated events in a single event. This is disabled by default, but could one day impact zuul. Work sorting out likely.
  - Server could stand to be upgraded as well but I think it is probably best to decouple these concerns.
- Cleaning up the openstack.org DNS zone (fungi 20240514)
  - Proposed for deletion: https://paste.opendev.org/show/bVHJchKcKBnlgTNRHLVK/

Open discussion

Upcoming Project Renames

(any additions should mention original->new full names and link to the corresponding project-config rename change in Gerrit) Changes should have their topic set to project-rename.

Rename example/foo -> example/bar: https://review.opendev.org/c/openstack/project-config/+/123456

Previous meetings

Previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/infra/ and earlier at http://eavesdrop.openstack.org/meetings/ci/

@@ Line 10: / Line 10: @@
 * Announcements
-** Gerrit User Summit details arriving soon. I've been told they would be interested to hear from us on how we do automated testing and management of Gerrit.
+** Clarkb out 5/17 and 5/20
 * Actions from last meeting
 * Specs Review
-** Mailman 3 spec https://review.opendev.org/810990
 * Topics
-** Improving OpenDev's CD throughput (clarkb 20211102)
+** Upgrading Old Servers (clarkb 20230627)
-*** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies
+*** https://etherpad.opendev.org/p/opendev-bionic-server-upgrades
-**** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs.
+*** https://etherpad.opendev.org/p/opendev-focal-server-upgrades
-***** Example 1: Combine service-gitea-lb and service-gitea jobs.
+*** Started looking at the wiki there are rough notes at: https://etherpad.opendev.org/p/opendev-bionic-server-upgrades#L58
-***** Example 2: Combine letsencrypt and nameserver jobs
+*** Meetpad is done. Just need to clean up old servers when we feel confident we won't rollback.
-***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job
+*** What is next?
-**** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs.
+** AFS Mirror cleanups (clarkb 20240220)
-**** https://review.opendev.org/c/opendev/system-config/+/807672
+*** Ubuntu Xenial cleanups are starting to show up under topic:drop-ubuntu-xenial
-***** should list dependencies for all jobs
+*** Can followup with webserver log processing to determine which other mirrors may be dead.
-***** zuul doesn't trigger on this?  not sure on best approach to make it mergable
+** Building Ubuntu Noble Nodes (clarkb 20240416)
-**** https://review.opendev.org/c/opendev/base-jobs/+/807807
+*** Next step is to add Noble to our mirrors
-***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks
+** Gerrit 3.9 Upgrade Planning (clarkb 20240423)
-***** this change makes split jobs to do this.  however, production remains the same as both are called.
+*** Upgrade prep notes are going in https://etherpad.opendev.org/p/gerrit-upgrade-3.9
-**** https://review.opendev.org/c/opendev/system-config/+/807808
+*** https://www.gerritcodereview.com/3.9.html Release Notes
-***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning.
+*** Downgrade is possible with an offline reindex using the old war.
-***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run
+*** Things to think about:
-** Gerrit Account cleanups (clarkb 20211102)
+**** Do we want to enable diff3 diffs on merge changes?
-*** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml
+**** Change topic limit seems to only apply to open changes. Being over the limit doesn't seem to impact the upgrade process
-** Fedora 34 test node booting problems (clarkb 20211102)
+**** https://gerrit-review.googlesource.com/c/gerrit/+/424404 Upstream change to have release war build target without webfonts
-*** Changes to Fedora's kernel packaging broke Xen
+**** There is a new batched ref updated event that collects multiple ref updated events in a single event. This is disabled by default, but could one day impact zuul. Work sorting out likely.
-*** Not yet sure if that may have also somehow broken OVH and iweb.
+*** Server could stand to be upgraded as well but I think it is probably best to decouple these concerns.
-** Zuul multi scheduler setup (clarkb 20211102)
+** Cleaning up the openstack.org DNS zone (fungi 20240514)
-*** Zuul ran with two schedulers for the first time over the last weekend.
+*** Proposed for deletion: https://paste.opendev.org/show/bVHJchKcKBnlgTNRHLVK/
-*** First jobs started by one scheduler and completed by another ran.
-*** Had to revert due to bugs in caching.
-*** Expect Zuul restarts as the scale out scheduler work in Zuul progresses.
-** FIPS testing in our CI system (clarkb 20211102)
-*** There is interest in testing various pieces of software against FIPS enabled systems in our Zuul.
-*** We are not building special FIPS images instead a Zuul role exists to update supported platforms and boot them into FIPS mode.
-*** Some tests have problems with ephemeral state being lost if the FIPS role runs too late. Then tests fail for unexpected reasons. Worth checking on reboot ordering relative to other test setup if there are problems.
 * Open discussion
@@ Line 53: / Line 45: @@
 == Upcoming Project Renames ==
 (any additions should mention original->new full names and link to the corresponding project-config rename change in Gerrit)
+Changes should have their topic set to project-rename.
-* Rename foo/example -> bar/example: https://review.opendev.org/123456
+* Rename example/foo -> example/bar: https://review.opendev.org/c/openstack/project-config/+/123456
 == Previous meetings ==
 Previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/infra/ and earlier at http://eavesdrop.openstack.org/meetings/ci/

Difference between revisions of "Meetings/InfraTeamMeeting"

Revision as of 18:00, 13 May 2024

Contents

Weekly Project Infrastructure team meeting

Agenda for next meeting

Upcoming Project Renames

Previous meetings