Jump to: navigation, search

Difference between revisions of "Meetings/InfraTeamMeeting"

(Agenda for next meeting)
(Agenda for next meeting)
 
(69 intermediate revisions by 6 users not shown)
Line 10: Line 10:
  
 
* Announcements
 
* Announcements
** Gerrit User Summit details arriving soon. I've been told they would be interested to hear from us on how we do automated testing and management of Gerrit.
+
** clarkb missing July 12 meeting
  
 
* Actions from last meeting
 
* Actions from last meeting
  
 
* Specs Review
 
* Specs Review
** Mailman 3 spec https://review.opendev.org/810990
 
  
 
* Topics
 
* Topics
** Improving OpenDev's CD throughput (clarkb 20211102)
+
** Improving OpenDev's CD throughput (clarkb 20220705)
*** We can run many of our jobs in parallel in all of our CD pipelines. But this requires we properly document/address dependencies
+
*** Bootstrapping bridge via Zuul is now a complicated subject. Can use zuul secrets to make it happen. Are we comfortable with this?
**** Need to understand our job dependencies and properly note them in Zuul config or address them by combining jobs.
+
*** https://review.opendev.org/c/opendev/infra-specs/+/821645 -- spec outlining some of the issues with secrets
***** Example 1: Combine service-gitea-lb and service-gitea jobs.
+
*** https://review.opendev.org/c/opendev/system-config/+/821155 -- sample of secret writing; more info in changelog
***** Example 2: Combine letsencrypt and nameserver jobs
+
*** Auto upgrades of Zuul are in place now.
***** Example 3: Have all jobs with webserver config express a dependency on the letsencrypt job
+
** Improving Grafana management tooling (clarkb 20220705)
**** Suggest we document the known job dependencies in a human readable format, then encode this into zuul, then we can switch to parallel runs.
+
*** Grafyaml doesn't properly support setting the color thresholds on graphs anymore (this makes failed states show red and happy states show green, we always seen green now)
**** https://review.opendev.org/c/opendev/system-config/+/807672
+
*** https://lists.opendev.org/pipermail/service-discuss/2022-July/000342.html
***** should list dependencies for all jobs
+
*** https://review.opendev.org/q/topic:grafana-json
***** zuul doesn't trigger on this?  not sure on best approach to make it mergable
+
** Zuul job POST_FAILURES (clarkb 20220705)
**** https://review.opendev.org/c/opendev/base-jobs/+/807807
+
*** TripleO and OSA are both seeing a higher than usual number of POST_FAILURES
***** currently every executor adds keys for bridge, then logs in and clones system-config before running playbooks
+
*** https://review.opendev.org/c/opendev/base-jobs/+/848027 Add remote log store location debug info to base jobs.
***** this change makes split jobs to do this.  however, production remains the same as both are called.
+
** Bastion host (ianw 20220628)
**** https://review.opendev.org/c/opendev/system-config/+/807808
+
*** worth moving ansible/openstacksdk to a venv?  system-config jobs first then production
***** this is a follow-on that adds a base job to clone system-config, and stops the other production jobs re-cloning.
+
*** c.f. https://review.opendev.org/c/opendev/system-config/+/847700
***** this job must run first, but then all other jobs can run in parallel, as they are all in the same buildset and using the same "view" of system-config for that particular run
+
*** bastion host OS upgrade.  prioin-place?  new host?  wait until have time to return to some of the bootstrapping/parallel job work?
** Gerrit Account cleanups (clarkb 20211102)
 
*** 33 conflicts remain. Clarkb has written notes on proposed plans for each user in the comments of review02:~clarkb/gerrit_user_cleanups/audit-results-annotated.yaml
 
** Fedora 34 test node booting problems (clarkb 20211102)
 
*** Changes to Fedora's kernel packaging broke Xen
 
*** Not yet sure if that may have also somehow broken OVH and iweb.
 
** Zuul multi scheduler setup (clarkb 20211102)
 
*** Zuul ran with two schedulers for the first time over the last weekend.
 
*** First jobs started by one scheduler and completed by another ran.
 
*** Had to revert due to bugs in caching.
 
*** Expect Zuul restarts as the scale out scheduler work in Zuul progresses.
 
** FIPS testing in our CI system (clarkb 20211102)
 
*** There is interest in testing various pieces of software against FIPS enabled systems in our Zuul.
 
*** We are not building special FIPS images instead a Zuul role exists to update supported platforms and boot them into FIPS mode.
 
*** Some tests have problems with ephemeral state being lost if the FIPS role runs too late. Then tests fail for unexpected reasons. Worth checking on reboot ordering relative to other test setup if there are problems.
 
  
 
* Open discussion
 
* Open discussion
Line 53: Line 38:
 
== Upcoming Project Renames ==
 
== Upcoming Project Renames ==
 
(any additions should mention original->new full names and link to the corresponding project-config rename change in Gerrit)
 
(any additions should mention original->new full names and link to the corresponding project-config rename change in Gerrit)
 +
Changes should have their topic set to project-rename.
  
 
* Rename foo/example -> bar/example: https://review.opendev.org/123456
 
* Rename foo/example -> bar/example: https://review.opendev.org/123456

Latest revision as of 19:15, 5 July 2022

Weekly Project Infrastructure team meeting

The OpenDev Team holds public weekly meetings in #opendev-meeting on OFTC, Tuesdays at 1900 UTC. Everyone interested in infrastructure and process surrounding automated testing and deployment is encouraged to attend.

Please feel free to add agenda items (and your IRC nick in parenthesis).

Agenda for next meeting

  • Announcements
    • clarkb missing July 12 meeting
  • Actions from last meeting
  • Specs Review
  • Open discussion

Upcoming Project Renames

(any additions should mention original->new full names and link to the corresponding project-config rename change in Gerrit) Changes should have their topic set to project-rename.

Previous meetings

Previous meetings, with their notes and logs, can be found at http://eavesdrop.openstack.org/meetings/infra/ and earlier at http://eavesdrop.openstack.org/meetings/ci/