- 1 Introduction
- 2 Train PTG Summary
- 2.1 Thursday 5/2/2019
- 2.1.1 Stein Retrospective
- 2.1.2 Ceph iSCSI Support
- 2.1.3 Cinder/Glance creating image from volume with Ceph
- 2.1.4 Glance image properties and Cinder encrypted volume key management
- 2.1.5 Encryption key manage on volume clone
- 2.1.6 Continued discussion of old CG API removal
- 2.1.7 Backup tests notifications leaking
- 2.1.8 Optional dependency install mechanism
- 2.1.9 Passwords in cinder.conf/oslo.config with Castellan driver
- 2.1.10 Fall mid-cycle planning
- 2.1.11 Migration from SQL-Alchemy to Alembic
- 2.1.12 Syncing scheduler stats in an HA environment
- 2.1.13 Pre-release checklist
- 2.1.14 Privsep and rootwrap
- 2.1.15 abc removal
- 2.1.16 Delete volume from DB
- 2.1.17 Quiesced snapshots
- 2.1.18 Talk about merging the RSD driver
- 2.1.19 3rd Party CI
- 2.1.20 Generic Backup Driver Discussion
- 2.1.21 Driver Folder Clean-up/Refactoring
- 2.1.22 Optimized Backup drivers
- 2.2 Friday 5/3/2019
- 2.2.1 Cinderclient design discussion
- 2.2.2 Default behavior of listing (volume|group} types=
- 2.2.3 Avoiding untyped volumes update
- 2.2.4 Leverage Hardware acceleration in Cinder
- 2.2.5 Cross Project time with Nova
- 2.2.6 Status of multiattach
- 2.2.7 Cinder mutable options
- 2.2.8 Cinder support for Glance multi-store feature
- 2.2.9 Supporting fast copy volume to image
- 2.2.10 Cinder handling image-associated metadata
- 2.2.11 Capabilities reporting update
- 2.2.12 python-cinderclient major version release
- 2.2.13 cinderclient integration with OpenStack
- 2.2.14 Continued Storyboard Discussion?
- 2.2.15 py37 failures
- 2.2.16 Placement Discussion
- 2.1 Thursday 5/2/2019
- 3 OpenInfra Summit Forum Sessions
This page contains a summary of the subjects covered during the Train Summit and PTG held in Denver, Colorado, USA, April 28 to May 4, 2019.
The full etherpad and all associated notes may be found here.
Train PTG Summary
- Summary: The release went relatively well though we had issues getting releases out We kept our deadlines and which was good and added the priority dashboard which was also beneficial.
- Action (jungleboyj): Clean up the Cinder Wiki pages.
- Action (team): Checkout dashboards and see if anything needs to be updated.
- Action (smcginnis): Add release notes for the cinderclient backports to call out the changes that upgrading will bring.
Ceph iSCSI Support
- Summary: There is great interest in getting this support in place from multiple consumers. We should continue to push trying to get this in place.
- Action (jungleboyj): Get the spec that Lenovo has started updated and pushed up for review.
- Action (eharney): Check with internal Red Hat teams to make sure that there are not others already working on this.
- Action (hemna): To reach out to the Ceph community and see how receptive they would be to client changes to support iSCSI use cases.
Cinder/Glance creating image from volume with Ceph
- Summary: Determined that the problem really needs to be handled on the Glance side so there was no action required by Cinder.
Glance image properties and Cinder encrypted volume key management
- Summary: Keys are not getting deleted from the HSM like they are supposed to be. Glance should be deleting keys when they are done being used. This was acceptable if we make it very clear the key is one that may be deleted.
- Action (rosmaita): Draft up a spec proposing this functionality.
- Action (eharney): Will write a Cinder spec to handle the transition from the old to the new approach of key management.
Encryption key manage on volume clone
- Summary: Right now when we clone volumes we just make a copy of the encryption key. Seems like this really should be a new key.
- Action (eharney): Go through how this work with snapshots -- snapshots keep the old key, volumes get a new key
- Action (eharney): Write a spec based on the results of the investigation.
Continued discussion of old CG API removal
- Summary: It appears that all drivers that still have the old functions in place actually route to the appropriate generic code. So, it should be safe to remove the old API.
- Action (smcginnis): To clean up the old code in the volume manager.
- Action (smcginnis): To clean up database tables for CGs (ConsistencyGroup and CGSnapshot)
- Action (smcginnis): Encourage drivers that still have the old code in place to remove the code.
Backup tests notifications leaking
- Summary: The leak of backup.createprogress notifications has been causing gate failures for some time. Can work around the issue by ignoring the notifications but we really should fix it.
- Action (eharney): Continue to work with Rajat to find the source of the problem.
Optional dependency install mechanism
- Summary: Drivers that require externally available packages don't work easily in containers. This isn't good given the general movement to the use of containers.
- Action (team): Watch new drivers for such dependencies. If they have them make sure they do a good try/except import of the package. Also ensure the licensing is appropriate.
- Action (hemna): Work on verifying the dependent packages and determining a way to resolve the problem for containers.
Passwords in cinder.conf/oslo.config with Castellan driver
- Summary: Consumers don't want to be putting clear passwords in the config files. There is a better way to do things and we should move to supporting it.
- Action (eharney): To investigate how we can implement this for Cinder.
Fall mid-cycle planning
- Summary: No one had objections to doing the mid-cycle at Lenovo again. So, we will plan for 8/21 to 8/23/2019 at the Lenovo Campus in Morrisville, NC
- Action (eharney): To get more East Coast Cinder people involved.
- Action (jungleboyj): To confirm with Lenovo and make sure to keep Tom Barron in the loop.
Migration from SQL-Alchemy to Alembic
- Summary: We need to do this at some point since SQL-Alchemy is going away. Glance has already done the work and we can learn from them.
- Action (smcginnis): Will look into compacting our DB upgrades again. Currently back at the Ocata level.
- Action (smcginnis): Try to get guidance on how to proceed from zzzeek.
Syncing scheduler stats in an HA environment
- Summary: As more people are running an active/active HA environment we need to think about this more to make sure that scheduler instances stay in sync.
- Action (e0ne): Update the old spec he wrote about this to address comments: https://review.opendev.org/#/c/556529/2
- 'Action (geguileo): To help out Ivan as he starts reworking the spec.
- Summary: We have made mistakes over the last couple of releases as far as getting libraries created before the release, etc. We need to do better. Hopefully creating a checklist will help. https://docs.openstack.org/cinder/latest/contributor/releasecycle.html
- Action (jungleboyj): To do follow-up patches to add additional details to the checklist.
- Action (eharney): Has additional comments on the content to merge.
Privsep and rootwrap
- Summary: We haven't really improved things in privsep as we are still using the rootwrap style of commands instead of breaking things down to using python functions to be more secure. We should improve this.
- Action (eharney): To take a look at his privsep patch for LIO and see if it could be pushed up.
- Action (eharney): Try to get better granularity on privileges
- Summary: ABC has never worked the way we intended. At this point is would be best to just remove it. We should plan to work on this in the U release as we are moving to Py3.
Delete volume from DB
- Summary: Support organizations would like a way to delete volumes that doesn't go through the driver code but people have concerns with going straight to the DB. Need to find a middle ground.
- Action (eharney): Going to look at the patch that was proposed and suggest that unmanage have an --ignore-state option added.
- Summary: People are still surprised we can't do this. We should work with Nova to see if we can make this happen.
Talk about merging the RSD driver
- Summary: 3rd Party CI looks good and the team has been responsive to comments. Close to being ready to merge.
- Action (team): To review the driver and try to get it in place.
- Action (e0ne): To make sure to review it given that he had experience with the PoC for the NVMe driver.
3rd Party CI
- Summary: We need to make sure that all 3rd Party CIs are running py3 by milestone 2 and that they have resolved all issues with the change in repo names.
- Action (jungleboyj): Make sure that all CIs are running py3 testing by milestone-2 in Train.
- Action (jungleboyj): Propose unsupported patches for those that fail to meet the requirement.
- Action (jungleboyj): Need to also ensure that all 3rd Party CIs are running the Cinder Tempest Plugin.
Generic Backup Driver Discussion
- Summary: The team would still like to get this in place and Ivan has continued to work on it. Need to review the patches that are out there to help this.
- Action (team): Review patches that are currently out there: https://review.opendev.org/#/q/status:open+project:openstack/cinder+branch:master+topic:bp/backup-host-selection-algorigthm and https://review.opendev.org/#/c/500094/
- Action (e0ne): To continue to push the team to review the patches.
Driver Folder Clean-up/Refactoring
- Summary: The volume driver folder is a bit of an inconsistent mess and it would be good to clean up. There is also a mess in the code to deal with.
- Action (smcginnis): Going to work on creating a patch to bring consistency to the subfolders in volume/drivers.
- Action (hemna): Will work to move exceptions for individual drivers out of cinder/exceptions.py into individual drivers.
- Action (jungleboyj): There is old code left around from previously removed drivers in os-brick. Review os-brick and remove what is appropriate.
Optimized Backup drivers
- Summary: NetApp indicated interest in creating an optimized backup driver for their storage backend. The team didn't have a concern with this idea.
- Action (erlon): To propose a review to implement this for the team to review.
Cinderclient design discussion
- Summary: There are issues in cinderclient when it comes to filtering. There are bugs and unexpected behavior that should be fixed.
- Action (whoami-rajat): Update patch to allow multiple '--filters' to be specified. https://review.opendev.org/#/c/587610/
- Action (whoami-rajat): Fix the bugs documented in the etherpad by Eric.
Default behavior of listing (volume|group} types=
- Summary: The default behavior between the client and the API is inconsistent. We should understand this and fix it.
- Action (whoami-rajat): Fix the bug where the server hides private types from the user that has access to them.
- Action (team): Review the patch that is out there and decide if we want to merge it: https://review.opendev.org/#/c/641698/
Avoiding untyped volumes update
- Summary: Despite additional design discussion we landed back on the fact that we want to continue with the original design proposal of creating a new default type that is unlikely to clash with anything that administrator previously created.
- Action (eharney): Write a but to evaluate whether our default volume type should be public.
- Action (e0ne): Write a bug to deal with the fact that default_volume_type can be set in cinder.conf but the type doesn't exist. You can create with no type, but it should fail.
Leverage Hardware acceleration in Cinder
- Summary: Intel is interested in speeding image compression/decompression using accelerators. Cinder team is ok with this but thinks it should be implemented in a generic manner via oslo-util or something similar.
- Action (lixiaoy1): Propose new functionality to Oslo.
- Action (lixiaoy1): Work with Cinder to get use of the new library function implemented. Same will need to be done with Glance.
- Action (lixiaoy1): Finally work with Nova to get the new way of compressing images and new format supported.
Cross Project time with Nova
- Summary: Discussed a number of different topics but there were no significant work items to come out of the discussion for Cinder. Details can be seen here: https://etherpad.openstack.org/p/ptg-train-xproj-nova-cinder
Status of multiattach
- Summary: Red Hat has started testing in their environment and they are seeing some race conditions, etc. They are addressing issues as they find them.
Cinder mutable options
- Summary: This is a community goal that we need to get appropriately implemented. Not totally clear as to where our implementation is at and we would like to address that.
- Action (erlon): Update this spec https://review.opendev.org/656011 to be current and include results of our discussion.
- Action (erlon): Make sure that we have the right plumbing in place to support this functionality. It sounds like we may already have it there.
- Action (erlon): Come up with a way to add to our sample config output to indicate which options are reloadable an not.
Cinder support for Glance multi-store feature
- Summary: Glance supports multiple stores and it should be possible to decide which store is used when Cinder interacts with Glance.
- Action (rosmaita): Work with abhishek to update the spec that is out there to initially implement this support via volume_type. If the approach isn't well received, other solutions can be considered.
Supporting fast copy volume to image
- Summary: Storage backends like Datera and Ceph can copy images faster using backend support, but there isn't currently a code path for this. We should correct this.
- Action (_alastor_): Propose the work he has done in Glance locally to upstream.
- Action (_alastor_): Work with rosmaita to get other details for implementing this worked out.
Cinder handling image-associated metadata
- Summary: We need to come up with a better framework for Cinder to handle Glance's image metadata. There are fields like those for the image signature that we don't want to be saving.
- Action (rosmaita): Will write up a spec with a proposal for the fields that shouldn't be saved in Cinder.
Capabilities reporting update
- Summary: HP actually started implementing a lot of this work back in Liberty. We just need to figure out where that work was left and keep pushing things forward.
- Action (_alastor_): Help the team document how this should be utilized by vendor's drivers.
- Action (eharney): Understand the capabilities support that was merged back in liberty.
- Action (eharney): Write a new spec that references the old spec and then makes appropriate updates on what the functionality really is.
- Action (team): Need to get the reference architectures updated to report capabilities properly.
- Action (team): If the reference architectures go well then we should probably add this as a requirement for drivers.
- Action (jungleboyj): Set up a bi-weekly meeting to discuss this.
python-cinderclient major version release
- Summary: There are a number of changes we want to get in before doing a Cinderclient release with a major version change. We agreed that we want to change the way we are handling MV to default to the highest available and then downgrade to the highest version that the server supports.
- Action (eharney): Update https://review.opendev.org/#/c/647871/ to implement the support agreed upon above.
- Action (rosmaita): Will wait to do a cinderclient release until after all the changes that require a major version bump are in place.
cinderclient integration with OpenStack
- Summary: Our current approach to doing client development is not great as thing are going into python-cinderclient and then eventually being picked up for openstackclient without any Cinder core review action. Would like to look at making 'openstack volume' commands an alias to Cinder commands made available via plugin.
- Action (abishop): Will reach out to Dean Troyer and find out if they would be supportive of this approach.
- Action (abishop): Investigate the questions and concerns raised during our discussion to get answers and bring them back to the team.
Continued Storyboard Discussion?
- Summary: Team is still not in a hurry to make this change. Manila isn't planning to move during Train. Don't know that anyone else is either so we are not going to push this further at this point in time.
- Summary: Team can't agree whether or not to add a py37 job to our tests because right now it is always failing. Sean thinks that we should add it as we know that OpenStack is going to be moving and we need to get test coverage in place.
- Action (jungleboyj): To reach out to Helen Walsh asking her team to investigate why their driver fails and to please address.
- Summary: Placement is agnostic as to what goes into the service. If we have information that can be of use to them, they will use it. No one, however, is asking for this right now so it was felt that our efforts could be better utilized elsewhere. So, no action is needed right now.
OpenInfra Summit Forum Sessions
Cinder Capability Reporting
We held a forum session to discuss how Cinder can better report capabilities from drivers. The goal being to make it possible for administrators to better understand the capabilities available from their storage backends. If this information is more readily accessible it should, then, be easier to create volume types that more completely utilize their storage backends.
- Etherpad: https://etherpad.openstack.org/p/denver-forum-cinder-improving-drvr-cap-rep
- YouTube Recording: https://www.youtube.com/watch?v=avZgbk8hh2s
The outcome of this session was a discovery that much of the work for this had already been done by HP back in the liberty release and that we really just needed to come to agreement upon completing the work. We added further discussion to our PTG agenda and you can see the results of that discussion above.
Cinder User Feedback Session
This forum session was a second attempt to give users of Cinder the opportunity to share their concerns about Cinder and to ask questions.
- Etherpad: https://etherpad.openstack.org/p/denver-forum-cinder-direct-user-feedback
- YouTube Recording: https://www.youtube.com/watch?v=UVN42jsUq-U
As with the past times we have tried this the session was lightly attended. The administrators in the room all indicated that Cinder generally works pretty well. There were requests for some functionality that had already gone in to recent releases. Also some questions around issues with the number of volumes that can be attached. That problem appears to be a limitation of virtio.