Jump to: navigation, search

CinderAntelopePTGSummary

Revision as of 17:50, 31 October 2022 by Rajatdhasmana (talk | contribs) (→‎Image cache issue when volume created from cache is of less size than cache)

Introduction

The Sixth virtual PTG for the 2023.1 Antelope cycle of Cinder was conducted from Tuesday, 18th October, 2022 to Friday, 21 October, 2022, 4 hours each day (1300-1700 UTC). This page will provide a summary of all the topics discussed throughout the PTG.

Cinder 2023.1 Antelope Virtual PTG 19 October, 2022


This document aims to give a summary of each session. More context is available on the cinder Zed PTG etherpad:


The sessions were recorded, so to get all the details of any discussion, you can watch/listen to the recording. Links to the recordings are located at appropriate places below.


Tuesday 18 October

recordings

User survey feedback

https://lists.openstack.org/pipermail/openstack-discuss/2022-October/030843.html https://docs.google.com/spreadsheets/d/1hHC4hg_Zt9FLYYJ7UA9iVomBbExUhhyJd2QpmrriiBQ/edit#gid=0

The user survey feedback comments are summarized in the following 3 sections:

1) Done:


2) Actionable:

  • Document HA deployments
  • Online retyping between different Ceph RBD backends (clusters) ==> Eric will try to look if libvirt supports it now
  • Improvements on encryption: key rotation, multiple LUKS keys ==> Could explore some ideas


3) Questions:

  • Real Active/Active ==> What does this mean specifically?
  • Live migration with Pure iSCSI ==> This should work in new OpenStack releases
  • Error management:
  • Better attach/detach cleanup on failure ==> For example not leaving volumes on reserved/detaching?
  • Better error handling when failed to create/mount/delete ==> User Messages?
  • Better support for cinder-backup services- especially the filesystem drivers. ==> Bug in driver?
  • Volume Group expansion ==> Extend volumes? Or more operations (which)?


User survey question review

The details provided by operators in the user survey feedback were vague and the team agreed to revise the questions to yield more useful information in the feedback.

The team proposed some good ideas as follows:

  • Ask them to provide driver along with protocol
  • Revise the list to mention driver with protocol for operators to select like NetApp iSCSI, HPE3PAR FC etc
  • Alphabetical ordering would be good and easy to find relevant driver-protocol combination
  • Be specific about the feedback, provide release, launchpad bug link if there is an issue


Based on the points, we've revised the user survey feedback questions in the following etherpad.

https://etherpad.opendev.org/p/antelope-ptg-cinder-user-survey-current-questions

SLURP release cadence

The concept of SLURP (Skip Level Upgrade Release Process) was introduced because six month upgrades are difficult infeasible, or undesirable for operators. 2023.1 Antelope will the the first SLURP release of OpenStack. following are some of the details to keep in mind with respect to SLURP and not SLURP releases.

  • every other release will be considered to be a “SLURP (Skip Level Upgrade Release Process)” release
  • Upgrades will be supported between “SLURP” releases, in addition to between adjacent major releases
  • Deployments wishing to move to a one year upgrade cycle will synchronize on a “SLURP” release, and then skip the following “not-SLURP” release
  • Testing: test upgrade between SLURP releases
  • Deprecations: deprecation, waiting, and removal can only happen in “SLURP” releases
  • Data migrations: Part of supporting “SLURP to SLURP” upgrades involves keeping a stable (read “compatible” not “unchanging”) database schema from “SLURP to SLURP”
  • Releasenotes: https://review.opendev.org/c/openstack/project-team-guide/+/843457


For detailed info: https://governance.openstack.org/tc/resolutions/20220210-release-cadence-adjustment.html

Cinder well know Encryption Problem

Presentation: https://docs.google.com/presentation/d/1HOHnO9T3BD1KO5uk_y34aWhMs_A5i9ANPn6zIujQxCk/edit

This has been a complex issue to handle and is being discussed since multiple PTGs. There was another topic discussed, "Allocation size vs requested size for specific storage provider like Dell PowerFlex", which had work items that would act as a initial base for the encryption work having the following work items:

  • Keep two DB fields for the user size and actual size
    • requested size -> user size
    • allocated size -> real size
  • Partition the volume and only the partition with user size should be visible inside the VM


The encryption work will follow up on this initial work to implement the following:

  • Calculate encryption header size to know how much user visible size is in the volume
  • start encrypting the volume on creation instead of doing it on first attachment

conclusions

action: Sofia to work on the encryption work after the initial base work is completed

Operator hour

Christian Rohmann joined us and briefed us about their deployment and the current pain points they have with respect to cinder. They were mostly focused on backup related things and a number of backup topics were discussed.


1) State of non-rbd cinder-backup drivers such as S3

The current problem is, non-rbd backends are not optimized as they copy data chunk by chunk. Also they don't work very well with different type of volumes like thin provisioned, encrypted etc.

To address this issue, we will need to implement a generic block tracking feature. We can split the feature into two: backend and frontend.

  • action: Gorka agrees to do a brain dump of what he looked into for future reference


2) Encryption layer for backups

We can implement an encryption layer on the backups using barbican or a static key. The key scope could be global or per project basis.


3) Backup features we currently have

We also discussed about the backup features we currently have so operators could make good use of it:

  • Limit concurrent backup/restore operations
  • Scale backup service vertically and horizontally to improve performance
    • We can configure the worker processes for backup
    • Run cinder backup in Active-Active


4) Some other issues that were mentioned and are pain points of operators:

  • Cannot recover a backup process if the service dies, would be good to have it continue where it left off
  • RBD image has a lock and there is no way to know who/what left it there
  • Interested in 512e/4k support for RBD
  • Resuming operations after restarts
  • auto migrate volumes in pool


Image cache issue when volume created from cache is of less size than cache

Image cache is a very useful feature that allows us to clone and extend the volume from cache instead of downloading the whole image again and again from glance, hence providing optimization.

The problem we face is, if the first volume created with image cache enabled is a large volume (say 100GB) and subsequent volumes created from same image are small sized (say 10GB) then subsequent volumes will also be created of same size as first volume (i.e. 100GB instead of 10GB).

We discussed Possible ways to fix it:

  1. Create the first entry with the requested size, if another request comes in with a smaller size then update the cache entry
  2. Create the cache entry with the minimum sized volume required by the image
  3. Use a tuple (image-id, size) to query the cache entries and have multiple cache entries associated to a single image


The solution described in 2. seems to be simplest and most straightforward to implement.

conclusion

Retrieved from "https://wiki.openstack.org/w/index.php?title=CinderAntelopePTGSummary&oldid=182139"