Jump to: navigation, search

Difference between revisions of "CinderXenaPTGSummary"

m (add "Greetings and some Cinder project business")
m (added "Not possible for a non-admin user to determine via API if a volume will be multi-attach")
Line 30: Line 30:
  
 
===Not possible for a non-admin user to determine via API if a volume will be multi-attach===
 
===Not possible for a non-admin user to determine via API if a volume will be multi-attach===
 +
 +
Here's the context for the discussion: https://github.com/kubernetes/cloud-provider-openstack/pull/1368#issuecomment-819442191 and the comments following that one.
 +
 +
Basic idea is that cloud end users want to run kubernetes in OpenStack clouds.  The kubernetes installation "operator" (kubernetes term, we're talking about a program, not the human operator of the OpenStack cloud) for an end user thus runs with normal OpenStack project credentials (not admin).  For this particular issue, the kubernetes installer program wants to allow multi-attach block devices if the underlying cinder in that cloud supports it. (It does this by looking at the volume types available to the end user and creating a kubernetes storage class for each volume type.)  Multiattach info is in volume-type extra-specs that are only visible to admins (because they could contain backend-specific info about the deployment that shouldn't be exposed to end users).  So currently the only way for the installer to determine if multi-attach is possible is to create a volume of each available volume-type and to check if the resulting volume is multiattach.  This is non-optimal (some might say time-consuming and stupid).
 +
 +
The issue of exposing non-sensitive extra-specs info to end users has come up before in the history of cinder, and the suggestion was to either name volume-types appropriately (e.g., "Gold level with multiattach") or to include this info in the volume-type description.  But this is variable from cloud to cloud and not suitable for machine consumption.  It would be better to have some kind of standard field for this in the volume-type-show response.
 +
 +
A few issues came up in the course of this discussion:
 +
* We should probably provide some clear documentation around this.  For instance, if you have a volume-type that can match multiple backends, whether a built volume is multiattach will depend both on (a) whether the volume type supports multi-attach, and (b) the scheduler puts the volume on a backend that also supports multiattach (which also depends on: (c) how you've configured the scheduler).  So to support this use case (or, really, any use case where an end user wants to know whether a volume will be multiattach without having to build a volume first), a cinder admin needs to make sure that volume-types supporting multiattach are tied to backends that also support it (and the scheduler is configured appropriately).
 +
** but what if the backend runs out of space?  Well, that's ok, the end user won't be able to create a volume in it.  But that seems better than getting a volume that doesn't have the properties you want.
 +
** this didn't come up in discussion, but it looks like there's a tension between having very general volume types (e.g., could be multiattach, but maybe not, but as long as there's space on some backend, you'll get a volume) and very specific types (e.g., only has multiattach <is> True extra spec if all the backends that map to the type support multiattach).  If we're just going to expose the multiattach extra spec, then a cinder admin can't have a general type that supports multiattach when available, because that won't meet the kubernetes use case we're discussing.  Now maybe that's not a big deal, because for multiattach, it seems like if you want multiattach for your workload, you really want it, and so it would be kind of pointless to have a multiattach volume-type that sometimes didn't give you a multiattach volume.  But I wonder if that's the case for other non-sensitive extra-specs.  What I'm getting at is I wonder whether what we want is to introduce volume-type-metadata where a cinder admin could expose whatever info the admin deems suitable for end users.  The downside is that the admin would have to configure info twice, once in extra-specs and once in volume-type-metadata, but it would also make it possible to add key/values for particular consumers (e.g., "k8s:supports_multiattach": "true" (these would have to be string values), "k8s:whatever": "something", or even metadata indicating that the volume-type should be used by kubernetes at all).  The upside is that the key/values could be defined by the kubernetes-cinder-csi operator, not cinder.
 +
* Manila currently exposes a set of backend capabilities to end users while keeping others secret: https://docs.openstack.org/manila/latest/admin/capabilities_and_extra_specs.html
 +
* This is related to the capabilities reporting that Eric discussed at the Denver PTG and put up an initial spec for: https://review.opendev.org/c/openstack/cinder-specs/+/655939
 +
** the basic idea is that although there's capabilities reporting defined (and implemented in some drivers), it's not specific enough to help an operator to write extra-specs (e.g., supports_qos: True doesn't help you write qos extra-specs for a backend (though it does tell you to look at the driver docs or code to figure out what to do))
 +
*** this is the init_vendor_properties function: see https://github.com/openstack/cinder/commit/0e2783360ce730beed3423bee31ad9726a51c8e1
 +
** implementing this wouldn't help the kubernetes use case directly (because this capabilities API wouldn't be exposed to end users), but getting it defined could help in the effort to identify suitable backend-capabilities that could later be exposed to end users (or at least developing a suitable vocabulary)
 +
* Walt pointed out that we currently expose some volume-admin-metadata to the cinder admin in the volume-show response, so that could be a model for what we do here, expose select extra-specs to end users in the volume-type-show response
 +
* might be better to add an API to ask what volume types support a feature the end user is interested in
 +
* useful capabilities to expose:
 +
** multiattach
 +
** availability zones
 +
** volume replication
 +
** online extend capability
 +
** QoS
 +
** might want to make this configurable (sort of like resource_filters) so cinder admin can decide what to expose?
 +
* this would be useful to Horizon or other UIs -- for example, don't offer an end user to try to do an extend of an attached volume of the volume-type doesn't support it
 
====Conclusions====
 
====Conclusions====
 +
* This would be a good addition to Cinder
 +
* For the OpenShift/kubernetes use case, exposing some extra-specs in a standard format to non-admin users is sufficient.
 +
* Matt Booth and Tom Barron will draft a spec
 +
* the Cinder team will review and help keep the spec moving along
 +
* Eric will resuscitate his capabilities API spec; the team expressed general support for it
  
 
===The Interoperability Working Group===
 
===The Interoperability Working Group===

Revision as of 13:13, 27 April 2021

Contents

Introduction

This page contains a summary of the subjects covered during the Cinder project sessions at the Project Team Gathering for the Xena development cycle, held virtually April 19-23, 2021. The Cinder project team met from Tuesday 20 April to Friday 23 April, for 3 hours each day (1300-1600 UTC), with Friday's sessions stretching to 4 hours.

The Cinder Team at the Xena (Virtual) PTG, April 2021.


This document aims to give a summary of each session. More context is available on the cinder PTG etherpad:


The sessions were recorded, so to get all the details of any discussion, you can watch/listen to the recording. Links to the recordings are located at appropriate places below.

Tuesday 20 April

recording

Greetings and some Cinder project business

Business in no particular order:

  • The final release from stable/train for all deliverables is 12 May 2021, so check for any critical bugs that should be backported and get those moving along right away.
  • We need to do a release from the rbd-iscsi-client to verify that all the tooling is working. Want to do this before any critical bug is discovered that will require a release. The patch formatting rbd-iscsi-client as an OpenStack project still needs reviews: https://review.opendev.org/c/openstack/rbd-iscsi-client/+/774748/
  • There's a patch up placing cinder-specific dates on the OpenStack release schedule. The only notable change from the usual dates is to move the spec freeze to 2 weeks after the R-18 midcycle. (One week didn't give people enough time to revise and resubmit, which required a bunch of spec freeze exceptions. Hopefully we can avoid that this time.) Please check over the dates and leave comments if you see any problems: https://review.opendev.org/c/openstack/releases/+/786951
  • The two midcycles at weeks R-18 and R-9 have been working well, so let's do that again. Format will be 2 hours. I propose holding them in the place of the cinder weekly meeting that week; that way, everyone should be able to attend for at least the one hour they have set aside for the cinder meeting. Time would probably be 1300-1500 UTC. Please check for conflicts:
    • R-18 midcycle: 2 June
    • R-9 midcycle: 4 August
  • Some quick links to be aware of
    • general info on cinder project: tiny.cc/cinder-info
    • cinder group info: tiny.cc/cinder-groups
  • We've got someone interested in helping with documentation. I've asked her to start by reviewing release notes on patches for clarity, grammar, etc. These are user-facing documents so it's helpful to get them correct.
  • We didn't schedule time for a Wallaby Release Cycle Retrospective, so maybe we can do that during happy hour today.

Not possible for a non-admin user to determine via API if a volume will be multi-attach

Here's the context for the discussion: https://github.com/kubernetes/cloud-provider-openstack/pull/1368#issuecomment-819442191 and the comments following that one.

Basic idea is that cloud end users want to run kubernetes in OpenStack clouds. The kubernetes installation "operator" (kubernetes term, we're talking about a program, not the human operator of the OpenStack cloud) for an end user thus runs with normal OpenStack project credentials (not admin). For this particular issue, the kubernetes installer program wants to allow multi-attach block devices if the underlying cinder in that cloud supports it. (It does this by looking at the volume types available to the end user and creating a kubernetes storage class for each volume type.) Multiattach info is in volume-type extra-specs that are only visible to admins (because they could contain backend-specific info about the deployment that shouldn't be exposed to end users). So currently the only way for the installer to determine if multi-attach is possible is to create a volume of each available volume-type and to check if the resulting volume is multiattach. This is non-optimal (some might say time-consuming and stupid).

The issue of exposing non-sensitive extra-specs info to end users has come up before in the history of cinder, and the suggestion was to either name volume-types appropriately (e.g., "Gold level with multiattach") or to include this info in the volume-type description. But this is variable from cloud to cloud and not suitable for machine consumption. It would be better to have some kind of standard field for this in the volume-type-show response.

A few issues came up in the course of this discussion:

  • We should probably provide some clear documentation around this. For instance, if you have a volume-type that can match multiple backends, whether a built volume is multiattach will depend both on (a) whether the volume type supports multi-attach, and (b) the scheduler puts the volume on a backend that also supports multiattach (which also depends on: (c) how you've configured the scheduler). So to support this use case (or, really, any use case where an end user wants to know whether a volume will be multiattach without having to build a volume first), a cinder admin needs to make sure that volume-types supporting multiattach are tied to backends that also support it (and the scheduler is configured appropriately).
    • but what if the backend runs out of space? Well, that's ok, the end user won't be able to create a volume in it. But that seems better than getting a volume that doesn't have the properties you want.
    • this didn't come up in discussion, but it looks like there's a tension between having very general volume types (e.g., could be multiattach, but maybe not, but as long as there's space on some backend, you'll get a volume) and very specific types (e.g., only has multiattach <is> True extra spec if all the backends that map to the type support multiattach). If we're just going to expose the multiattach extra spec, then a cinder admin can't have a general type that supports multiattach when available, because that won't meet the kubernetes use case we're discussing. Now maybe that's not a big deal, because for multiattach, it seems like if you want multiattach for your workload, you really want it, and so it would be kind of pointless to have a multiattach volume-type that sometimes didn't give you a multiattach volume. But I wonder if that's the case for other non-sensitive extra-specs. What I'm getting at is I wonder whether what we want is to introduce volume-type-metadata where a cinder admin could expose whatever info the admin deems suitable for end users. The downside is that the admin would have to configure info twice, once in extra-specs and once in volume-type-metadata, but it would also make it possible to add key/values for particular consumers (e.g., "k8s:supports_multiattach": "true" (these would have to be string values), "k8s:whatever": "something", or even metadata indicating that the volume-type should be used by kubernetes at all). The upside is that the key/values could be defined by the kubernetes-cinder-csi operator, not cinder.
  • Manila currently exposes a set of backend capabilities to end users while keeping others secret: https://docs.openstack.org/manila/latest/admin/capabilities_and_extra_specs.html
  • This is related to the capabilities reporting that Eric discussed at the Denver PTG and put up an initial spec for: https://review.opendev.org/c/openstack/cinder-specs/+/655939
    • the basic idea is that although there's capabilities reporting defined (and implemented in some drivers), it's not specific enough to help an operator to write extra-specs (e.g., supports_qos: True doesn't help you write qos extra-specs for a backend (though it does tell you to look at the driver docs or code to figure out what to do))
    • implementing this wouldn't help the kubernetes use case directly (because this capabilities API wouldn't be exposed to end users), but getting it defined could help in the effort to identify suitable backend-capabilities that could later be exposed to end users (or at least developing a suitable vocabulary)
  • Walt pointed out that we currently expose some volume-admin-metadata to the cinder admin in the volume-show response, so that could be a model for what we do here, expose select extra-specs to end users in the volume-type-show response
  • might be better to add an API to ask what volume types support a feature the end user is interested in
  • useful capabilities to expose:
    • multiattach
    • availability zones
    • volume replication
    • online extend capability
    • QoS
    • might want to make this configurable (sort of like resource_filters) so cinder admin can decide what to expose?
  • this would be useful to Horizon or other UIs -- for example, don't offer an end user to try to do an extend of an attached volume of the volume-type doesn't support it

Conclusions

  • This would be a good addition to Cinder
  • For the OpenShift/kubernetes use case, exposing some extra-specs in a standard format to non-admin users is sufficient.
  • Matt Booth and Tom Barron will draft a spec
  • the Cinder team will review and help keep the spec moving along
  • Eric will resuscitate his capabilities API spec; the team expressed general support for it

The Interoperability Working Group

Conclusions

Wallaby cycle retrospective

Conclusions

Wednesday 21 April

recordings

Removing the Block Storage API v2

Conclusions

mypy status and next steps

Conclusions

Quotas testing

Conclusions

Fix up volume driver classes

Conclusions

Cinder throttle and cgroup v2

Conclusions

Cross-project meeting with Nova

Conclusions

Thursday 22 April (Drivers' Day)

recordings

Using Software Factory for Cinder Third Party CI

Conclusions

NVMe-oF and MDRAID replication approach - next steps for connector and agent

Conclusions

How to handle retyping/migrating nonencrypted volumes to encrypted volumes of the same size

Conclusions

Small topics

What gate tests need to be done for A/A coverage of a driver?

Using cinder oslo-versioned-objects (OVOs) in tests instead of dicts

Friday 23 April

recordings

Market trends and new cinder features

Conclusions

Snapshotting attached volumes

Conclusions

Multiple volumes goes to a same backend/pool as scheduler intances (HA) use only its internal state to balance the volumes among pools

Conclusions

Making the backup process asynchronous

Conclusions

Several small topics

Volume list queried by error state should contain error_managing state

Support revert any snapshot to the volume

Update volume az after service restart when the volume backend az was changed

OpenStack client still doesn't support cinder microversions

Conclusions

Consistent and Secure RBAC

Conclusions