OpenStack - User contributions [en]

Manila/Meetings

2016-10-31T16:47:34Z

Clinton Knight:

= Weekly Manila team meeting =
'''NOTE MEETING TIME: Thursday at 15:00 UTC'''

If you're interested in management of shared filesystems for OpenStack, we have a weekly meetings in <code><nowiki>#openstack-meeting-alt</nowiki></code>, on Thursdays at 15:00 UTC. Please feel free to add items to the agenda below. NOTE: When adding topics please include your IRC name so we know who's topic it is and how to get more info.

'''Previous meeting logs can be found here:''' http://eavesdrop.openstack.org/meetings/manila/

== Next meeting ==
'''NOTE:''' ''Include your IRC nickname next to agenda items so that you can be called upon in the meeting and arrive at the meeting promptly if placing items in agenda. You might want to put this on your calendar if you are adding items.''

'''Scheduled for 3 Nov, 2016, 15:00 UTC'''
# Spec review & prioritization
#* https://review.openstack.org/#/q/project:%22%255Eopenstack/manila-specs%22++status:open
# Open Discussion

== Previous meetings ==

'''Scheduled for 20 Oct, 2016, 15:00 UTC'''
# Specs and Deadlines (bswartz)
#* https://review.openstack.org/#/c/374883/
# Adding invalid user to the access rule changes valid access rule state from 'active' to 'error' (ravichandran)
# Open Discussion

'''Scheduled for 13 Oct, 2016, 15:00 UTC'''
# Ocata Design Summit
#* https://www.openstack.org/summit/barcelona-2016/summit-schedule/global-search?t=Manila%3A
# PTG

'''Scheduled for 6 Oct, 2016, 15:00 UTC'''
# Microversion-ing addition or removal of transitional ('micro') states (gouthamr)
#* https://review.openstack.org/#/c/369668/

'''Scheduled for 29 Sep, 2016, 15:00 UTC'''
# Ocata Design Summit
#* https://etherpad.openstack.org/p/manila-ocata-design-summit-topics
# Open Discussion

'''Scheduled for 22 Sep, 2016, 15:00 UTC'''
# Specs process
#* https://review.openstack.org/#/c/374883

'''Scheduled for 8 Sep, 2016, 15:00 UTC'''
# RC1 status
#* https://launchpad.net/manila/+milestone/newton-rc1
# Project Logo
#* http://civs.cs.cornell.edu/cgi-bin/vote.pl?id=E_6f0d111cec78c5ef&akey=d34a751f2d084d79
# Container Driver Security (aovchinnikov)
#* https://review.openstack.org/#/c/353463/
# Security "vulnerability-managed" tag for manila (tbarron)
# Zanata translations for manila-client and manila-ui (tbarron)
# Tempest direction (tbarron)

'''Scheduled for 1 Sep, 2016, 15:00 UTC'''
# Mascot (bswartz)
# Vulnerability Management

'''Scheduled for 25 Aug, 2016, 15:00 UTC'''
# Huawei share replication and CI
# IPv6 (cknight)
# Open Discussion

'''Scheduled for 18 Aug, 2016, 15:00 UTC'''
# Driver private data admin API
#* Disagreement abounds on how this API is to be designed, tested, and presented.
#* https://review.openstack.org/#/c/315346/
# Scenario tests are broken
#* http://logs.openstack.org/86/309286/34/check/gate-manila-tempest-dsvm-scenario/610f96e/console.html
# Revert to snapshot
#* https://review.openstack.org/#/c/340502/
#* https://review.openstack.org/#/c/356682/

"Scheduled for 11 Aug, 2016, 15:00 UTC"
# None

'''Scheduled for 4 Aug, 2016, 15:00 UTC'''
# Migration APIs
#* http://lists.openstack.org/pipermail/openstack-dev/2016-August/100682.html
# Correct approach for avoiding races

'''Scheduled for 28 Jul, 2016, 15:00 UTC'''
# Stepping Down as Docs Liaison (dustins)
#* Current job responsibilities don't give me the time to devote to this that I'd like to have, so it should go to someone else with a passion for documentation and has the willingness and desire for taking up the mantle.
# N-3 Review focus
#* https://etherpad.openstack.org/p/manila-newton3-priorities

'''Scheduled for 21 Jul, 2016, 15:00 UTC'''
# Migrate API reference into tree (dgonzalez)
#* https://review.openstack.org/#/c/313874/

'''Scheduled for 14 Jul, 2016, 15:00 UTC'''
# N-2 milestone
# Managing initial security (of the root directory) for new shares
# Mascot/Logo discussion
# Tempest alternatives

'''Scheduled for 30 Jun, 2016, 15:00 UTC'''
# non-disruptive nfs-ganesha export modifications (rraja/tbarron)
#* https://sourceforge.net/p/nfs-ganesha/mailman/message/35173839
# share backup spec discussion (zhongjun)
#* https://review.openstack.org/#/c/330306

'''Scheduled for 23 Jun, 2016, 15:00 UTC'''
# Implement extend and shrink features for glusterfs driver (DaiDV)
# Do all drivers have to inherit from driver.ShareDriver? (mkoderer)
# Creating a second share instance for driver-assisted migration (ganso)

'''Scheduled for 16 Jun, 2016, 15:00 UTC'''
# A share support multiple protocols (zhongjun)
#* https://review.openstack.org/#/c/329392/
# Drivers with minimum size for shares (tpsilva)

'''Scheduled for 9 Jun, 2016, 15:00 UTC'''
# HPB feature (mkoderer)
# the fate of nova-network support in manila (gouthamr)
# multi-AZ tests in the gate

'''Scheduled for 2 Jun, 2016, 15:00 UTC'''
# N-1 milestone (bswartz)
# Decision on APIv1 removal for newton (bswartz)
# Model updates from drivers (bswartz)
# Midcycle meetup
# Spec update
#* New spec - Extending share networks to span subnets - https://review.openstack.org/#/c/323646/ (gouthamr)

'''Scheduled for 26 May, 2016, 15:00 UTC'''
# N-1 milestone
# Update access remaining work
# Topics not covered at design summit https://etherpad.openstack.org/p/newton-manila-contributor-meetup
#* Nova-network removal
#* APIv1 removal
#* Model updates from the driver

'''Scheduled for 19 May, 2016, 15:00 UTC'''
# Topics not covered at design summit https://etherpad.openstack.org/p/newton-manila-contributor-meetup
#* UI Customization
# Share Backup(zhongjun)
# Snapshot Restore (cknight)
# Specs (ganso)

'''Scheduled for 12 May, 2016, 15:00 UTC'''
# Driver private storage: https://review.openstack.org/#/c/315346/ (xyang)
#* Admin wants to check driver private storage before doing some actions on a resource, i.e., force delete a share or unmanage a share.
#* The patch proposes to add admin API to retrieve driver private storage data. Alternative approach proposed by Valeriy to use manila-manage (i.e., manila-manage driver-private-storage <uuid of a resource>). Which one should we choose?
# Topics not covered at design summit https://etherpad.openstack.org/p/newton-manila-contributor-meetup
#* Py3 functional tests
#* Replication with share networks
#* Service-check
#* Container driver

'''Scheduled for 5 May, 2016, 15:00 UTC'''
# Topics not covered at design summit https://etherpad.openstack.org/p/newton-manila-contributor-meetup
#* Documentation update (dustins/gouthamr)
#* User messages
#* Experimental features in UI
#* Share/snapshot size mismatch
# BP Specs (tbarron)
#* Need to ensure that the repo doesn't become "a wasteland of partial ideas where hope goes to die." (See http://lists.openstack.org/pipermail/openstack-dev/2016-May/094026.html)

'''Scheduled for 14 Apr, 2016, 15:00 UTC'''
# Manage API retrying (gouthamr/nidhimittalhada)
# Tempest test structuring - resource sharing and cleanup issues (akerr)
# Snapshot support common capability in the documentation (gouthamr)

'''Scheduled for 7 Apr, 2016, 15:00 UTC'''
# Access rules (bswartz)
# Design summit planning (bswartz)
#* Link: https://etherpad.openstack.org/p/manila-newton-summit-topics
# Release notes, continued (gouthamr)
#* Link: https://review.openstack.org/#/c/300656/
# Midcycle meetup
# Access Groups
#* https://wiki.openstack.org/wiki/Manila/design/access_groups
# Revert to snapshot

'''Scheduled for 31 Mar, 2016, 15:00 UTC'''
# Mitaka release status
#* RC1 expected to be the Mitaka release build
#* Last day for a re-spin is today
#** http://lists.openstack.org/pipermail/openstack-dev/2016-March/090877.html
# Design summit planning
#* Final schedule confirms 2 fishbowls, 4 working sessions, 1 half-day meetup
#** http://lists.openstack.org/pipermail/openstack-dev/2016-March/091032.html
#* Voting open now for summit topics
#** https://etherpad.openstack.org/p/manila-newton-summit-topics
# Manila Documentation (gouthamr)
#* Where do our docs live? how do we maintain them? (In Tree: devref, adminref | out of tree: install-guide, config-ref, cloud-admin guide)
#* Need a docs liaison
#** https://wiki.openstack.org/wiki/CrossProjectLiaisons#Documentation
#* Release Notes
#** http://docs.openstack.org/developer/reno/usage.html
# Managing/unmanaging replicated shares and snapshots (gouthamr)

'''Scheduled for 24 Mar, 2016, 15:00 UTC'''
# Design summit planning
#* https://etherpad.openstack.org/p/manila-newton-summit-topics
# Question about share servers (Jay.Xu)
#* Is there a way to get gateway IP address when the API setup_server() is invoked. I did not see this attribute in the input parameter network_info. Gateway IP is mandatory to create the interface for EMC storage which will be release coming soon, so it is expected that we can add the gateway IP into NetworkAllocation model if not.

'''Scheduled for 10 Mar, 2016, 15:00 UTC'''
# Common Capabilities Matrix (dustins)
#* https://review.openstack.org/#/c/288504/
#* I'm adding a table in the Developer Reference documentation to show which common capabilities are supported by which backends and when that support started.
#* I've got most drivers covered now.
#* If you want me to add your capabilities, leave a review on the patch and I'll update the table!

'''Scheduled for 25 Feb, 2016, 15:00 UTC'''

# OpenStack bug smash event (toabctl, mkoderer)
## link: https://etherpad.openstack.org/p/OpenStack-Bug-Smash-Mitaka
# Status of new drivers
## M.3 deadline is a week away.
## Only a few things have merged in the last couple days. At this rate, many things won't make it.
## Zuul is busy and will only get worse until the deadline.
## Multiple Manila gate breakages have occurred recently, and more are likely.
## The Manila tests are very unreliable at concurrency > 1.
## The generic driver is too slow to pass the gate at concurrency = 1.
## The new drivers *are* fast enough to run at concurrency = 1.
## The new drivers may not be 100% done, but they are done enough to pass reliably at concurrency = 1.
## We should switch voting to the new drivers NOW, or start throwing stuff overboard for the release.
# Next Manila mid-cycle location
## Proposal is to do it in Germany
## Need to know who could attend at that location.
## Would it make a difference if the Cinder mid-cycle happens in Dublin, and the Manila meeting is immediately before or after the Cinder one?

'''Scheduled for 18 Feb, 2016, 15:00 UTC'''

# Feature Proposal Freeze (bswartz)
# Open Discussion

'''Scheduled for 11 Feb, 2016, 15:00 UTC'''

# Hierarchical port binding support (mkoderer)
#* https://wiki.openstack.org/wiki/Manila/design/manila-newton-hpb-support
#* https://blueprints.launchpad.net/manila/+spec/manila-hpb-support
#* https://review.openstack.org/#/c/277731/
# Next Manila mid-cycle (mkoderer)
#* https://etherpad.openstack.org/p/manila-n-release-midcycle
# Replication + ZFS Driver (bswartz)
#* https://review.openstack.org/#/c/238572
# Driver FPF (bswartz)
#* https://review.openstack.org/#/c/279090/
#* https://review.openstack.org/#/c/278169/
# HDFS CI (bswartz)
'''Scheduled for 4 Feb, 2016, 15:00 UTC'''
# Data service handling protocol mount commands (ganso)
#* https://etherpad.openstack.org/p/manila-data-service-mount-meeting
# Separate process for copy on data service (tpsilva)
# ZFS driver (bswartz)
# First party drivers share migration support cooperation (ganso)

'''Scheduled for 28 Jan, 2016, 15:00 UTC'''
# Hierarchical port binding support (mkoderer)
#* https://etherpad.openstack.org/p/manila-hierarchical-port-binding
# Core reviewer team and other manila-related projects (bswartz)
# Replication status (bswartz)

'''Scheduled for 21 Jan, 2016, 15:00 UTC'''
# Update_access backwards compatibility approach discussion. (ganso)
# Exposing the Data Service to drivers. (ganso)
# Snapshot manage/unmanage (bswartz)

'''Scheduled for 17 Dec, 2015, 15:00 UTC'''
# Proposal for share groups (cknight)
# Liberty.1 stable release (bswartz)

'''Scheduled for 10 Dec, 2015, 15:00 UTC'''
# The most of the driver already report the "'QoS_support': False" to scheduler. Does we need to remove it or save it as Qos support flat? and write 'QoS_support' to capabilities_and_extra_specs.rst?(zhongjun)
##https://github.com/openstack/manila/blob/master/manila/share/drivers/hpe/hpe_3par_driver.py#L391
##https://github.com/openstack/manila/blob/master/manila/share/drivers/hitachi/hds_hnas.py#L354
##https://github.com/openstack/manila/blob/master/manila/share/drivers/netapp/dataontap/cluster_mode/lib_base.py#L257
##https://github.com/openstack/manila/blob/master/manila/scheduler/host_manager.py#L111
##https://review.openstack.org/#/c/247286/4/doc/source/devref/capabilities_and_extra_specs.rst
# Architectural concerns and interoperability of new features (cknight)

'''Scheduled for 3 Dec, 2015, 15:00 UTC'''
# Midcycle meetup
# Add "data loss" flag in DB migrations which lead to losing of existing data and update "manila-manage db sync" command to forbid applying such migrations without "--force" flag (u_glide)
# Reorganization of Manila scheduler to merge oslo-incubator code and simplify future maintenance (cknight)
## https://review.openstack.org/#/c/252060/
## Mostly for awareness, this is now merged and should be backwards compatible.
## Cinder could benefit from a similar treatment. And with the incubator gone, we could explore a common library with the Cinder team, since the heretofore shared components are certain to diverge over time, but I'm not sure how receptive they would be. Xing?
# CI reliability
## Why should we merge patches for drivers whose CI systems consistently fail? This negates the value of CI. I think that accepting patches that aren't proven by CI does a disservice to everyone, most of all to the driver owners themselves. Refusing to merge driver patches without a good CI result for that driver seems like a gentler and more uniform-over-time approach than threatening to throw out a driver near the end of a release. And why make our illustrious and overworked PTL be the CI policeman when all core reviewers could simply glance at the corresponding CI result before giving +A? (cknight)
## http://ci-watch.tintri.com/project?project=manila
## https://review.openstack.org/#/c/232293/
## https://review.openstack.org/#/c/235522/
## https://review.openstack.org/#/c/235726/
## https://review.openstack.org/#/c/249999/
## https://review.openstack.org/#/c/237454/

'''Scheduled for 19 Nov, 2015, 15:00 UTC'''
# GB --> GiB (markstur)
# OpenStack Manuals and Devref (markstur)
#* Config Ref (and others) are moving to .RST!
#* How should we document prior releases in devref?
# Midcycle Meetup Date (bswartz)
# Deadlines for features (bswartz)
#* Feature proposal freeze vs Big Features
#* New drivers
# Data Copy Service network approach discussion (ganso)
#* https://drive.google.com/file/d/0B30HPZmaePCHMlAzcjJlYm5rRTA/view?usp=sharing

'''Scheduled for 12 Nov, 2015, 15:00 UTC'''
# Which method is better for Manila QoS(independent way or just use extra specs ) (zhongjun)
#* link:http://paste.openstack.org/show/477677/
#*If use extra specs to config QoS, Does we first need add “QoS_support=true/false” in share_type. Does we need to add QoS_support flag in common capabilities
#**https://github.com/openstack/manila/blob/master/doc/source/devref/capabilities_and_extra_specs.rst)?
#** eg: share_type: qos_support='<is> True' or '<is> False'
#** netapp:max_iops=100 or qos:netapp:max_iops=100 or qos:max_iops=100
#** driver update: qos_support=true
#** An alternative could be to have the driver report a range
# Manila DR update (ameade)
#* Still no reviews on https://review.openstack.org/#/c/238572/
#* First-party driver implementation? Which driver, current generic or new solution? When do we need this implemented and who wants to do it?
#* Replication_change share state - This is the state of the share when any of it's replicas are being promoted. Is 'replication_change' good or is there a better phrase?
# Manila Driver minimum requirements document update (ganso)
# Manila Data Copy service name (bswartz)
#*https://review.openstack.org/#/c/244286/
# Upcoming change for Manila CI hooks (vponomaryov)
#*https://review.openstack.org/#/c/243233/
# Manilaclient enhancement to provide request_id when set http_log_debug is True. It will co-ordinate the request and its id, so it will be helpful to filter the log and the failed request. (Jay Xu)
#*http://paste.openstack.org/show/478675/

'''Scheduled for 5 Nov, 2015, 15:00 UTC'''
# Decide on read-only access rules as a required feature
# Manila DR - Design/Code feedback (gouthamr)
#* [https://wiki.openstack.org/wiki/Manila/design/manila-mitaka-data-replication Design Document]
#* [https://review.openstack.org/#/c/238572/ Core Implementation on Gerrit]
#* [https://review.openstack.org/#/c/235448/ Client Implementation on Gerrit]

'''Scheduled for 1 Oct, 2015, 15:00 UTC'''
# Liberty RC2 status
#* https://launchpad.net/manila/+milestone/liberty-rc2
#* https://review.openstack.org/#/q/project:openstack/manila+branch:stable/liberty,n,z
# Mitaka Design Summit
#* https://etherpad.openstack.org/p/manila-mitaka-summit-topics
# Vendor driver docs
#* https://github.com/openstack/manila/blob/master/doc/source/devref/share_back_ends_feature_support_mapping.rst
# Manila Client Nova Style Microversion Support (cfouts)
#* https://review.openstack.org/#/c/229142/
#* http://docs.openstack.org/developer/manila/devref/api_microversion_dev.html

'''Scheduled for 24 Sep, 2015, 15:00 UTC'''
# 3rd-party CI status
#* http://ec2-54-67-102-119.us-west-1.compute.amazonaws.com:5000/?project=openstack/manila&user=&timeframe=72&start=&end=&page_size=500
#* http://ci-watch.tintri.com/project?project=manila
# Liberty RC status
# Support for QoS in Manila (zhongjun2)
#* https://blueprints.launchpad.net/manila/+spec/manila-support-qos
#* https://wiki.openstack.org/wiki/Manila/QoS
# Reminder for backend driver maintainers to update feature support table (toabctl)
#* http://docs.openstack.org/developer/manila/devref/share_back_ends_feature_support_mapping.html
# Regression when using NetApp - license issue with netapp_lib (toabctl)
#* https://bugs.launchpad.net/cinder/+bug/1499334
# Mitaka Design Summit
#* Propose topics - https://etherpad.openstack.org/p/manila-mitaka-summit-topics
# Vendor driver docs
# Open Discussion

Manila/Meetings

2016-08-24T16:32:58Z

Clinton Knight: /* Next meeting */

= Weekly Manila team meeting =
'''NOTE MEETING TIME: Thursday at 15:00 UTC'''

If you're interested in management of shared filesystems for OpenStack, we have a weekly meetings in <code><nowiki>#openstack-meeting-alt</nowiki></code>, on Thursdays at 15:00 UTC. Please feel free to add items to the agenda below. NOTE: When adding topics please include your IRC name so we know who's topic it is and how to get more info.

'''Previous meeting logs can be found here:''' http://eavesdrop.openstack.org/meetings/manila/

== Next meeting ==
'''NOTE:''' ''Include your IRC nickname next to agenda items so that you can be called upon in the meeting and arrive at the meeting promptly if placing items in agenda. You might want to put this on your calendar if you are adding items.''

'''Scheduled for 25 Aug, 2016, 15:00 UTC'''
# Huawei share replication and CI
# IPv6 (cknight)
# Open Discussion

== Previous meetings ==

'''Scheduled for 18 Aug, 2016, 15:00 UTC'''
# Driver private data admin API
#* Disagreement abounds on how this API is to be designed, tested, and presented.
#* https://review.openstack.org/#/c/315346/
# Scenario tests are broken
#* http://logs.openstack.org/86/309286/34/check/gate-manila-tempest-dsvm-scenario/610f96e/console.html
# Revert to snapshot
#* https://review.openstack.org/#/c/340502/
#* https://review.openstack.org/#/c/356682/

"Scheduled for 11 Aug, 2016, 15:00 UTC"
# None

'''Scheduled for 4 Aug, 2016, 15:00 UTC'''
# Migration APIs
#* http://lists.openstack.org/pipermail/openstack-dev/2016-August/100682.html
# Correct approach for avoiding races

'''Scheduled for 28 Jul, 2016, 15:00 UTC'''
# Stepping Down as Docs Liaison (dustins)
#* Current job responsibilities don't give me the time to devote to this that I'd like to have, so it should go to someone else with a passion for documentation and has the willingness and desire for taking up the mantle.
# N-3 Review focus
#* https://etherpad.openstack.org/p/manila-newton3-priorities

'''Scheduled for 21 Jul, 2016, 15:00 UTC'''
# Migrate API reference into tree (dgonzalez)
#* https://review.openstack.org/#/c/313874/

'''Scheduled for 14 Jul, 2016, 15:00 UTC'''
# N-2 milestone
# Managing initial security (of the root directory) for new shares
# Mascot/Logo discussion
# Tempest alternatives

'''Scheduled for 30 Jun, 2016, 15:00 UTC'''
# non-disruptive nfs-ganesha export modifications (rraja/tbarron)
#* https://sourceforge.net/p/nfs-ganesha/mailman/message/35173839
# share backup spec discussion (zhongjun)
#* https://review.openstack.org/#/c/330306

'''Scheduled for 23 Jun, 2016, 15:00 UTC'''
# Implement extend and shrink features for glusterfs driver (DaiDV)
# Do all drivers have to inherit from driver.ShareDriver? (mkoderer)
# Creating a second share instance for driver-assisted migration (ganso)

'''Scheduled for 16 Jun, 2016, 15:00 UTC'''
# A share support multiple protocols (zhongjun)
#* https://review.openstack.org/#/c/329392/
# Drivers with minimum size for shares (tpsilva)

'''Scheduled for 9 Jun, 2016, 15:00 UTC'''
# HPB feature (mkoderer)
# the fate of nova-network support in manila (gouthamr)
# multi-AZ tests in the gate

'''Scheduled for 2 Jun, 2016, 15:00 UTC'''
# N-1 milestone (bswartz)
# Decision on APIv1 removal for newton (bswartz)
# Model updates from drivers (bswartz)
# Midcycle meetup
# Spec update
#* New spec - Extending share networks to span subnets - https://review.openstack.org/#/c/323646/ (gouthamr)

'''Scheduled for 26 May, 2016, 15:00 UTC'''
# N-1 milestone
# Update access remaining work
# Topics not covered at design summit https://etherpad.openstack.org/p/newton-manila-contributor-meetup
#* Nova-network removal
#* APIv1 removal
#* Model updates from the driver

'''Scheduled for 19 May, 2016, 15:00 UTC'''
# Topics not covered at design summit https://etherpad.openstack.org/p/newton-manila-contributor-meetup
#* UI Customization
# Share Backup(zhongjun)
# Snapshot Restore (cknight)
# Specs (ganso)

'''Scheduled for 12 May, 2016, 15:00 UTC'''
# Driver private storage: https://review.openstack.org/#/c/315346/ (xyang)
#* Admin wants to check driver private storage before doing some actions on a resource, i.e., force delete a share or unmanage a share.
#* The patch proposes to add admin API to retrieve driver private storage data. Alternative approach proposed by Valeriy to use manila-manage (i.e., manila-manage driver-private-storage <uuid of a resource>). Which one should we choose?
# Topics not covered at design summit https://etherpad.openstack.org/p/newton-manila-contributor-meetup
#* Py3 functional tests
#* Replication with share networks
#* Service-check
#* Container driver

'''Scheduled for 5 May, 2016, 15:00 UTC'''
# Topics not covered at design summit https://etherpad.openstack.org/p/newton-manila-contributor-meetup
#* Documentation update (dustins/gouthamr)
#* User messages
#* Experimental features in UI
#* Share/snapshot size mismatch
# BP Specs (tbarron)
#* Need to ensure that the repo doesn't become "a wasteland of partial ideas where hope goes to die." (See http://lists.openstack.org/pipermail/openstack-dev/2016-May/094026.html)

'''Scheduled for 14 Apr, 2016, 15:00 UTC'''
# Manage API retrying (gouthamr/nidhimittalhada)
# Tempest test structuring - resource sharing and cleanup issues (akerr)
# Snapshot support common capability in the documentation (gouthamr)

'''Scheduled for 7 Apr, 2016, 15:00 UTC'''
# Access rules (bswartz)
# Design summit planning (bswartz)
#* Link: https://etherpad.openstack.org/p/manila-newton-summit-topics
# Release notes, continued (gouthamr)
#* Link: https://review.openstack.org/#/c/300656/
# Midcycle meetup
# Access Groups
#* https://wiki.openstack.org/wiki/Manila/design/access_groups
# Revert to snapshot

'''Scheduled for 31 Mar, 2016, 15:00 UTC'''
# Mitaka release status
#* RC1 expected to be the Mitaka release build
#* Last day for a re-spin is today
#** http://lists.openstack.org/pipermail/openstack-dev/2016-March/090877.html
# Design summit planning
#* Final schedule confirms 2 fishbowls, 4 working sessions, 1 half-day meetup
#** http://lists.openstack.org/pipermail/openstack-dev/2016-March/091032.html
#* Voting open now for summit topics
#** https://etherpad.openstack.org/p/manila-newton-summit-topics
# Manila Documentation (gouthamr)
#* Where do our docs live? how do we maintain them? (In Tree: devref, adminref | out of tree: install-guide, config-ref, cloud-admin guide)
#* Need a docs liaison
#** https://wiki.openstack.org/wiki/CrossProjectLiaisons#Documentation
#* Release Notes
#** http://docs.openstack.org/developer/reno/usage.html
# Managing/unmanaging replicated shares and snapshots (gouthamr)

'''Scheduled for 24 Mar, 2016, 15:00 UTC'''
# Design summit planning
#* https://etherpad.openstack.org/p/manila-newton-summit-topics
# Question about share servers (Jay.Xu)
#* Is there a way to get gateway IP address when the API setup_server() is invoked. I did not see this attribute in the input parameter network_info. Gateway IP is mandatory to create the interface for EMC storage which will be release coming soon, so it is expected that we can add the gateway IP into NetworkAllocation model if not.

'''Scheduled for 10 Mar, 2016, 15:00 UTC'''
# Common Capabilities Matrix (dustins)
#* https://review.openstack.org/#/c/288504/
#* I'm adding a table in the Developer Reference documentation to show which common capabilities are supported by which backends and when that support started.
#* I've got most drivers covered now.
#* If you want me to add your capabilities, leave a review on the patch and I'll update the table!

'''Scheduled for 25 Feb, 2016, 15:00 UTC'''

# OpenStack bug smash event (toabctl, mkoderer)
## link: https://etherpad.openstack.org/p/OpenStack-Bug-Smash-Mitaka
# Status of new drivers
## M.3 deadline is a week away.
## Only a few things have merged in the last couple days. At this rate, many things won't make it.
## Zuul is busy and will only get worse until the deadline.
## Multiple Manila gate breakages have occurred recently, and more are likely.
## The Manila tests are very unreliable at concurrency > 1.
## The generic driver is too slow to pass the gate at concurrency = 1.
## The new drivers *are* fast enough to run at concurrency = 1.
## The new drivers may not be 100% done, but they are done enough to pass reliably at concurrency = 1.
## We should switch voting to the new drivers NOW, or start throwing stuff overboard for the release.
# Next Manila mid-cycle location
## Proposal is to do it in Germany
## Need to know who could attend at that location.
## Would it make a difference if the Cinder mid-cycle happens in Dublin, and the Manila meeting is immediately before or after the Cinder one?

'''Scheduled for 18 Feb, 2016, 15:00 UTC'''

# Feature Proposal Freeze (bswartz)
# Open Discussion

'''Scheduled for 11 Feb, 2016, 15:00 UTC'''

# Hierarchical port binding support (mkoderer)
#* https://wiki.openstack.org/wiki/Manila/design/manila-newton-hpb-support
#* https://blueprints.launchpad.net/manila/+spec/manila-hpb-support
#* https://review.openstack.org/#/c/277731/
# Next Manila mid-cycle (mkoderer)
#* https://etherpad.openstack.org/p/manila-n-release-midcycle
# Replication + ZFS Driver (bswartz)
#* https://review.openstack.org/#/c/238572
# Driver FPF (bswartz)
#* https://review.openstack.org/#/c/279090/
#* https://review.openstack.org/#/c/278169/
# HDFS CI (bswartz)
'''Scheduled for 4 Feb, 2016, 15:00 UTC'''
# Data service handling protocol mount commands (ganso)
#* https://etherpad.openstack.org/p/manila-data-service-mount-meeting
# Separate process for copy on data service (tpsilva)
# ZFS driver (bswartz)
# First party drivers share migration support cooperation (ganso)

'''Scheduled for 28 Jan, 2016, 15:00 UTC'''
# Hierarchical port binding support (mkoderer)
#* https://etherpad.openstack.org/p/manila-hierarchical-port-binding
# Core reviewer team and other manila-related projects (bswartz)
# Replication status (bswartz)

'''Scheduled for 21 Jan, 2016, 15:00 UTC'''
# Update_access backwards compatibility approach discussion. (ganso)
# Exposing the Data Service to drivers. (ganso)
# Snapshot manage/unmanage (bswartz)

'''Scheduled for 17 Dec, 2015, 15:00 UTC'''
# Proposal for share groups (cknight)
# Liberty.1 stable release (bswartz)

'''Scheduled for 10 Dec, 2015, 15:00 UTC'''
# The most of the driver already report the "'QoS_support': False" to scheduler. Does we need to remove it or save it as Qos support flat? and write 'QoS_support' to capabilities_and_extra_specs.rst?(zhongjun)
##https://github.com/openstack/manila/blob/master/manila/share/drivers/hpe/hpe_3par_driver.py#L391
##https://github.com/openstack/manila/blob/master/manila/share/drivers/hitachi/hds_hnas.py#L354
##https://github.com/openstack/manila/blob/master/manila/share/drivers/netapp/dataontap/cluster_mode/lib_base.py#L257
##https://github.com/openstack/manila/blob/master/manila/scheduler/host_manager.py#L111
##https://review.openstack.org/#/c/247286/4/doc/source/devref/capabilities_and_extra_specs.rst
# Architectural concerns and interoperability of new features (cknight)

'''Scheduled for 3 Dec, 2015, 15:00 UTC'''
# Midcycle meetup
# Add "data loss" flag in DB migrations which lead to losing of existing data and update "manila-manage db sync" command to forbid applying such migrations without "--force" flag (u_glide)
# Reorganization of Manila scheduler to merge oslo-incubator code and simplify future maintenance (cknight)
## https://review.openstack.org/#/c/252060/
## Mostly for awareness, this is now merged and should be backwards compatible.
## Cinder could benefit from a similar treatment. And with the incubator gone, we could explore a common library with the Cinder team, since the heretofore shared components are certain to diverge over time, but I'm not sure how receptive they would be. Xing?
# CI reliability
## Why should we merge patches for drivers whose CI systems consistently fail? This negates the value of CI. I think that accepting patches that aren't proven by CI does a disservice to everyone, most of all to the driver owners themselves. Refusing to merge driver patches without a good CI result for that driver seems like a gentler and more uniform-over-time approach than threatening to throw out a driver near the end of a release. And why make our illustrious and overworked PTL be the CI policeman when all core reviewers could simply glance at the corresponding CI result before giving +A? (cknight)
## http://ci-watch.tintri.com/project?project=manila
## https://review.openstack.org/#/c/232293/
## https://review.openstack.org/#/c/235522/
## https://review.openstack.org/#/c/235726/
## https://review.openstack.org/#/c/249999/
## https://review.openstack.org/#/c/237454/

'''Scheduled for 19 Nov, 2015, 15:00 UTC'''
# GB --> GiB (markstur)
# OpenStack Manuals and Devref (markstur)
#* Config Ref (and others) are moving to .RST!
#* How should we document prior releases in devref?
# Midcycle Meetup Date (bswartz)
# Deadlines for features (bswartz)
#* Feature proposal freeze vs Big Features
#* New drivers
# Data Copy Service network approach discussion (ganso)
#* https://drive.google.com/file/d/0B30HPZmaePCHMlAzcjJlYm5rRTA/view?usp=sharing

'''Scheduled for 12 Nov, 2015, 15:00 UTC'''
# Which method is better for Manila QoS(independent way or just use extra specs ) (zhongjun)
#* link:http://paste.openstack.org/show/477677/
#*If use extra specs to config QoS, Does we first need add “QoS_support=true/false” in share_type. Does we need to add QoS_support flag in common capabilities
#**https://github.com/openstack/manila/blob/master/doc/source/devref/capabilities_and_extra_specs.rst)?
#** eg: share_type: qos_support='<is> True' or '<is> False'
#** netapp:max_iops=100 or qos:netapp:max_iops=100 or qos:max_iops=100
#** driver update: qos_support=true
#** An alternative could be to have the driver report a range
# Manila DR update (ameade)
#* Still no reviews on https://review.openstack.org/#/c/238572/
#* First-party driver implementation? Which driver, current generic or new solution? When do we need this implemented and who wants to do it?
#* Replication_change share state - This is the state of the share when any of it's replicas are being promoted. Is 'replication_change' good or is there a better phrase?
# Manila Driver minimum requirements document update (ganso)
# Manila Data Copy service name (bswartz)
#*https://review.openstack.org/#/c/244286/
# Upcoming change for Manila CI hooks (vponomaryov)
#*https://review.openstack.org/#/c/243233/
# Manilaclient enhancement to provide request_id when set http_log_debug is True. It will co-ordinate the request and its id, so it will be helpful to filter the log and the failed request. (Jay Xu)
#*http://paste.openstack.org/show/478675/

'''Scheduled for 5 Nov, 2015, 15:00 UTC'''
# Decide on read-only access rules as a required feature
# Manila DR - Design/Code feedback (gouthamr)
#* [https://wiki.openstack.org/wiki/Manila/design/manila-mitaka-data-replication Design Document]
#* [https://review.openstack.org/#/c/238572/ Core Implementation on Gerrit]
#* [https://review.openstack.org/#/c/235448/ Client Implementation on Gerrit]

'''Scheduled for 1 Oct, 2015, 15:00 UTC'''
# Liberty RC2 status
#* https://launchpad.net/manila/+milestone/liberty-rc2
#* https://review.openstack.org/#/q/project:openstack/manila+branch:stable/liberty,n,z
# Mitaka Design Summit
#* https://etherpad.openstack.org/p/manila-mitaka-summit-topics
# Vendor driver docs
#* https://github.com/openstack/manila/blob/master/doc/source/devref/share_back_ends_feature_support_mapping.rst
# Manila Client Nova Style Microversion Support (cfouts)
#* https://review.openstack.org/#/c/229142/
#* http://docs.openstack.org/developer/manila/devref/api_microversion_dev.html

'''Scheduled for 24 Sep, 2015, 15:00 UTC'''
# 3rd-party CI status
#* http://ec2-54-67-102-119.us-west-1.compute.amazonaws.com:5000/?project=openstack/manila&user=&timeframe=72&start=&end=&page_size=500
#* http://ci-watch.tintri.com/project?project=manila
# Liberty RC status
# Support for QoS in Manila (zhongjun2)
#* https://blueprints.launchpad.net/manila/+spec/manila-support-qos
#* https://wiki.openstack.org/wiki/Manila/QoS
# Reminder for backend driver maintainers to update feature support table (toabctl)
#* http://docs.openstack.org/developer/manila/devref/share_back_ends_feature_support_mapping.html
# Regression when using NetApp - license issue with netapp_lib (toabctl)
#* https://bugs.launchpad.net/cinder/+bug/1499334
# Mitaka Design Summit
#* Propose topics - https://etherpad.openstack.org/p/manila-mitaka-summit-topics
# Vendor driver docs
# Open Discussion

Manila/Meetings

2016-08-24T16:31:24Z

Clinton Knight: /* Next meeting */

= Weekly Manila team meeting =
'''NOTE MEETING TIME: Thursday at 15:00 UTC'''

If you're interested in management of shared filesystems for OpenStack, we have a weekly meetings in <code><nowiki>#openstack-meeting-alt</nowiki></code>, on Thursdays at 15:00 UTC. Please feel free to add items to the agenda below. NOTE: When adding topics please include your IRC name so we know who's topic it is and how to get more info.

'''Previous meeting logs can be found here:''' http://eavesdrop.openstack.org/meetings/manila/

== Next meeting ==
'''NOTE:''' ''Include your IRC nickname next to agenda items so that you can be called upon in the meeting and arrive at the meeting promptly if placing items in agenda. You might want to put this on your calendar if you are adding items.''

'''Scheduled for 25 Aug, 2016, 15:00 UTC'''
# Huawei share replication and CI
# IPv6
# Open Discussion

== Previous meetings ==

'''Scheduled for 18 Aug, 2016, 15:00 UTC'''
# Driver private data admin API
#* Disagreement abounds on how this API is to be designed, tested, and presented.
#* https://review.openstack.org/#/c/315346/
# Scenario tests are broken
#* http://logs.openstack.org/86/309286/34/check/gate-manila-tempest-dsvm-scenario/610f96e/console.html
# Revert to snapshot
#* https://review.openstack.org/#/c/340502/
#* https://review.openstack.org/#/c/356682/

"Scheduled for 11 Aug, 2016, 15:00 UTC"
# None

'''Scheduled for 4 Aug, 2016, 15:00 UTC'''
# Migration APIs
#* http://lists.openstack.org/pipermail/openstack-dev/2016-August/100682.html
# Correct approach for avoiding races

'''Scheduled for 28 Jul, 2016, 15:00 UTC'''
# Stepping Down as Docs Liaison (dustins)
#* Current job responsibilities don't give me the time to devote to this that I'd like to have, so it should go to someone else with a passion for documentation and has the willingness and desire for taking up the mantle.
# N-3 Review focus
#* https://etherpad.openstack.org/p/manila-newton3-priorities

'''Scheduled for 21 Jul, 2016, 15:00 UTC'''
# Migrate API reference into tree (dgonzalez)
#* https://review.openstack.org/#/c/313874/

'''Scheduled for 14 Jul, 2016, 15:00 UTC'''
# N-2 milestone
# Managing initial security (of the root directory) for new shares
# Mascot/Logo discussion
# Tempest alternatives

'''Scheduled for 30 Jun, 2016, 15:00 UTC'''
# non-disruptive nfs-ganesha export modifications (rraja/tbarron)
#* https://sourceforge.net/p/nfs-ganesha/mailman/message/35173839
# share backup spec discussion (zhongjun)
#* https://review.openstack.org/#/c/330306

'''Scheduled for 23 Jun, 2016, 15:00 UTC'''
# Implement extend and shrink features for glusterfs driver (DaiDV)
# Do all drivers have to inherit from driver.ShareDriver? (mkoderer)
# Creating a second share instance for driver-assisted migration (ganso)

'''Scheduled for 16 Jun, 2016, 15:00 UTC'''
# A share support multiple protocols (zhongjun)
#* https://review.openstack.org/#/c/329392/
# Drivers with minimum size for shares (tpsilva)

'''Scheduled for 9 Jun, 2016, 15:00 UTC'''
# HPB feature (mkoderer)
# the fate of nova-network support in manila (gouthamr)
# multi-AZ tests in the gate

'''Scheduled for 2 Jun, 2016, 15:00 UTC'''
# N-1 milestone (bswartz)
# Decision on APIv1 removal for newton (bswartz)
# Model updates from drivers (bswartz)
# Midcycle meetup
# Spec update
#* New spec - Extending share networks to span subnets - https://review.openstack.org/#/c/323646/ (gouthamr)

'''Scheduled for 26 May, 2016, 15:00 UTC'''
# N-1 milestone
# Update access remaining work
# Topics not covered at design summit https://etherpad.openstack.org/p/newton-manila-contributor-meetup
#* Nova-network removal
#* APIv1 removal
#* Model updates from the driver

'''Scheduled for 19 May, 2016, 15:00 UTC'''
# Topics not covered at design summit https://etherpad.openstack.org/p/newton-manila-contributor-meetup
#* UI Customization
# Share Backup(zhongjun)
# Snapshot Restore (cknight)
# Specs (ganso)

'''Scheduled for 12 May, 2016, 15:00 UTC'''
# Driver private storage: https://review.openstack.org/#/c/315346/ (xyang)
#* Admin wants to check driver private storage before doing some actions on a resource, i.e., force delete a share or unmanage a share.
#* The patch proposes to add admin API to retrieve driver private storage data. Alternative approach proposed by Valeriy to use manila-manage (i.e., manila-manage driver-private-storage <uuid of a resource>). Which one should we choose?
# Topics not covered at design summit https://etherpad.openstack.org/p/newton-manila-contributor-meetup
#* Py3 functional tests
#* Replication with share networks
#* Service-check
#* Container driver

'''Scheduled for 5 May, 2016, 15:00 UTC'''
# Topics not covered at design summit https://etherpad.openstack.org/p/newton-manila-contributor-meetup
#* Documentation update (dustins/gouthamr)
#* User messages
#* Experimental features in UI
#* Share/snapshot size mismatch
# BP Specs (tbarron)
#* Need to ensure that the repo doesn't become "a wasteland of partial ideas where hope goes to die." (See http://lists.openstack.org/pipermail/openstack-dev/2016-May/094026.html)

'''Scheduled for 14 Apr, 2016, 15:00 UTC'''
# Manage API retrying (gouthamr/nidhimittalhada)
# Tempest test structuring - resource sharing and cleanup issues (akerr)
# Snapshot support common capability in the documentation (gouthamr)

'''Scheduled for 7 Apr, 2016, 15:00 UTC'''
# Access rules (bswartz)
# Design summit planning (bswartz)
#* Link: https://etherpad.openstack.org/p/manila-newton-summit-topics
# Release notes, continued (gouthamr)
#* Link: https://review.openstack.org/#/c/300656/
# Midcycle meetup
# Access Groups
#* https://wiki.openstack.org/wiki/Manila/design/access_groups
# Revert to snapshot

'''Scheduled for 31 Mar, 2016, 15:00 UTC'''
# Mitaka release status
#* RC1 expected to be the Mitaka release build
#* Last day for a re-spin is today
#** http://lists.openstack.org/pipermail/openstack-dev/2016-March/090877.html
# Design summit planning
#* Final schedule confirms 2 fishbowls, 4 working sessions, 1 half-day meetup
#** http://lists.openstack.org/pipermail/openstack-dev/2016-March/091032.html
#* Voting open now for summit topics
#** https://etherpad.openstack.org/p/manila-newton-summit-topics
# Manila Documentation (gouthamr)
#* Where do our docs live? how do we maintain them? (In Tree: devref, adminref | out of tree: install-guide, config-ref, cloud-admin guide)
#* Need a docs liaison
#** https://wiki.openstack.org/wiki/CrossProjectLiaisons#Documentation
#* Release Notes
#** http://docs.openstack.org/developer/reno/usage.html
# Managing/unmanaging replicated shares and snapshots (gouthamr)

'''Scheduled for 24 Mar, 2016, 15:00 UTC'''
# Design summit planning
#* https://etherpad.openstack.org/p/manila-newton-summit-topics
# Question about share servers (Jay.Xu)
#* Is there a way to get gateway IP address when the API setup_server() is invoked. I did not see this attribute in the input parameter network_info. Gateway IP is mandatory to create the interface for EMC storage which will be release coming soon, so it is expected that we can add the gateway IP into NetworkAllocation model if not.

'''Scheduled for 10 Mar, 2016, 15:00 UTC'''
# Common Capabilities Matrix (dustins)
#* https://review.openstack.org/#/c/288504/
#* I'm adding a table in the Developer Reference documentation to show which common capabilities are supported by which backends and when that support started.
#* I've got most drivers covered now.
#* If you want me to add your capabilities, leave a review on the patch and I'll update the table!

'''Scheduled for 25 Feb, 2016, 15:00 UTC'''

# OpenStack bug smash event (toabctl, mkoderer)
## link: https://etherpad.openstack.org/p/OpenStack-Bug-Smash-Mitaka
# Status of new drivers
## M.3 deadline is a week away.
## Only a few things have merged in the last couple days. At this rate, many things won't make it.
## Zuul is busy and will only get worse until the deadline.
## Multiple Manila gate breakages have occurred recently, and more are likely.
## The Manila tests are very unreliable at concurrency > 1.
## The generic driver is too slow to pass the gate at concurrency = 1.
## The new drivers *are* fast enough to run at concurrency = 1.
## The new drivers may not be 100% done, but they are done enough to pass reliably at concurrency = 1.
## We should switch voting to the new drivers NOW, or start throwing stuff overboard for the release.
# Next Manila mid-cycle location
## Proposal is to do it in Germany
## Need to know who could attend at that location.
## Would it make a difference if the Cinder mid-cycle happens in Dublin, and the Manila meeting is immediately before or after the Cinder one?

'''Scheduled for 18 Feb, 2016, 15:00 UTC'''

# Feature Proposal Freeze (bswartz)
# Open Discussion

'''Scheduled for 11 Feb, 2016, 15:00 UTC'''

# Hierarchical port binding support (mkoderer)
#* https://wiki.openstack.org/wiki/Manila/design/manila-newton-hpb-support
#* https://blueprints.launchpad.net/manila/+spec/manila-hpb-support
#* https://review.openstack.org/#/c/277731/
# Next Manila mid-cycle (mkoderer)
#* https://etherpad.openstack.org/p/manila-n-release-midcycle
# Replication + ZFS Driver (bswartz)
#* https://review.openstack.org/#/c/238572
# Driver FPF (bswartz)
#* https://review.openstack.org/#/c/279090/
#* https://review.openstack.org/#/c/278169/
# HDFS CI (bswartz)
'''Scheduled for 4 Feb, 2016, 15:00 UTC'''
# Data service handling protocol mount commands (ganso)
#* https://etherpad.openstack.org/p/manila-data-service-mount-meeting
# Separate process for copy on data service (tpsilva)
# ZFS driver (bswartz)
# First party drivers share migration support cooperation (ganso)

'''Scheduled for 28 Jan, 2016, 15:00 UTC'''
# Hierarchical port binding support (mkoderer)
#* https://etherpad.openstack.org/p/manila-hierarchical-port-binding
# Core reviewer team and other manila-related projects (bswartz)
# Replication status (bswartz)

'''Scheduled for 21 Jan, 2016, 15:00 UTC'''
# Update_access backwards compatibility approach discussion. (ganso)
# Exposing the Data Service to drivers. (ganso)
# Snapshot manage/unmanage (bswartz)

'''Scheduled for 17 Dec, 2015, 15:00 UTC'''
# Proposal for share groups (cknight)
# Liberty.1 stable release (bswartz)

'''Scheduled for 10 Dec, 2015, 15:00 UTC'''
# The most of the driver already report the "'QoS_support': False" to scheduler. Does we need to remove it or save it as Qos support flat? and write 'QoS_support' to capabilities_and_extra_specs.rst?(zhongjun)
##https://github.com/openstack/manila/blob/master/manila/share/drivers/hpe/hpe_3par_driver.py#L391
##https://github.com/openstack/manila/blob/master/manila/share/drivers/hitachi/hds_hnas.py#L354
##https://github.com/openstack/manila/blob/master/manila/share/drivers/netapp/dataontap/cluster_mode/lib_base.py#L257
##https://github.com/openstack/manila/blob/master/manila/scheduler/host_manager.py#L111
##https://review.openstack.org/#/c/247286/4/doc/source/devref/capabilities_and_extra_specs.rst
# Architectural concerns and interoperability of new features (cknight)

'''Scheduled for 3 Dec, 2015, 15:00 UTC'''
# Midcycle meetup
# Add "data loss" flag in DB migrations which lead to losing of existing data and update "manila-manage db sync" command to forbid applying such migrations without "--force" flag (u_glide)
# Reorganization of Manila scheduler to merge oslo-incubator code and simplify future maintenance (cknight)
## https://review.openstack.org/#/c/252060/
## Mostly for awareness, this is now merged and should be backwards compatible.
## Cinder could benefit from a similar treatment. And with the incubator gone, we could explore a common library with the Cinder team, since the heretofore shared components are certain to diverge over time, but I'm not sure how receptive they would be. Xing?
# CI reliability
## Why should we merge patches for drivers whose CI systems consistently fail? This negates the value of CI. I think that accepting patches that aren't proven by CI does a disservice to everyone, most of all to the driver owners themselves. Refusing to merge driver patches without a good CI result for that driver seems like a gentler and more uniform-over-time approach than threatening to throw out a driver near the end of a release. And why make our illustrious and overworked PTL be the CI policeman when all core reviewers could simply glance at the corresponding CI result before giving +A? (cknight)
## http://ci-watch.tintri.com/project?project=manila
## https://review.openstack.org/#/c/232293/
## https://review.openstack.org/#/c/235522/
## https://review.openstack.org/#/c/235726/
## https://review.openstack.org/#/c/249999/
## https://review.openstack.org/#/c/237454/

'''Scheduled for 19 Nov, 2015, 15:00 UTC'''
# GB --> GiB (markstur)
# OpenStack Manuals and Devref (markstur)
#* Config Ref (and others) are moving to .RST!
#* How should we document prior releases in devref?
# Midcycle Meetup Date (bswartz)
# Deadlines for features (bswartz)
#* Feature proposal freeze vs Big Features
#* New drivers
# Data Copy Service network approach discussion (ganso)
#* https://drive.google.com/file/d/0B30HPZmaePCHMlAzcjJlYm5rRTA/view?usp=sharing

'''Scheduled for 12 Nov, 2015, 15:00 UTC'''
# Which method is better for Manila QoS(independent way or just use extra specs ) (zhongjun)
#* link:http://paste.openstack.org/show/477677/
#*If use extra specs to config QoS, Does we first need add “QoS_support=true/false” in share_type. Does we need to add QoS_support flag in common capabilities
#**https://github.com/openstack/manila/blob/master/doc/source/devref/capabilities_and_extra_specs.rst)?
#** eg: share_type: qos_support='<is> True' or '<is> False'
#** netapp:max_iops=100 or qos:netapp:max_iops=100 or qos:max_iops=100
#** driver update: qos_support=true
#** An alternative could be to have the driver report a range
# Manila DR update (ameade)
#* Still no reviews on https://review.openstack.org/#/c/238572/
#* First-party driver implementation? Which driver, current generic or new solution? When do we need this implemented and who wants to do it?
#* Replication_change share state - This is the state of the share when any of it's replicas are being promoted. Is 'replication_change' good or is there a better phrase?
# Manila Driver minimum requirements document update (ganso)
# Manila Data Copy service name (bswartz)
#*https://review.openstack.org/#/c/244286/
# Upcoming change for Manila CI hooks (vponomaryov)
#*https://review.openstack.org/#/c/243233/
# Manilaclient enhancement to provide request_id when set http_log_debug is True. It will co-ordinate the request and its id, so it will be helpful to filter the log and the failed request. (Jay Xu)
#*http://paste.openstack.org/show/478675/

'''Scheduled for 5 Nov, 2015, 15:00 UTC'''
# Decide on read-only access rules as a required feature
# Manila DR - Design/Code feedback (gouthamr)
#* [https://wiki.openstack.org/wiki/Manila/design/manila-mitaka-data-replication Design Document]
#* [https://review.openstack.org/#/c/238572/ Core Implementation on Gerrit]
#* [https://review.openstack.org/#/c/235448/ Client Implementation on Gerrit]

'''Scheduled for 1 Oct, 2015, 15:00 UTC'''
# Liberty RC2 status
#* https://launchpad.net/manila/+milestone/liberty-rc2
#* https://review.openstack.org/#/q/project:openstack/manila+branch:stable/liberty,n,z
# Mitaka Design Summit
#* https://etherpad.openstack.org/p/manila-mitaka-summit-topics
# Vendor driver docs
#* https://github.com/openstack/manila/blob/master/doc/source/devref/share_back_ends_feature_support_mapping.rst
# Manila Client Nova Style Microversion Support (cfouts)
#* https://review.openstack.org/#/c/229142/
#* http://docs.openstack.org/developer/manila/devref/api_microversion_dev.html

'''Scheduled for 24 Sep, 2015, 15:00 UTC'''
# 3rd-party CI status
#* http://ec2-54-67-102-119.us-west-1.compute.amazonaws.com:5000/?project=openstack/manila&user=&timeframe=72&start=&end=&page_size=500
#* http://ci-watch.tintri.com/project?project=manila
# Liberty RC status
# Support for QoS in Manila (zhongjun2)
#* https://blueprints.launchpad.net/manila/+spec/manila-support-qos
#* https://wiki.openstack.org/wiki/Manila/QoS
# Reminder for backend driver maintainers to update feature support table (toabctl)
#* http://docs.openstack.org/developer/manila/devref/share_back_ends_feature_support_mapping.html
# Regression when using NetApp - license issue with netapp_lib (toabctl)
#* https://bugs.launchpad.net/cinder/+bug/1499334
# Mitaka Design Summit
#* Propose topics - https://etherpad.openstack.org/p/manila-mitaka-summit-topics
# Vendor driver docs
# Open Discussion

Manila/Meetings

2016-08-15T21:13:38Z

Clinton Knight: /* Next meeting */

= Weekly Manila team meeting =
'''NOTE MEETING TIME: Thursday at 15:00 UTC'''

If you're interested in management of shared filesystems for OpenStack, we have a weekly meetings in <code><nowiki>#openstack-meeting-alt</nowiki></code>, on Thursdays at 15:00 UTC. Please feel free to add items to the agenda below. NOTE: When adding topics please include your IRC name so we know who's topic it is and how to get more info.

'''Previous meeting logs can be found here:''' http://eavesdrop.openstack.org/meetings/manila/

== Next meeting ==
'''NOTE:''' ''Include your IRC nickname next to agenda items so that you can be called upon in the meeting and arrive at the meeting promptly if placing items in agenda. You might want to put this on your calendar if you are adding items.''

'''Scheduled for 18 Aug, 2016, 15:00 UTC'''
# Driver private data admin API
#* Disagreement abounds on how this API is to be designed, tested, and presented.
#* https://review.openstack.org/#/c/315346/
# Open Discussion

== Previous meetings ==

"Scheduled for 11 Aug, 2016, 15:00 UTC"
# None

'''Scheduled for 4 Aug, 2016, 15:00 UTC'''
# Migration APIs
#* http://lists.openstack.org/pipermail/openstack-dev/2016-August/100682.html
# Correct approach for avoiding races

'''Scheduled for 28 Jul, 2016, 15:00 UTC'''
# Stepping Down as Docs Liaison (dustins)
#* Current job responsibilities don't give me the time to devote to this that I'd like to have, so it should go to someone else with a passion for documentation and has the willingness and desire for taking up the mantle.
# N-3 Review focus
#* https://etherpad.openstack.org/p/manila-newton3-priorities

'''Scheduled for 21 Jul, 2016, 15:00 UTC'''
# Migrate API reference into tree (dgonzalez)
#* https://review.openstack.org/#/c/313874/

'''Scheduled for 14 Jul, 2016, 15:00 UTC'''
# N-2 milestone
# Managing initial security (of the root directory) for new shares
# Mascot/Logo discussion
# Tempest alternatives

'''Scheduled for 30 Jun, 2016, 15:00 UTC'''
# non-disruptive nfs-ganesha export modifications (rraja/tbarron)
#* https://sourceforge.net/p/nfs-ganesha/mailman/message/35173839
# share backup spec discussion (zhongjun)
#* https://review.openstack.org/#/c/330306

'''Scheduled for 23 Jun, 2016, 15:00 UTC'''
# Implement extend and shrink features for glusterfs driver (DaiDV)
# Do all drivers have to inherit from driver.ShareDriver? (mkoderer)
# Creating a second share instance for driver-assisted migration (ganso)

'''Scheduled for 16 Jun, 2016, 15:00 UTC'''
# A share support multiple protocols (zhongjun)
#* https://review.openstack.org/#/c/329392/
# Drivers with minimum size for shares (tpsilva)

'''Scheduled for 9 Jun, 2016, 15:00 UTC'''
# HPB feature (mkoderer)
# the fate of nova-network support in manila (gouthamr)
# multi-AZ tests in the gate

'''Scheduled for 2 Jun, 2016, 15:00 UTC'''
# N-1 milestone (bswartz)
# Decision on APIv1 removal for newton (bswartz)
# Model updates from drivers (bswartz)
# Midcycle meetup
# Spec update
#* New spec - Extending share networks to span subnets - https://review.openstack.org/#/c/323646/ (gouthamr)

'''Scheduled for 26 May, 2016, 15:00 UTC'''
# N-1 milestone
# Update access remaining work
# Topics not covered at design summit https://etherpad.openstack.org/p/newton-manila-contributor-meetup
#* Nova-network removal
#* APIv1 removal
#* Model updates from the driver

'''Scheduled for 19 May, 2016, 15:00 UTC'''
# Topics not covered at design summit https://etherpad.openstack.org/p/newton-manila-contributor-meetup
#* UI Customization
# Share Backup(zhongjun)
# Snapshot Restore (cknight)
# Specs (ganso)

'''Scheduled for 12 May, 2016, 15:00 UTC'''
# Driver private storage: https://review.openstack.org/#/c/315346/ (xyang)
#* Admin wants to check driver private storage before doing some actions on a resource, i.e., force delete a share or unmanage a share.
#* The patch proposes to add admin API to retrieve driver private storage data. Alternative approach proposed by Valeriy to use manila-manage (i.e., manila-manage driver-private-storage <uuid of a resource>). Which one should we choose?
# Topics not covered at design summit https://etherpad.openstack.org/p/newton-manila-contributor-meetup
#* Py3 functional tests
#* Replication with share networks
#* Service-check
#* Container driver

'''Scheduled for 5 May, 2016, 15:00 UTC'''
# Topics not covered at design summit https://etherpad.openstack.org/p/newton-manila-contributor-meetup
#* Documentation update (dustins/gouthamr)
#* User messages
#* Experimental features in UI
#* Share/snapshot size mismatch
# BP Specs (tbarron)
#* Need to ensure that the repo doesn't become "a wasteland of partial ideas where hope goes to die." (See http://lists.openstack.org/pipermail/openstack-dev/2016-May/094026.html)

'''Scheduled for 14 Apr, 2016, 15:00 UTC'''
# Manage API retrying (gouthamr/nidhimittalhada)
# Tempest test structuring - resource sharing and cleanup issues (akerr)
# Snapshot support common capability in the documentation (gouthamr)

'''Scheduled for 7 Apr, 2016, 15:00 UTC'''
# Access rules (bswartz)
# Design summit planning (bswartz)
#* Link: https://etherpad.openstack.org/p/manila-newton-summit-topics
# Release notes, continued (gouthamr)
#* Link: https://review.openstack.org/#/c/300656/
# Midcycle meetup
# Access Groups
#* https://wiki.openstack.org/wiki/Manila/design/access_groups
# Revert to snapshot

'''Scheduled for 31 Mar, 2016, 15:00 UTC'''
# Mitaka release status
#* RC1 expected to be the Mitaka release build
#* Last day for a re-spin is today
#** http://lists.openstack.org/pipermail/openstack-dev/2016-March/090877.html
# Design summit planning
#* Final schedule confirms 2 fishbowls, 4 working sessions, 1 half-day meetup
#** http://lists.openstack.org/pipermail/openstack-dev/2016-March/091032.html
#* Voting open now for summit topics
#** https://etherpad.openstack.org/p/manila-newton-summit-topics
# Manila Documentation (gouthamr)
#* Where do our docs live? how do we maintain them? (In Tree: devref, adminref | out of tree: install-guide, config-ref, cloud-admin guide)
#* Need a docs liaison
#** https://wiki.openstack.org/wiki/CrossProjectLiaisons#Documentation
#* Release Notes
#** http://docs.openstack.org/developer/reno/usage.html
# Managing/unmanaging replicated shares and snapshots (gouthamr)

'''Scheduled for 24 Mar, 2016, 15:00 UTC'''
# Design summit planning
#* https://etherpad.openstack.org/p/manila-newton-summit-topics
# Question about share servers (Jay.Xu)
#* Is there a way to get gateway IP address when the API setup_server() is invoked. I did not see this attribute in the input parameter network_info. Gateway IP is mandatory to create the interface for EMC storage which will be release coming soon, so it is expected that we can add the gateway IP into NetworkAllocation model if not.

'''Scheduled for 10 Mar, 2016, 15:00 UTC'''
# Common Capabilities Matrix (dustins)
#* https://review.openstack.org/#/c/288504/
#* I'm adding a table in the Developer Reference documentation to show which common capabilities are supported by which backends and when that support started.
#* I've got most drivers covered now.
#* If you want me to add your capabilities, leave a review on the patch and I'll update the table!

'''Scheduled for 25 Feb, 2016, 15:00 UTC'''

# OpenStack bug smash event (toabctl, mkoderer)
## link: https://etherpad.openstack.org/p/OpenStack-Bug-Smash-Mitaka
# Status of new drivers
## M.3 deadline is a week away.
## Only a few things have merged in the last couple days. At this rate, many things won't make it.
## Zuul is busy and will only get worse until the deadline.
## Multiple Manila gate breakages have occurred recently, and more are likely.
## The Manila tests are very unreliable at concurrency > 1.
## The generic driver is too slow to pass the gate at concurrency = 1.
## The new drivers *are* fast enough to run at concurrency = 1.
## The new drivers may not be 100% done, but they are done enough to pass reliably at concurrency = 1.
## We should switch voting to the new drivers NOW, or start throwing stuff overboard for the release.
# Next Manila mid-cycle location
## Proposal is to do it in Germany
## Need to know who could attend at that location.
## Would it make a difference if the Cinder mid-cycle happens in Dublin, and the Manila meeting is immediately before or after the Cinder one?

'''Scheduled for 18 Feb, 2016, 15:00 UTC'''

# Feature Proposal Freeze (bswartz)
# Open Discussion

'''Scheduled for 11 Feb, 2016, 15:00 UTC'''

# Hierarchical port binding support (mkoderer)
#* https://wiki.openstack.org/wiki/Manila/design/manila-newton-hpb-support
#* https://blueprints.launchpad.net/manila/+spec/manila-hpb-support
#* https://review.openstack.org/#/c/277731/
# Next Manila mid-cycle (mkoderer)
#* https://etherpad.openstack.org/p/manila-n-release-midcycle
# Replication + ZFS Driver (bswartz)
#* https://review.openstack.org/#/c/238572
# Driver FPF (bswartz)
#* https://review.openstack.org/#/c/279090/
#* https://review.openstack.org/#/c/278169/
# HDFS CI (bswartz)
'''Scheduled for 4 Feb, 2016, 15:00 UTC'''
# Data service handling protocol mount commands (ganso)
#* https://etherpad.openstack.org/p/manila-data-service-mount-meeting
# Separate process for copy on data service (tpsilva)
# ZFS driver (bswartz)
# First party drivers share migration support cooperation (ganso)

'''Scheduled for 28 Jan, 2016, 15:00 UTC'''
# Hierarchical port binding support (mkoderer)
#* https://etherpad.openstack.org/p/manila-hierarchical-port-binding
# Core reviewer team and other manila-related projects (bswartz)
# Replication status (bswartz)

'''Scheduled for 21 Jan, 2016, 15:00 UTC'''
# Update_access backwards compatibility approach discussion. (ganso)
# Exposing the Data Service to drivers. (ganso)
# Snapshot manage/unmanage (bswartz)

'''Scheduled for 17 Dec, 2015, 15:00 UTC'''
# Proposal for share groups (cknight)
# Liberty.1 stable release (bswartz)

'''Scheduled for 10 Dec, 2015, 15:00 UTC'''
# The most of the driver already report the "'QoS_support': False" to scheduler. Does we need to remove it or save it as Qos support flat? and write 'QoS_support' to capabilities_and_extra_specs.rst?(zhongjun)
##https://github.com/openstack/manila/blob/master/manila/share/drivers/hpe/hpe_3par_driver.py#L391
##https://github.com/openstack/manila/blob/master/manila/share/drivers/hitachi/hds_hnas.py#L354
##https://github.com/openstack/manila/blob/master/manila/share/drivers/netapp/dataontap/cluster_mode/lib_base.py#L257
##https://github.com/openstack/manila/blob/master/manila/scheduler/host_manager.py#L111
##https://review.openstack.org/#/c/247286/4/doc/source/devref/capabilities_and_extra_specs.rst
# Architectural concerns and interoperability of new features (cknight)

'''Scheduled for 3 Dec, 2015, 15:00 UTC'''
# Midcycle meetup
# Add "data loss" flag in DB migrations which lead to losing of existing data and update "manila-manage db sync" command to forbid applying such migrations without "--force" flag (u_glide)
# Reorganization of Manila scheduler to merge oslo-incubator code and simplify future maintenance (cknight)
## https://review.openstack.org/#/c/252060/
## Mostly for awareness, this is now merged and should be backwards compatible.
## Cinder could benefit from a similar treatment. And with the incubator gone, we could explore a common library with the Cinder team, since the heretofore shared components are certain to diverge over time, but I'm not sure how receptive they would be. Xing?
# CI reliability
## Why should we merge patches for drivers whose CI systems consistently fail? This negates the value of CI. I think that accepting patches that aren't proven by CI does a disservice to everyone, most of all to the driver owners themselves. Refusing to merge driver patches without a good CI result for that driver seems like a gentler and more uniform-over-time approach than threatening to throw out a driver near the end of a release. And why make our illustrious and overworked PTL be the CI policeman when all core reviewers could simply glance at the corresponding CI result before giving +A? (cknight)
## http://ci-watch.tintri.com/project?project=manila
## https://review.openstack.org/#/c/232293/
## https://review.openstack.org/#/c/235522/
## https://review.openstack.org/#/c/235726/
## https://review.openstack.org/#/c/249999/
## https://review.openstack.org/#/c/237454/

'''Scheduled for 19 Nov, 2015, 15:00 UTC'''
# GB --> GiB (markstur)
# OpenStack Manuals and Devref (markstur)
#* Config Ref (and others) are moving to .RST!
#* How should we document prior releases in devref?
# Midcycle Meetup Date (bswartz)
# Deadlines for features (bswartz)
#* Feature proposal freeze vs Big Features
#* New drivers
# Data Copy Service network approach discussion (ganso)
#* https://drive.google.com/file/d/0B30HPZmaePCHMlAzcjJlYm5rRTA/view?usp=sharing

'''Scheduled for 12 Nov, 2015, 15:00 UTC'''
# Which method is better for Manila QoS(independent way or just use extra specs ) (zhongjun)
#* link:http://paste.openstack.org/show/477677/
#*If use extra specs to config QoS, Does we first need add “QoS_support=true/false” in share_type. Does we need to add QoS_support flag in common capabilities
#**https://github.com/openstack/manila/blob/master/doc/source/devref/capabilities_and_extra_specs.rst)?
#** eg: share_type: qos_support='<is> True' or '<is> False'
#** netapp:max_iops=100 or qos:netapp:max_iops=100 or qos:max_iops=100
#** driver update: qos_support=true
#** An alternative could be to have the driver report a range
# Manila DR update (ameade)
#* Still no reviews on https://review.openstack.org/#/c/238572/
#* First-party driver implementation? Which driver, current generic or new solution? When do we need this implemented and who wants to do it?
#* Replication_change share state - This is the state of the share when any of it's replicas are being promoted. Is 'replication_change' good or is there a better phrase?
# Manila Driver minimum requirements document update (ganso)
# Manila Data Copy service name (bswartz)
#*https://review.openstack.org/#/c/244286/
# Upcoming change for Manila CI hooks (vponomaryov)
#*https://review.openstack.org/#/c/243233/
# Manilaclient enhancement to provide request_id when set http_log_debug is True. It will co-ordinate the request and its id, so it will be helpful to filter the log and the failed request. (Jay Xu)
#*http://paste.openstack.org/show/478675/

'''Scheduled for 5 Nov, 2015, 15:00 UTC'''
# Decide on read-only access rules as a required feature
# Manila DR - Design/Code feedback (gouthamr)
#* [https://wiki.openstack.org/wiki/Manila/design/manila-mitaka-data-replication Design Document]
#* [https://review.openstack.org/#/c/238572/ Core Implementation on Gerrit]
#* [https://review.openstack.org/#/c/235448/ Client Implementation on Gerrit]

'''Scheduled for 1 Oct, 2015, 15:00 UTC'''
# Liberty RC2 status
#* https://launchpad.net/manila/+milestone/liberty-rc2
#* https://review.openstack.org/#/q/project:openstack/manila+branch:stable/liberty,n,z
# Mitaka Design Summit
#* https://etherpad.openstack.org/p/manila-mitaka-summit-topics
# Vendor driver docs
#* https://github.com/openstack/manila/blob/master/doc/source/devref/share_back_ends_feature_support_mapping.rst
# Manila Client Nova Style Microversion Support (cfouts)
#* https://review.openstack.org/#/c/229142/
#* http://docs.openstack.org/developer/manila/devref/api_microversion_dev.html

'''Scheduled for 24 Sep, 2015, 15:00 UTC'''
# 3rd-party CI status
#* http://ec2-54-67-102-119.us-west-1.compute.amazonaws.com:5000/?project=openstack/manila&user=&timeframe=72&start=&end=&page_size=500
#* http://ci-watch.tintri.com/project?project=manila
# Liberty RC status
# Support for QoS in Manila (zhongjun2)
#* https://blueprints.launchpad.net/manila/+spec/manila-support-qos
#* https://wiki.openstack.org/wiki/Manila/QoS
# Reminder for backend driver maintainers to update feature support table (toabctl)
#* http://docs.openstack.org/developer/manila/devref/share_back_ends_feature_support_mapping.html
# Regression when using NetApp - license issue with netapp_lib (toabctl)
#* https://bugs.launchpad.net/cinder/+bug/1499334
# Mitaka Design Summit
#* Propose topics - https://etherpad.openstack.org/p/manila-mitaka-summit-topics
# Vendor driver docs
# Open Discussion

Manila/Meetings

2016-08-15T21:12:04Z

Clinton Knight: /* Previous meetings */

= Weekly Manila team meeting =
'''NOTE MEETING TIME: Thursday at 15:00 UTC'''

If you're interested in management of shared filesystems for OpenStack, we have a weekly meetings in <code><nowiki>#openstack-meeting-alt</nowiki></code>, on Thursdays at 15:00 UTC. Please feel free to add items to the agenda below. NOTE: When adding topics please include your IRC name so we know who's topic it is and how to get more info.

'''Previous meeting logs can be found here:''' http://eavesdrop.openstack.org/meetings/manila/

== Next meeting ==
'''NOTE:''' ''Include your IRC nickname next to agenda items so that you can be called upon in the meeting and arrive at the meeting promptly if placing items in agenda. You might want to put this on your calendar if you are adding items.''

'''Scheduled for 11 Aug, 2016, 15:00 UTC'''
# Open Discussion

== Previous meetings ==

"Scheduled for 11 Aug, 2016, 15:00 UTC"
# None

'''Scheduled for 4 Aug, 2016, 15:00 UTC'''
# Migration APIs
#* http://lists.openstack.org/pipermail/openstack-dev/2016-August/100682.html
# Correct approach for avoiding races

'''Scheduled for 28 Jul, 2016, 15:00 UTC'''
# Stepping Down as Docs Liaison (dustins)
#* Current job responsibilities don't give me the time to devote to this that I'd like to have, so it should go to someone else with a passion for documentation and has the willingness and desire for taking up the mantle.
# N-3 Review focus
#* https://etherpad.openstack.org/p/manila-newton3-priorities

'''Scheduled for 21 Jul, 2016, 15:00 UTC'''
# Migrate API reference into tree (dgonzalez)
#* https://review.openstack.org/#/c/313874/

'''Scheduled for 14 Jul, 2016, 15:00 UTC'''
# N-2 milestone
# Managing initial security (of the root directory) for new shares
# Mascot/Logo discussion
# Tempest alternatives

'''Scheduled for 30 Jun, 2016, 15:00 UTC'''
# non-disruptive nfs-ganesha export modifications (rraja/tbarron)
#* https://sourceforge.net/p/nfs-ganesha/mailman/message/35173839
# share backup spec discussion (zhongjun)
#* https://review.openstack.org/#/c/330306

'''Scheduled for 23 Jun, 2016, 15:00 UTC'''
# Implement extend and shrink features for glusterfs driver (DaiDV)
# Do all drivers have to inherit from driver.ShareDriver? (mkoderer)
# Creating a second share instance for driver-assisted migration (ganso)

'''Scheduled for 16 Jun, 2016, 15:00 UTC'''
# A share support multiple protocols (zhongjun)
#* https://review.openstack.org/#/c/329392/
# Drivers with minimum size for shares (tpsilva)

'''Scheduled for 9 Jun, 2016, 15:00 UTC'''
# HPB feature (mkoderer)
# the fate of nova-network support in manila (gouthamr)
# multi-AZ tests in the gate

'''Scheduled for 2 Jun, 2016, 15:00 UTC'''
# N-1 milestone (bswartz)
# Decision on APIv1 removal for newton (bswartz)
# Model updates from drivers (bswartz)
# Midcycle meetup
# Spec update
#* New spec - Extending share networks to span subnets - https://review.openstack.org/#/c/323646/ (gouthamr)

'''Scheduled for 26 May, 2016, 15:00 UTC'''
# N-1 milestone
# Update access remaining work
# Topics not covered at design summit https://etherpad.openstack.org/p/newton-manila-contributor-meetup
#* Nova-network removal
#* APIv1 removal
#* Model updates from the driver

'''Scheduled for 19 May, 2016, 15:00 UTC'''
# Topics not covered at design summit https://etherpad.openstack.org/p/newton-manila-contributor-meetup
#* UI Customization
# Share Backup(zhongjun)
# Snapshot Restore (cknight)
# Specs (ganso)

'''Scheduled for 12 May, 2016, 15:00 UTC'''
# Driver private storage: https://review.openstack.org/#/c/315346/ (xyang)
#* Admin wants to check driver private storage before doing some actions on a resource, i.e., force delete a share or unmanage a share.
#* The patch proposes to add admin API to retrieve driver private storage data. Alternative approach proposed by Valeriy to use manila-manage (i.e., manila-manage driver-private-storage <uuid of a resource>). Which one should we choose?
# Topics not covered at design summit https://etherpad.openstack.org/p/newton-manila-contributor-meetup
#* Py3 functional tests
#* Replication with share networks
#* Service-check
#* Container driver

'''Scheduled for 5 May, 2016, 15:00 UTC'''
# Topics not covered at design summit https://etherpad.openstack.org/p/newton-manila-contributor-meetup
#* Documentation update (dustins/gouthamr)
#* User messages
#* Experimental features in UI
#* Share/snapshot size mismatch
# BP Specs (tbarron)
#* Need to ensure that the repo doesn't become "a wasteland of partial ideas where hope goes to die." (See http://lists.openstack.org/pipermail/openstack-dev/2016-May/094026.html)

'''Scheduled for 14 Apr, 2016, 15:00 UTC'''
# Manage API retrying (gouthamr/nidhimittalhada)
# Tempest test structuring - resource sharing and cleanup issues (akerr)
# Snapshot support common capability in the documentation (gouthamr)

'''Scheduled for 7 Apr, 2016, 15:00 UTC'''
# Access rules (bswartz)
# Design summit planning (bswartz)
#* Link: https://etherpad.openstack.org/p/manila-newton-summit-topics
# Release notes, continued (gouthamr)
#* Link: https://review.openstack.org/#/c/300656/
# Midcycle meetup
# Access Groups
#* https://wiki.openstack.org/wiki/Manila/design/access_groups
# Revert to snapshot

'''Scheduled for 31 Mar, 2016, 15:00 UTC'''
# Mitaka release status
#* RC1 expected to be the Mitaka release build
#* Last day for a re-spin is today
#** http://lists.openstack.org/pipermail/openstack-dev/2016-March/090877.html
# Design summit planning
#* Final schedule confirms 2 fishbowls, 4 working sessions, 1 half-day meetup
#** http://lists.openstack.org/pipermail/openstack-dev/2016-March/091032.html
#* Voting open now for summit topics
#** https://etherpad.openstack.org/p/manila-newton-summit-topics
# Manila Documentation (gouthamr)
#* Where do our docs live? how do we maintain them? (In Tree: devref, adminref | out of tree: install-guide, config-ref, cloud-admin guide)
#* Need a docs liaison
#** https://wiki.openstack.org/wiki/CrossProjectLiaisons#Documentation
#* Release Notes
#** http://docs.openstack.org/developer/reno/usage.html
# Managing/unmanaging replicated shares and snapshots (gouthamr)

'''Scheduled for 24 Mar, 2016, 15:00 UTC'''
# Design summit planning
#* https://etherpad.openstack.org/p/manila-newton-summit-topics
# Question about share servers (Jay.Xu)
#* Is there a way to get gateway IP address when the API setup_server() is invoked. I did not see this attribute in the input parameter network_info. Gateway IP is mandatory to create the interface for EMC storage which will be release coming soon, so it is expected that we can add the gateway IP into NetworkAllocation model if not.

'''Scheduled for 10 Mar, 2016, 15:00 UTC'''
# Common Capabilities Matrix (dustins)
#* https://review.openstack.org/#/c/288504/
#* I'm adding a table in the Developer Reference documentation to show which common capabilities are supported by which backends and when that support started.
#* I've got most drivers covered now.
#* If you want me to add your capabilities, leave a review on the patch and I'll update the table!

'''Scheduled for 25 Feb, 2016, 15:00 UTC'''

# OpenStack bug smash event (toabctl, mkoderer)
## link: https://etherpad.openstack.org/p/OpenStack-Bug-Smash-Mitaka
# Status of new drivers
## M.3 deadline is a week away.
## Only a few things have merged in the last couple days. At this rate, many things won't make it.
## Zuul is busy and will only get worse until the deadline.
## Multiple Manila gate breakages have occurred recently, and more are likely.
## The Manila tests are very unreliable at concurrency > 1.
## The generic driver is too slow to pass the gate at concurrency = 1.
## The new drivers *are* fast enough to run at concurrency = 1.
## The new drivers may not be 100% done, but they are done enough to pass reliably at concurrency = 1.
## We should switch voting to the new drivers NOW, or start throwing stuff overboard for the release.
# Next Manila mid-cycle location
## Proposal is to do it in Germany
## Need to know who could attend at that location.
## Would it make a difference if the Cinder mid-cycle happens in Dublin, and the Manila meeting is immediately before or after the Cinder one?

'''Scheduled for 18 Feb, 2016, 15:00 UTC'''

# Feature Proposal Freeze (bswartz)
# Open Discussion

'''Scheduled for 11 Feb, 2016, 15:00 UTC'''

# Hierarchical port binding support (mkoderer)
#* https://wiki.openstack.org/wiki/Manila/design/manila-newton-hpb-support
#* https://blueprints.launchpad.net/manila/+spec/manila-hpb-support
#* https://review.openstack.org/#/c/277731/
# Next Manila mid-cycle (mkoderer)
#* https://etherpad.openstack.org/p/manila-n-release-midcycle
# Replication + ZFS Driver (bswartz)
#* https://review.openstack.org/#/c/238572
# Driver FPF (bswartz)
#* https://review.openstack.org/#/c/279090/
#* https://review.openstack.org/#/c/278169/
# HDFS CI (bswartz)
'''Scheduled for 4 Feb, 2016, 15:00 UTC'''
# Data service handling protocol mount commands (ganso)
#* https://etherpad.openstack.org/p/manila-data-service-mount-meeting
# Separate process for copy on data service (tpsilva)
# ZFS driver (bswartz)
# First party drivers share migration support cooperation (ganso)

'''Scheduled for 28 Jan, 2016, 15:00 UTC'''
# Hierarchical port binding support (mkoderer)
#* https://etherpad.openstack.org/p/manila-hierarchical-port-binding
# Core reviewer team and other manila-related projects (bswartz)
# Replication status (bswartz)

'''Scheduled for 21 Jan, 2016, 15:00 UTC'''
# Update_access backwards compatibility approach discussion. (ganso)
# Exposing the Data Service to drivers. (ganso)
# Snapshot manage/unmanage (bswartz)

'''Scheduled for 17 Dec, 2015, 15:00 UTC'''
# Proposal for share groups (cknight)
# Liberty.1 stable release (bswartz)

'''Scheduled for 10 Dec, 2015, 15:00 UTC'''
# The most of the driver already report the "'QoS_support': False" to scheduler. Does we need to remove it or save it as Qos support flat? and write 'QoS_support' to capabilities_and_extra_specs.rst?(zhongjun)
##https://github.com/openstack/manila/blob/master/manila/share/drivers/hpe/hpe_3par_driver.py#L391
##https://github.com/openstack/manila/blob/master/manila/share/drivers/hitachi/hds_hnas.py#L354
##https://github.com/openstack/manila/blob/master/manila/share/drivers/netapp/dataontap/cluster_mode/lib_base.py#L257
##https://github.com/openstack/manila/blob/master/manila/scheduler/host_manager.py#L111
##https://review.openstack.org/#/c/247286/4/doc/source/devref/capabilities_and_extra_specs.rst
# Architectural concerns and interoperability of new features (cknight)

'''Scheduled for 3 Dec, 2015, 15:00 UTC'''
# Midcycle meetup
# Add "data loss" flag in DB migrations which lead to losing of existing data and update "manila-manage db sync" command to forbid applying such migrations without "--force" flag (u_glide)
# Reorganization of Manila scheduler to merge oslo-incubator code and simplify future maintenance (cknight)
## https://review.openstack.org/#/c/252060/
## Mostly for awareness, this is now merged and should be backwards compatible.
## Cinder could benefit from a similar treatment. And with the incubator gone, we could explore a common library with the Cinder team, since the heretofore shared components are certain to diverge over time, but I'm not sure how receptive they would be. Xing?
# CI reliability
## Why should we merge patches for drivers whose CI systems consistently fail? This negates the value of CI. I think that accepting patches that aren't proven by CI does a disservice to everyone, most of all to the driver owners themselves. Refusing to merge driver patches without a good CI result for that driver seems like a gentler and more uniform-over-time approach than threatening to throw out a driver near the end of a release. And why make our illustrious and overworked PTL be the CI policeman when all core reviewers could simply glance at the corresponding CI result before giving +A? (cknight)
## http://ci-watch.tintri.com/project?project=manila
## https://review.openstack.org/#/c/232293/
## https://review.openstack.org/#/c/235522/
## https://review.openstack.org/#/c/235726/
## https://review.openstack.org/#/c/249999/
## https://review.openstack.org/#/c/237454/

'''Scheduled for 19 Nov, 2015, 15:00 UTC'''
# GB --> GiB (markstur)
# OpenStack Manuals and Devref (markstur)
#* Config Ref (and others) are moving to .RST!
#* How should we document prior releases in devref?
# Midcycle Meetup Date (bswartz)
# Deadlines for features (bswartz)
#* Feature proposal freeze vs Big Features
#* New drivers
# Data Copy Service network approach discussion (ganso)
#* https://drive.google.com/file/d/0B30HPZmaePCHMlAzcjJlYm5rRTA/view?usp=sharing

'''Scheduled for 12 Nov, 2015, 15:00 UTC'''
# Which method is better for Manila QoS(independent way or just use extra specs ) (zhongjun)
#* link:http://paste.openstack.org/show/477677/
#*If use extra specs to config QoS, Does we first need add “QoS_support=true/false” in share_type. Does we need to add QoS_support flag in common capabilities
#**https://github.com/openstack/manila/blob/master/doc/source/devref/capabilities_and_extra_specs.rst)?
#** eg: share_type: qos_support='<is> True' or '<is> False'
#** netapp:max_iops=100 or qos:netapp:max_iops=100 or qos:max_iops=100
#** driver update: qos_support=true
#** An alternative could be to have the driver report a range
# Manila DR update (ameade)
#* Still no reviews on https://review.openstack.org/#/c/238572/
#* First-party driver implementation? Which driver, current generic or new solution? When do we need this implemented and who wants to do it?
#* Replication_change share state - This is the state of the share when any of it's replicas are being promoted. Is 'replication_change' good or is there a better phrase?
# Manila Driver minimum requirements document update (ganso)
# Manila Data Copy service name (bswartz)
#*https://review.openstack.org/#/c/244286/
# Upcoming change for Manila CI hooks (vponomaryov)
#*https://review.openstack.org/#/c/243233/
# Manilaclient enhancement to provide request_id when set http_log_debug is True. It will co-ordinate the request and its id, so it will be helpful to filter the log and the failed request. (Jay Xu)
#*http://paste.openstack.org/show/478675/

'''Scheduled for 5 Nov, 2015, 15:00 UTC'''
# Decide on read-only access rules as a required feature
# Manila DR - Design/Code feedback (gouthamr)
#* [https://wiki.openstack.org/wiki/Manila/design/manila-mitaka-data-replication Design Document]
#* [https://review.openstack.org/#/c/238572/ Core Implementation on Gerrit]
#* [https://review.openstack.org/#/c/235448/ Client Implementation on Gerrit]

'''Scheduled for 1 Oct, 2015, 15:00 UTC'''
# Liberty RC2 status
#* https://launchpad.net/manila/+milestone/liberty-rc2
#* https://review.openstack.org/#/q/project:openstack/manila+branch:stable/liberty,n,z
# Mitaka Design Summit
#* https://etherpad.openstack.org/p/manila-mitaka-summit-topics
# Vendor driver docs
#* https://github.com/openstack/manila/blob/master/doc/source/devref/share_back_ends_feature_support_mapping.rst
# Manila Client Nova Style Microversion Support (cfouts)
#* https://review.openstack.org/#/c/229142/
#* http://docs.openstack.org/developer/manila/devref/api_microversion_dev.html

'''Scheduled for 24 Sep, 2015, 15:00 UTC'''
# 3rd-party CI status
#* http://ec2-54-67-102-119.us-west-1.compute.amazonaws.com:5000/?project=openstack/manila&user=&timeframe=72&start=&end=&page_size=500
#* http://ci-watch.tintri.com/project?project=manila
# Liberty RC status
# Support for QoS in Manila (zhongjun2)
#* https://blueprints.launchpad.net/manila/+spec/manila-support-qos
#* https://wiki.openstack.org/wiki/Manila/QoS
# Reminder for backend driver maintainers to update feature support table (toabctl)
#* http://docs.openstack.org/developer/manila/devref/share_back_ends_feature_support_mapping.html
# Regression when using NetApp - license issue with netapp_lib (toabctl)
#* https://bugs.launchpad.net/cinder/+bug/1499334
# Mitaka Design Summit
#* Propose topics - https://etherpad.openstack.org/p/manila-mitaka-summit-topics
# Vendor driver docs
# Open Discussion

Manila/Meetings

2016-05-19T14:55:59Z

Clinton Knight: /* Next meeting */

= Weekly Manila team meeting =
'''NOTE MEETING TIME: Thursday at 15:00 UTC'''

If you're interested in management of shared filesystems for OpenStack, we have a weekly meetings in <code><nowiki>#openstack-meeting-alt</nowiki></code>, on Thursdays at 15:00 UTC. Please feel free to add items to the agenda below. NOTE: When adding topics please include your IRC name so we know who's topic it is and how to get more info.

== Next meeting ==
'''NOTE:''' ''Include your IRC nickname next to agenda items so that you can be called upon in the meeting and arrive at the meeting promptly if placing items in agenda. You might want to put this on your calendar if you are adding items.''

'''Scheduled for 19 May, 2016, 15:00 UTC'''
# Topics not covered at design summit https://etherpad.openstack.org/p/newton-manila-contributor-meetup
#* UI Customization
#* Nova-network removal
#* APIv1 removal
#* Model updates from the driver
# Share Backup(zhongjun)
# Snapshot Restore (cknight)
# Open Discussion

== Previous meetings ==

'''Scheduled for 12 May, 2016, 15:00 UTC'''
# Driver private storage: https://review.openstack.org/#/c/315346/ (xyang)
#* Admin wants to check driver private storage before doing some actions on a resource, i.e., force delete a share or unmanage a share.
#* The patch proposes to add admin API to retrieve driver private storage data. Alternative approach proposed by Valeriy to use manila-manage (i.e., manila-manage driver-private-storage <uuid of a resource>). Which one should we choose?
# Topics not covered at design summit https://etherpad.openstack.org/p/newton-manila-contributor-meetup
#* Py3 functional tests
#* Replication with share networks
#* Service-check
#* Container driver

'''Scheduled for 5 May, 2016, 15:00 UTC'''
# Topics not covered at design summit https://etherpad.openstack.org/p/newton-manila-contributor-meetup
#* Documentation update (dustins/gouthamr)
#* User messages
#* Experimental features in UI
#* Share/snapshot size mismatch
# BP Specs (tbarron)
#* Need to ensure that the repo doesn't become "a wasteland of partial ideas where hope goes to die." (See http://lists.openstack.org/pipermail/openstack-dev/2016-May/094026.html)

'''Scheduled for 14 Apr, 2016, 15:00 UTC'''
# Manage API retrying (gouthamr/nidhimittalhada)
# Tempest test structuring - resource sharing and cleanup issues (akerr)
# Snapshot support common capability in the documentation (gouthamr)

'''Scheduled for 7 Apr, 2016, 15:00 UTC'''
# Access rules (bswartz)
# Design summit planning (bswartz)
#* Link: https://etherpad.openstack.org/p/manila-newton-summit-topics
# Release notes, continued (gouthamr)
#* Link: https://review.openstack.org/#/c/300656/
# Midcycle meetup
# Access Groups
#* https://wiki.openstack.org/wiki/Manila/design/access_groups
# Revert to snapshot

'''Scheduled for 31 Mar, 2016, 15:00 UTC'''
# Mitaka release status
#* RC1 expected to be the Mitaka release build
#* Last day for a re-spin is today
#** http://lists.openstack.org/pipermail/openstack-dev/2016-March/090877.html
# Design summit planning
#* Final schedule confirms 2 fishbowls, 4 working sessions, 1 half-day meetup
#** http://lists.openstack.org/pipermail/openstack-dev/2016-March/091032.html
#* Voting open now for summit topics
#** https://etherpad.openstack.org/p/manila-newton-summit-topics
# Manila Documentation (gouthamr)
#* Where do our docs live? how do we maintain them? (In Tree: devref, adminref | out of tree: install-guide, config-ref, cloud-admin guide)
#* Need a docs liaison
#** https://wiki.openstack.org/wiki/CrossProjectLiaisons#Documentation
#* Release Notes
#** http://docs.openstack.org/developer/reno/usage.html
# Managing/unmanaging replicated shares and snapshots (gouthamr)

'''Scheduled for 24 Mar, 2016, 15:00 UTC'''
# Design summit planning
#* https://etherpad.openstack.org/p/manila-newton-summit-topics
# Question about share servers (Jay.Xu)
#* Is there a way to get gateway IP address when the API setup_server() is invoked. I did not see this attribute in the input parameter network_info. Gateway IP is mandatory to create the interface for EMC storage which will be release coming soon, so it is expected that we can add the gateway IP into NetworkAllocation model if not.

'''Scheduled for 10 Mar, 2016, 15:00 UTC'''
# Common Capabilities Matrix (dustins)
#* https://review.openstack.org/#/c/288504/
#* I'm adding a table in the Developer Reference documentation to show which common capabilities are supported by which backends and when that support started.
#* I've got most drivers covered now.
#* If you want me to add your capabilities, leave a review on the patch and I'll update the table!

'''Scheduled for 25 Feb, 2016, 15:00 UTC'''

# OpenStack bug smash event (toabctl, mkoderer)
## link: https://etherpad.openstack.org/p/OpenStack-Bug-Smash-Mitaka
# Status of new drivers
## M.3 deadline is a week away.
## Only a few things have merged in the last couple days. At this rate, many things won't make it.
## Zuul is busy and will only get worse until the deadline.
## Multiple Manila gate breakages have occurred recently, and more are likely.
## The Manila tests are very unreliable at concurrency > 1.
## The generic driver is too slow to pass the gate at concurrency = 1.
## The new drivers *are* fast enough to run at concurrency = 1.
## The new drivers may not be 100% done, but they are done enough to pass reliably at concurrency = 1.
## We should switch voting to the new drivers NOW, or start throwing stuff overboard for the release.
# Next Manila mid-cycle location
## Proposal is to do it in Germany
## Need to know who could attend at that location.
## Would it make a difference if the Cinder mid-cycle happens in Dublin, and the Manila meeting is immediately before or after the Cinder one?

'''Scheduled for 18 Feb, 2016, 15:00 UTC'''

# Feature Proposal Freeze (bswartz)
# Open Discussion

'''Scheduled for 11 Feb, 2016, 15:00 UTC'''

# Hierarchical port binding support (mkoderer)
#* https://wiki.openstack.org/wiki/Manila/design/manila-newton-hpb-support
#* https://blueprints.launchpad.net/manila/+spec/manila-hpb-support
#* https://review.openstack.org/#/c/277731/
# Next Manila mid-cycle (mkoderer)
#* https://etherpad.openstack.org/p/manila-n-release-midcycle
# Replication + ZFS Driver (bswartz)
#* https://review.openstack.org/#/c/238572
# Driver FPF (bswartz)
#* https://review.openstack.org/#/c/279090/
#* https://review.openstack.org/#/c/278169/
# HDFS CI (bswartz)
'''Scheduled for 4 Feb, 2016, 15:00 UTC'''
# Data service handling protocol mount commands (ganso)
#* https://etherpad.openstack.org/p/manila-data-service-mount-meeting
# Separate process for copy on data service (tpsilva)
# ZFS driver (bswartz)
# First party drivers share migration support cooperation (ganso)

'''Scheduled for 28 Jan, 2016, 15:00 UTC'''
# Hierarchical port binding support (mkoderer)
#* https://etherpad.openstack.org/p/manila-hierarchical-port-binding
# Core reviewer team and other manila-related projects (bswartz)
# Replication status (bswartz)

'''Scheduled for 21 Jan, 2016, 15:00 UTC'''
# Update_access backwards compatibility approach discussion. (ganso)
# Exposing the Data Service to drivers. (ganso)
# Snapshot manage/unmanage (bswartz)

'''Scheduled for 17 Dec, 2015, 15:00 UTC'''
# Proposal for share groups (cknight)
# Liberty.1 stable release (bswartz)

'''Scheduled for 10 Dec, 2015, 15:00 UTC'''
# The most of the driver already report the "'QoS_support': False" to scheduler. Does we need to remove it or save it as Qos support flat? and write 'QoS_support' to capabilities_and_extra_specs.rst?(zhongjun)
##https://github.com/openstack/manila/blob/master/manila/share/drivers/hpe/hpe_3par_driver.py#L391
##https://github.com/openstack/manila/blob/master/manila/share/drivers/hitachi/hds_hnas.py#L354
##https://github.com/openstack/manila/blob/master/manila/share/drivers/netapp/dataontap/cluster_mode/lib_base.py#L257
##https://github.com/openstack/manila/blob/master/manila/scheduler/host_manager.py#L111
##https://review.openstack.org/#/c/247286/4/doc/source/devref/capabilities_and_extra_specs.rst
# Architectural concerns and interoperability of new features (cknight)

'''Scheduled for 3 Dec, 2015, 15:00 UTC'''
# Midcycle meetup
# Add "data loss" flag in DB migrations which lead to losing of existing data and update "manila-manage db sync" command to forbid applying such migrations without "--force" flag (u_glide)
# Reorganization of Manila scheduler to merge oslo-incubator code and simplify future maintenance (cknight)
## https://review.openstack.org/#/c/252060/
## Mostly for awareness, this is now merged and should be backwards compatible.
## Cinder could benefit from a similar treatment. And with the incubator gone, we could explore a common library with the Cinder team, since the heretofore shared components are certain to diverge over time, but I'm not sure how receptive they would be. Xing?
# CI reliability
## Why should we merge patches for drivers whose CI systems consistently fail? This negates the value of CI. I think that accepting patches that aren't proven by CI does a disservice to everyone, most of all to the driver owners themselves. Refusing to merge driver patches without a good CI result for that driver seems like a gentler and more uniform-over-time approach than threatening to throw out a driver near the end of a release. And why make our illustrious and overworked PTL be the CI policeman when all core reviewers could simply glance at the corresponding CI result before giving +A? (cknight)
## http://ci-watch.tintri.com/project?project=manila
## https://review.openstack.org/#/c/232293/
## https://review.openstack.org/#/c/235522/
## https://review.openstack.org/#/c/235726/
## https://review.openstack.org/#/c/249999/
## https://review.openstack.org/#/c/237454/

'''Scheduled for 19 Nov, 2015, 15:00 UTC'''
# GB --> GiB (markstur)
# OpenStack Manuals and Devref (markstur)
#* Config Ref (and others) are moving to .RST!
#* How should we document prior releases in devref?
# Midcycle Meetup Date (bswartz)
# Deadlines for features (bswartz)
#* Feature proposal freeze vs Big Features
#* New drivers
# Data Copy Service network approach discussion (ganso)
#* https://drive.google.com/file/d/0B30HPZmaePCHMlAzcjJlYm5rRTA/view?usp=sharing

'''Scheduled for 12 Nov, 2015, 15:00 UTC'''
# Which method is better for Manila QoS(independent way or just use extra specs ) (zhongjun)
#* link:http://paste.openstack.org/show/477677/
#*If use extra specs to config QoS, Does we first need add “QoS_support=true/false” in share_type. Does we need to add QoS_support flag in common capabilities
#**https://github.com/openstack/manila/blob/master/doc/source/devref/capabilities_and_extra_specs.rst)?
#** eg: share_type: qos_support='<is> True' or '<is> False'
#** netapp:max_iops=100 or qos:netapp:max_iops=100 or qos:max_iops=100
#** driver update: qos_support=true
#** An alternative could be to have the driver report a range
# Manila DR update (ameade)
#* Still no reviews on https://review.openstack.org/#/c/238572/
#* First-party driver implementation? Which driver, current generic or new solution? When do we need this implemented and who wants to do it?
#* Replication_change share state - This is the state of the share when any of it's replicas are being promoted. Is 'replication_change' good or is there a better phrase?
# Manila Driver minimum requirements document update (ganso)
# Manila Data Copy service name (bswartz)
#*https://review.openstack.org/#/c/244286/
# Upcoming change for Manila CI hooks (vponomaryov)
#*https://review.openstack.org/#/c/243233/
# Manilaclient enhancement to provide request_id when set http_log_debug is True. It will co-ordinate the request and its id, so it will be helpful to filter the log and the failed request. (Jay Xu)
#*http://paste.openstack.org/show/478675/

'''Scheduled for 5 Nov, 2015, 15:00 UTC'''
# Decide on read-only access rules as a required feature
# Manila DR - Design/Code feedback (gouthamr)
#* [https://wiki.openstack.org/wiki/Manila/design/manila-mitaka-data-replication Design Document]
#* [https://review.openstack.org/#/c/238572/ Core Implementation on Gerrit]
#* [https://review.openstack.org/#/c/235448/ Client Implementation on Gerrit]

'''Scheduled for 1 Oct, 2015, 15:00 UTC'''
# Liberty RC2 status
#* https://launchpad.net/manila/+milestone/liberty-rc2
#* https://review.openstack.org/#/q/project:openstack/manila+branch:stable/liberty,n,z
# Mitaka Design Summit
#* https://etherpad.openstack.org/p/manila-mitaka-summit-topics
# Vendor driver docs
#* https://github.com/openstack/manila/blob/master/doc/source/devref/share_back_ends_feature_support_mapping.rst
# Manila Client Nova Style Microversion Support (cfouts)
#* https://review.openstack.org/#/c/229142/
#* http://docs.openstack.org/developer/manila/devref/api_microversion_dev.html

'''Scheduled for 24 Sep, 2015, 15:00 UTC'''
# 3rd-party CI status
#* http://ec2-54-67-102-119.us-west-1.compute.amazonaws.com:5000/?project=openstack/manila&user=&timeframe=72&start=&end=&page_size=500
#* http://ci-watch.tintri.com/project?project=manila
# Liberty RC status
# Support for QoS in Manila (zhongjun2)
#* https://blueprints.launchpad.net/manila/+spec/manila-support-qos
#* https://wiki.openstack.org/wiki/Manila/QoS
# Reminder for backend driver maintainers to update feature support table (toabctl)
#* http://docs.openstack.org/developer/manila/devref/share_back_ends_feature_support_mapping.html
# Regression when using NetApp - license issue with netapp_lib (toabctl)
#* https://bugs.launchpad.net/cinder/+bug/1499334
# Mitaka Design Summit
#* Propose topics - https://etherpad.openstack.org/p/manila-mitaka-summit-topics
# Vendor driver docs
# Open Discussion

http://eavesdrop.openstack.org/meetings/manila/

Manila/design/manila-generic-groups

2016-04-22T20:07:20Z

Clinton Knight: /* Implementation details */

= Manila generic groups =
== Executive summary ==
Manila needs a grouping construct that, like shares, is a 1st-class atomic data type. Our experience with CGs has demonstrated the complexity of adding a grouping capability, yet there are other use cases such as migration, replication, and backup in which some storage controllers could only offer such features on share groups. CGs also highlighted the poor optics of an advanced feature with comparatively little potential for vendor support. And adding new grouping constructs for each new feature is not technically feasible. All of the above may be addressed by generic groups, which we think is a clean extension to the original architecture of Manila.
== An Intractable Matrix ==
Manila is at an exciting time in its development, where all the simple features are available and the community is focused on adding high-value features such as migration, replication, and consistency groups. Experimental code is available for each of these, but little consideration has been given to the obvious need for these complex features to interact not only with each other but also with whatever additional features were added later. And looking a little deeper, it is apparent that each of these has limitations that could be solved by a single architectural enhancement.
=== Consistency groups ===
Consistency groups (CGs) are the only construct in Manila that operates on a group of shares. For some, a CG implies a guarantee of ordered writes by a storage controller, while for others a CG is a mechanism for taking consistent point-in-time snapshots of a set of shares. A group with either attribute may have value by itself, but the CGs implementation doesn’t distinguish between them. 

CGs are a highly specialized construct with dedicated CLI commands, REST APIs, database tables, scheduler filters, and driver APIs that totaled over 9900 lines. None of these are reusable for anything else, and none of the rest of Manila has any awareness of CGs. Even worse, despite all the complexity of the CG feature, only a small minority of storage backends can support them, so the code-to-value ratio is very low and the limited availability of CGs ensures the user experience is inconsistent between clouds.

=== Replication ===
Replication is another high value feature, and the Manila implementation is arguably clean and flexible, but as constituted there is no core ability to replicate groups of shares. It seems reasonable to implement the feature iteratively, beginning with replication of single shares. However, there has been the tacit acknowledgement that some backends would not be able to replicate individual shares, even though those same backends could potentially replicate a group of shares if the group were constituted in a certain way. 

Following the precedent established with CGs, one approach would be to introduce ‘replication groups’ whereby multiple shares could be replicated together. But this would introduce yet another set of APIs, tables, etc. that are too specialized to use for anything else. 

Another approach might be to define CGs as the grouping unit of replication, but this makes little sense because consistency groups (however we define them) and replication are distinct features that shouldn’t be artificially bundled for the sake of expediency of definition or implementation.
=== Migration ===
Just as some backends may only be able to replicate shares in groups, it follows that those backends may also need to migrate shares in groups. And in the case of CGs, it doesn’t make sense to migrate CG members individually; the whole group must be moved, requiring the migration engine to be CG aware. 

Consider the following reasonable use cases:
* Replicate a consistency group
* Snapshot a replication group
* Migrate a replication group
* Retype shares in a consistency group
* Retype shares in a replication group
* Backup a consistency group
* Backup a replication group
* Apply a common policy to a set of shares
 
On the current path, the support matrix has a dimension dedicated solely to features, so each feature must be coded to be interoperable with every other feature. This quickly becomes a support and testing nightmare, and it becomes exponentially more complicated to add more features going forward. 

Fundamentally, the problem is that we are adding features without an underlying architectural framework on which to hang them. To escape the matrix, we must step back and rethink a few things.

== A simple solution ==
So to arrive at a solution, let’s enumerate what we have:
* Primitive objects (shares) with supported operations controlled by share types that vary by backend
* A very specific group object (CGs) with limited support potential by backends
* A number of APIs (CGs, etc.) that many users can’t use at all and appear grafted into the project
 
And what would we prefer:
* A uniform API and user experience that varies as little as possible by backend
* A clean way to group shares so we can do CGs, multi-object replication, migration, etc.
 
How do we get here?
* Universally available groups of primitive objects with supported operations that vary by backend
 
=== Idea #1: Introduce a generic Share Group object ===
Just as Manila has ‘shares’, it should also have ‘share groups’. In its simplest form, unlike CGs, a share group should not guarantee any specialized operation. Instead, it should merely constitute an atomic Manila data type on which nearly any Manila action is available. For example, given a share group, the user should be able to select the group in the Manila UI and invoke features such as snapshot, clone, backup, migrate, replicate, retype, etc. 

In the general case, group actions are handled by the share manager layer (or potentially the common share driver superclass). For example, invoking ‘snapshot’ on a group causes the share manager to take a snapshot of each group member individually. The resulting group snapshot object may be used to create a new group, not unlike how CG snapshots were implemented. 

There are numerous advantages to this approach. Every driver, no matter how unsophisticated, can leverage the manageability goodness of groups. The user experience is uniform, since groups are always available regardless of which backends are present and because nearly every Manila action available on shares is also available on groups. 

Of course, feature-rich backends would like to add their secret sauce, whether it be CGs, group-based replication, or whatever else comes along that might be easier/cheaper/faster to do to groups of shares. That brings us to a related idea.

=== Idea #2: Introduce Share Group types ===
Just as Manila has ‘share types’, it should also have ‘share group types’. Any driver that can perform a group operation in an advantaged way may report that as a group capability, such as:
* Ordered writes
* Consistent snapshots
* Group replication
* Group backup
* Group modification
 
As with share types, the cloud administrator predefines share group types that may contain extra specs corresponding to the group capabilities reported by the backends. The admin also specifies which share type(s) a given group type may contain. When creating a group, the user specifies the group type and share type, and the scheduler then creates the group on one of the backends that match the specified share and share group type. 

Anytime a group action, such as ‘snapshot’, comes into the share manager, the manager checks whether its driver offers an advantaged implementation of that operation. If not, the manager handles the workflow itself as described above. But if so, the manager routes the workflow to its driver for fulfillment. 

The advantages of this approach should be obvious. Development and testing are simplified because there isn’t a need to define and test a different set of group management APIs for each feature, or to test every combination of every feature. Instead of becoming an N-by-N matrix of interacting features, Manila largely becomes an N-by-2 matrix of actions that may be invoked on either individual shares or share groups. Users and admins are already familiar with share types, so introducing share group types would seem a natural and consistent evolution of the same foundational concept.

The matrix would look like:
{| class="wikitable"
|-
! Share action !! Share Group action
|-
| Create (volume type) || Create (volume types, group type)
|-
| Delete || Delete (group, optionally all members)
|-
| Snapshot || Snapshot (may or may not be a CG snapshot)
|-
| Create from snapshot / Clone || Create from group snapshot / Clone group
|-
| Replicate || Replicate
|-
| Backup || Backup
|-
| Retype || Retype
|-
| Migrate || Migrate
|-
| Extend / shrink || N/A
|}

== Manila implementation plan ==
Implementing generic groups in Manila should be a straightforward series of steps:
# Implement generic groups. Because we did CGs, we already know all parts of the codebase that must change to support any kind of group. So the simplest approach is to modify the CG code to morph it into the generic groups feature.
# Add the share group type feature by duplicating and customizing the share type code.
# Enhance the scheduler to place groups according to share group types. Like #1, this is already informed by the CG project.
# Implement group snapshots in the share manager to demonstrate group snapshots in any driver.
# Plumb group snapshots to a CG-capable driver to demonstrate CG functionality in the new framework.
# Update Tempest to cover all of the above.
# Update Manila client to change CGs to share groups.
# Enhance Manila client with share group types.
# Add share group support to Horizon. We never built CG support into Horizon, so this is all-new work that now has much broader appeal and applicability.
# Add additional group actions over time (migrate, replicate, retype, clone, …). At this point, new group capabilities become vertical slices that are simple to add incrementally.
 
Because we implemented CGs just recently as an experimental feature, we have the freedom to replace that code without deprecation or upgrade considerations. Steps 1-8 would get Manila to parity with the CG feature added in Liberty, would require only a few person-weeks of effort, and would better position Manila for long-term evolution and supportability.
=== Implementation details ===
There are few things to note, several of which were already solved during the CG work. 

Groups are first-class objects in Manila, and operations on groups are treated as atomic. To enable support by as many backends as possible, Manila will still maintain DB objects for both group and member snapshots, just as was done with CGs. 

The capabilities of a group will all be public extra specs, similar to snapshot_support in share types. Users will need to know what a group can do.

A few actions, such as extend & shrink, are inherently applicable only to individual shares. One could theoretically apply extend to a group, increasing the size of each member, but that seems like an unlikely use case. Any actions in this category must remain available to group members, and other actions such as taking snapshots of group members could be allowed, but actions such as migration or replication would be available only at the group level and not on its members. 

A group is limited to a single backend. Allowing groups that span backends is theoretically possible, but that would require fanout of operations from the API layer to multiple share managers across the asynchronous event bus, which would lead to complicated synchronization and state management for little operational benefit. 

As was done with Manila CGs, a driver may optionally limit a group to either the confines of a pool or an entire backend. It is known that pools are the unit of data motion (i.e. replication or migration) for some backends, so we think drivers need this flexibility. 

In a departure from the CG implementation, a group type may support multiple share types, but a group may only contain shares of a single type. This restriction could be revisited in a later release, but it avoids numerous challenges:

* With multiple share types in a group, the number of potential share/group combinations becomes much larger and it becomes more likely that a group is requested that cannot be scheduled.
* Operators would have to test and support all share/group combinations.
* Migration becomes more complicated, especially if a group spans pools, since the destination would have to support the same combination of group & share types.
* Replication is similarly affected.
* Retype gets weird if some but not all group members must be migrated.

 We considered reusing share types for groups as well. But share types include a set of public extra specs that may not map well to groups. And by adding group types as a separate object, there can be little confusion about their purpose and use. 

Some have noted that extra specs tend to be treated as an AND operation, where all features must be available for a backend to be chosen. It is conceivable that even if a backend can replicate a group or take consistent snapshots of a group, it might not be able to perform both operations on the same group. But this problem already exists with share types and hasn’t been a serious issue. Creating types is an admin-only operation and the burden remains on the admin to understand the capabilities of the backends in use and to create the share types and share group types appropriately. 

Note that this proposal explicitly does not address pool or backend replication, which is fundamentally different. Actions on shares or share groups are intended for tenants, whereas a pool or backend can contain data from multiple tenants. So pool or backend operations, while serving potentially valuable use cases, are inherently admin-only workflows that would be designed and exercised differently should Manila support them.

=== Share Group APIs ===

It is possible to design a REST API that seamlessly handles both shares and share groups with little duplication of APIs. But at this point in Manila's development, it is arguably too late to radically redesign the API.

It may be possible to overload some of the existing APIs to handle both shares and share groups. For example, POST /shares/{id}/action could accept the ID of a share or share group and just do the right thing. But other APIs, such as GET /shares are less practical to overload, since a single endpoint would be returning objects of different types.

It seems better to merely duplicate a few APIs with group versions as needed. For example:
* POST /shares --> POST /groups
* POST /shares/{share_id}/action --> POST /groups/{group_id}/action
* POST /snapshots --> POST /group_snapshots

== Acknowledgments ==
Most of the analysis and recommendations contained herein apply equally well to Cinder, which is arguably much further down the path of crippling complexity. 

This proposal was developed by cknight with input from Manila community members bswartz, ameade, tbarron, and cfouts. The Cinder conversations about grouping, as well as the groundbreaking contributions of CGs to Cinder, all of which inspired this proposal, were led by xyang.

Manila/design/manila-generic-groups

2016-04-22T20:05:33Z

Clinton Knight: /* Implementation details */

= Manila generic groups =
== Executive summary ==
Manila needs a grouping construct that, like shares, is a 1st-class atomic data type. Our experience with CGs has demonstrated the complexity of adding a grouping capability, yet there are other use cases such as migration, replication, and backup in which some storage controllers could only offer such features on share groups. CGs also highlighted the poor optics of an advanced feature with comparatively little potential for vendor support. And adding new grouping constructs for each new feature is not technically feasible. All of the above may be addressed by generic groups, which we think is a clean extension to the original architecture of Manila.
== An Intractable Matrix ==
Manila is at an exciting time in its development, where all the simple features are available and the community is focused on adding high-value features such as migration, replication, and consistency groups. Experimental code is available for each of these, but little consideration has been given to the obvious need for these complex features to interact not only with each other but also with whatever additional features were added later. And looking a little deeper, it is apparent that each of these has limitations that could be solved by a single architectural enhancement.
=== Consistency groups ===
Consistency groups (CGs) are the only construct in Manila that operates on a group of shares. For some, a CG implies a guarantee of ordered writes by a storage controller, while for others a CG is a mechanism for taking consistent point-in-time snapshots of a set of shares. A group with either attribute may have value by itself, but the CGs implementation doesn’t distinguish between them. 

CGs are a highly specialized construct with dedicated CLI commands, REST APIs, database tables, scheduler filters, and driver APIs that totaled over 9900 lines. None of these are reusable for anything else, and none of the rest of Manila has any awareness of CGs. Even worse, despite all the complexity of the CG feature, only a small minority of storage backends can support them, so the code-to-value ratio is very low and the limited availability of CGs ensures the user experience is inconsistent between clouds.

=== Replication ===
Replication is another high value feature, and the Manila implementation is arguably clean and flexible, but as constituted there is no core ability to replicate groups of shares. It seems reasonable to implement the feature iteratively, beginning with replication of single shares. However, there has been the tacit acknowledgement that some backends would not be able to replicate individual shares, even though those same backends could potentially replicate a group of shares if the group were constituted in a certain way. 

Following the precedent established with CGs, one approach would be to introduce ‘replication groups’ whereby multiple shares could be replicated together. But this would introduce yet another set of APIs, tables, etc. that are too specialized to use for anything else. 

Another approach might be to define CGs as the grouping unit of replication, but this makes little sense because consistency groups (however we define them) and replication are distinct features that shouldn’t be artificially bundled for the sake of expediency of definition or implementation.
=== Migration ===
Just as some backends may only be able to replicate shares in groups, it follows that those backends may also need to migrate shares in groups. And in the case of CGs, it doesn’t make sense to migrate CG members individually; the whole group must be moved, requiring the migration engine to be CG aware. 

Consider the following reasonable use cases:
* Replicate a consistency group
* Snapshot a replication group
* Migrate a replication group
* Retype shares in a consistency group
* Retype shares in a replication group
* Backup a consistency group
* Backup a replication group
* Apply a common policy to a set of shares
 
On the current path, the support matrix has a dimension dedicated solely to features, so each feature must be coded to be interoperable with every other feature. This quickly becomes a support and testing nightmare, and it becomes exponentially more complicated to add more features going forward. 

Fundamentally, the problem is that we are adding features without an underlying architectural framework on which to hang them. To escape the matrix, we must step back and rethink a few things.

== A simple solution ==
So to arrive at a solution, let’s enumerate what we have:
* Primitive objects (shares) with supported operations controlled by share types that vary by backend
* A very specific group object (CGs) with limited support potential by backends
* A number of APIs (CGs, etc.) that many users can’t use at all and appear grafted into the project
 
And what would we prefer:
* A uniform API and user experience that varies as little as possible by backend
* A clean way to group shares so we can do CGs, multi-object replication, migration, etc.
 
How do we get here?
* Universally available groups of primitive objects with supported operations that vary by backend
 
=== Idea #1: Introduce a generic Share Group object ===
Just as Manila has ‘shares’, it should also have ‘share groups’. In its simplest form, unlike CGs, a share group should not guarantee any specialized operation. Instead, it should merely constitute an atomic Manila data type on which nearly any Manila action is available. For example, given a share group, the user should be able to select the group in the Manila UI and invoke features such as snapshot, clone, backup, migrate, replicate, retype, etc. 

In the general case, group actions are handled by the share manager layer (or potentially the common share driver superclass). For example, invoking ‘snapshot’ on a group causes the share manager to take a snapshot of each group member individually. The resulting group snapshot object may be used to create a new group, not unlike how CG snapshots were implemented. 

There are numerous advantages to this approach. Every driver, no matter how unsophisticated, can leverage the manageability goodness of groups. The user experience is uniform, since groups are always available regardless of which backends are present and because nearly every Manila action available on shares is also available on groups. 

Of course, feature-rich backends would like to add their secret sauce, whether it be CGs, group-based replication, or whatever else comes along that might be easier/cheaper/faster to do to groups of shares. That brings us to a related idea.

=== Idea #2: Introduce Share Group types ===
Just as Manila has ‘share types’, it should also have ‘share group types’. Any driver that can perform a group operation in an advantaged way may report that as a group capability, such as:
* Ordered writes
* Consistent snapshots
* Group replication
* Group backup
* Group modification
 
As with share types, the cloud administrator predefines share group types that may contain extra specs corresponding to the group capabilities reported by the backends. The admin also specifies which share type(s) a given group type may contain. When creating a group, the user specifies the group type and share type, and the scheduler then creates the group on one of the backends that match the specified share and share group type. 

Anytime a group action, such as ‘snapshot’, comes into the share manager, the manager checks whether its driver offers an advantaged implementation of that operation. If not, the manager handles the workflow itself as described above. But if so, the manager routes the workflow to its driver for fulfillment. 

The advantages of this approach should be obvious. Development and testing are simplified because there isn’t a need to define and test a different set of group management APIs for each feature, or to test every combination of every feature. Instead of becoming an N-by-N matrix of interacting features, Manila largely becomes an N-by-2 matrix of actions that may be invoked on either individual shares or share groups. Users and admins are already familiar with share types, so introducing share group types would seem a natural and consistent evolution of the same foundational concept.

The matrix would look like:
{| class="wikitable"
|-
! Share action !! Share Group action
|-
| Create (volume type) || Create (volume types, group type)
|-
| Delete || Delete (group, optionally all members)
|-
| Snapshot || Snapshot (may or may not be a CG snapshot)
|-
| Create from snapshot / Clone || Create from group snapshot / Clone group
|-
| Replicate || Replicate
|-
| Backup || Backup
|-
| Retype || Retype
|-
| Migrate || Migrate
|-
| Extend / shrink || N/A
|}

== Manila implementation plan ==
Implementing generic groups in Manila should be a straightforward series of steps:
# Implement generic groups. Because we did CGs, we already know all parts of the codebase that must change to support any kind of group. So the simplest approach is to modify the CG code to morph it into the generic groups feature.
# Add the share group type feature by duplicating and customizing the share type code.
# Enhance the scheduler to place groups according to share group types. Like #1, this is already informed by the CG project.
# Implement group snapshots in the share manager to demonstrate group snapshots in any driver.
# Plumb group snapshots to a CG-capable driver to demonstrate CG functionality in the new framework.
# Update Tempest to cover all of the above.
# Update Manila client to change CGs to share groups.
# Enhance Manila client with share group types.
# Add share group support to Horizon. We never built CG support into Horizon, so this is all-new work that now has much broader appeal and applicability.
# Add additional group actions over time (migrate, replicate, retype, clone, …). At this point, new group capabilities become vertical slices that are simple to add incrementally.
 
Because we implemented CGs just recently as an experimental feature, we have the freedom to replace that code without deprecation or upgrade considerations. Steps 1-8 would get Manila to parity with the CG feature added in Liberty, would require only a few person-weeks of effort, and would better position Manila for long-term evolution and supportability.
=== Implementation details ===
There are few things to note, several of which were already solved during the CG work. 

Groups are first-class objects in Manila, and operations on groups are treated as atomic. To enable support by as many backends as possible, Manila will still maintain DB objects for both group and member snapshots, just as was done with CGs. 

A few actions, such as extend & shrink, are inherently applicable only to individual shares. One could theoretically apply extend to a group, increasing the size of each member, but that seems like an unlikely use case. Any actions in this category must remain available to group members, and other actions such as taking snapshots of group members could be allowed, but actions such as migration or replication would be available only at the group level and not on its members. 

A group is limited to a single backend. Allowing groups that span backends is theoretically possible, but that would require fanout of operations from the API layer to multiple share managers across the asynchronous event bus, which would lead to complicated synchronization and state management for little operational benefit. 

As was done with Manila CGs, a driver may optionally limit a group to either the confines of a pool or an entire backend. It is known that pools are the unit of data motion (i.e. replication or migration) for some backends, so we think drivers need this flexibility. 

In a departure from the CG implementation, a group type may support multiple share types, but a group may only contain shares of a single type. This restriction could be revisited in a later release, but it avoids numerous challenges:

* With multiple share types in a group, the number of potential share/group combinations becomes much larger and it becomes more likely that a group is requested that cannot be scheduled.
* Operators would have to test and support all share/group combinations.
* Migration becomes more complicated, especially if a group spans pools, since the destination would have to support the same combination of group & share types.
* Replication is similarly affected.
* Retype gets weird if some but not all group members must be migrated.

 We considered reusing share types for groups as well. But share types include a set of public extra specs that may not map well to groups. And by adding group types as a separate object, there can be little confusion about their purpose and use. 

Some have noted that extra specs tend to be treated as an AND operation, where all features must be available for a backend to be chosen. It is conceivable that even if a backend can replicate a group or take consistent snapshots of a group, it might not be able to perform both operations on the same group. But this problem already exists with share types and hasn’t been a serious issue. Creating types is an admin-only operation and the burden remains on the admin to understand the capabilities of the backends in use and to create the share types and share group types appropriately. 

Note that this proposal explicitly does not address pool or backend replication, which is fundamentally different. Actions on shares or share groups are intended for tenants, whereas a pool or backend can contain data from multiple tenants. So pool or backend operations, while serving potentially valuable use cases, are inherently admin-only workflows that would be designed and exercised differently should Manila support them.

=== Share Group APIs ===

It is possible to design a REST API that seamlessly handles both shares and share groups with little duplication of APIs. But at this point in Manila's development, it is arguably too late to radically redesign the API.

It may be possible to overload some of the existing APIs to handle both shares and share groups. For example, POST /shares/{id}/action could accept the ID of a share or share group and just do the right thing. But other APIs, such as GET /shares are less practical to overload, since a single endpoint would be returning objects of different types.

It seems better to merely duplicate a few APIs with group versions as needed. For example:
* POST /shares --> POST /groups
* POST /shares/{share_id}/action --> POST /groups/{group_id}/action
* POST /snapshots --> POST /group_snapshots

== Acknowledgments ==
Most of the analysis and recommendations contained herein apply equally well to Cinder, which is arguably much further down the path of crippling complexity. 

This proposal was developed by cknight with input from Manila community members bswartz, ameade, tbarron, and cfouts. The Cinder conversations about grouping, as well as the groundbreaking contributions of CGs to Cinder, all of which inspired this proposal, were led by xyang.

Manila/design/manila-generic-groups

2016-04-22T20:04:55Z

Clinton Knight: /* Implementation details */

= Manila generic groups =
== Executive summary ==
Manila needs a grouping construct that, like shares, is a 1st-class atomic data type. Our experience with CGs has demonstrated the complexity of adding a grouping capability, yet there are other use cases such as migration, replication, and backup in which some storage controllers could only offer such features on share groups. CGs also highlighted the poor optics of an advanced feature with comparatively little potential for vendor support. And adding new grouping constructs for each new feature is not technically feasible. All of the above may be addressed by generic groups, which we think is a clean extension to the original architecture of Manila.
== An Intractable Matrix ==
Manila is at an exciting time in its development, where all the simple features are available and the community is focused on adding high-value features such as migration, replication, and consistency groups. Experimental code is available for each of these, but little consideration has been given to the obvious need for these complex features to interact not only with each other but also with whatever additional features were added later. And looking a little deeper, it is apparent that each of these has limitations that could be solved by a single architectural enhancement.
=== Consistency groups ===
Consistency groups (CGs) are the only construct in Manila that operates on a group of shares. For some, a CG implies a guarantee of ordered writes by a storage controller, while for others a CG is a mechanism for taking consistent point-in-time snapshots of a set of shares. A group with either attribute may have value by itself, but the CGs implementation doesn’t distinguish between them. 

CGs are a highly specialized construct with dedicated CLI commands, REST APIs, database tables, scheduler filters, and driver APIs that totaled over 9900 lines. None of these are reusable for anything else, and none of the rest of Manila has any awareness of CGs. Even worse, despite all the complexity of the CG feature, only a small minority of storage backends can support them, so the code-to-value ratio is very low and the limited availability of CGs ensures the user experience is inconsistent between clouds.

=== Replication ===
Replication is another high value feature, and the Manila implementation is arguably clean and flexible, but as constituted there is no core ability to replicate groups of shares. It seems reasonable to implement the feature iteratively, beginning with replication of single shares. However, there has been the tacit acknowledgement that some backends would not be able to replicate individual shares, even though those same backends could potentially replicate a group of shares if the group were constituted in a certain way. 

Following the precedent established with CGs, one approach would be to introduce ‘replication groups’ whereby multiple shares could be replicated together. But this would introduce yet another set of APIs, tables, etc. that are too specialized to use for anything else. 

Another approach might be to define CGs as the grouping unit of replication, but this makes little sense because consistency groups (however we define them) and replication are distinct features that shouldn’t be artificially bundled for the sake of expediency of definition or implementation.
=== Migration ===
Just as some backends may only be able to replicate shares in groups, it follows that those backends may also need to migrate shares in groups. And in the case of CGs, it doesn’t make sense to migrate CG members individually; the whole group must be moved, requiring the migration engine to be CG aware. 

Consider the following reasonable use cases:
* Replicate a consistency group
* Snapshot a replication group
* Migrate a replication group
* Retype shares in a consistency group
* Retype shares in a replication group
* Backup a consistency group
* Backup a replication group
* Apply a common policy to a set of shares
 
On the current path, the support matrix has a dimension dedicated solely to features, so each feature must be coded to be interoperable with every other feature. This quickly becomes a support and testing nightmare, and it becomes exponentially more complicated to add more features going forward. 

Fundamentally, the problem is that we are adding features without an underlying architectural framework on which to hang them. To escape the matrix, we must step back and rethink a few things.

== A simple solution ==
So to arrive at a solution, let’s enumerate what we have:
* Primitive objects (shares) with supported operations controlled by share types that vary by backend
* A very specific group object (CGs) with limited support potential by backends
* A number of APIs (CGs, etc.) that many users can’t use at all and appear grafted into the project
 
And what would we prefer:
* A uniform API and user experience that varies as little as possible by backend
* A clean way to group shares so we can do CGs, multi-object replication, migration, etc.
 
How do we get here?
* Universally available groups of primitive objects with supported operations that vary by backend
 
=== Idea #1: Introduce a generic Share Group object ===
Just as Manila has ‘shares’, it should also have ‘share groups’. In its simplest form, unlike CGs, a share group should not guarantee any specialized operation. Instead, it should merely constitute an atomic Manila data type on which nearly any Manila action is available. For example, given a share group, the user should be able to select the group in the Manila UI and invoke features such as snapshot, clone, backup, migrate, replicate, retype, etc. 

In the general case, group actions are handled by the share manager layer (or potentially the common share driver superclass). For example, invoking ‘snapshot’ on a group causes the share manager to take a snapshot of each group member individually. The resulting group snapshot object may be used to create a new group, not unlike how CG snapshots were implemented. 

There are numerous advantages to this approach. Every driver, no matter how unsophisticated, can leverage the manageability goodness of groups. The user experience is uniform, since groups are always available regardless of which backends are present and because nearly every Manila action available on shares is also available on groups. 

Of course, feature-rich backends would like to add their secret sauce, whether it be CGs, group-based replication, or whatever else comes along that might be easier/cheaper/faster to do to groups of shares. That brings us to a related idea.

=== Idea #2: Introduce Share Group types ===
Just as Manila has ‘share types’, it should also have ‘share group types’. Any driver that can perform a group operation in an advantaged way may report that as a group capability, such as:
* Ordered writes
* Consistent snapshots
* Group replication
* Group backup
* Group modification
 
As with share types, the cloud administrator predefines share group types that may contain extra specs corresponding to the group capabilities reported by the backends. The admin also specifies which share type(s) a given group type may contain. When creating a group, the user specifies the group type and share type, and the scheduler then creates the group on one of the backends that match the specified share and share group type. 

Anytime a group action, such as ‘snapshot’, comes into the share manager, the manager checks whether its driver offers an advantaged implementation of that operation. If not, the manager handles the workflow itself as described above. But if so, the manager routes the workflow to its driver for fulfillment. 

The advantages of this approach should be obvious. Development and testing are simplified because there isn’t a need to define and test a different set of group management APIs for each feature, or to test every combination of every feature. Instead of becoming an N-by-N matrix of interacting features, Manila largely becomes an N-by-2 matrix of actions that may be invoked on either individual shares or share groups. Users and admins are already familiar with share types, so introducing share group types would seem a natural and consistent evolution of the same foundational concept.

The matrix would look like:
{| class="wikitable"
|-
! Share action !! Share Group action
|-
| Create (volume type) || Create (volume types, group type)
|-
| Delete || Delete (group, optionally all members)
|-
| Snapshot || Snapshot (may or may not be a CG snapshot)
|-
| Create from snapshot / Clone || Create from group snapshot / Clone group
|-
| Replicate || Replicate
|-
| Backup || Backup
|-
| Retype || Retype
|-
| Migrate || Migrate
|-
| Extend / shrink || N/A
|}

== Manila implementation plan ==
Implementing generic groups in Manila should be a straightforward series of steps:
# Implement generic groups. Because we did CGs, we already know all parts of the codebase that must change to support any kind of group. So the simplest approach is to modify the CG code to morph it into the generic groups feature.
# Add the share group type feature by duplicating and customizing the share type code.
# Enhance the scheduler to place groups according to share group types. Like #1, this is already informed by the CG project.
# Implement group snapshots in the share manager to demonstrate group snapshots in any driver.
# Plumb group snapshots to a CG-capable driver to demonstrate CG functionality in the new framework.
# Update Tempest to cover all of the above.
# Update Manila client to change CGs to share groups.
# Enhance Manila client with share group types.
# Add share group support to Horizon. We never built CG support into Horizon, so this is all-new work that now has much broader appeal and applicability.
# Add additional group actions over time (migrate, replicate, retype, clone, …). At this point, new group capabilities become vertical slices that are simple to add incrementally.
 
Because we implemented CGs just recently as an experimental feature, we have the freedom to replace that code without deprecation or upgrade considerations. Steps 1-8 would get Manila to parity with the CG feature added in Liberty, would require only a few person-weeks of effort, and would better position Manila for long-term evolution and supportability.
=== Implementation details ===
There are few things to note, several of which were already solved during the CG work. 

Groups are first-class objects in Manila, and operations on groups are treated as atomic. To enable support by as many backends as possible, Manila will still maintain DB objects for both group and member snapshots, just as was done with CGs. 

A few actions, such as extend & shrink, are inherently applicable only to individual shares. One could theoretically apply extend to a group, increasing the size of each member, but that seems like an unlikely use case. Any actions in this category must remain available to group members, and other actions such as taking snapshots of group members could be allowed, but actions such as migration or replication would be available only at the group level and not on its members. 

A group is limited to a single backend. Allowing groups that span backends is theoretically possible, but that would require fanout of operations from the API layer to multiple share managers across the asynchronous event bus, which would lead to complicated synchronization and state management for little operational benefit. 

As was done with Manila CGs, a driver may optionally limit a group to either the confines of a pool or an entire backend. It is known that pools are the unit of data motion (i.e. replication or migration) for some backends, so we think drivers need this flexibility. 

In a departure from the CG implementation, a group type may support multiple share types, but a group may only contain shares of a single type. This restriction could be lifted in a later release, but it avoids numerous challenges:

* With multiple share types in a group, the number of potential share/group combinations becomes much larger and it becomes more likely that a group is requested that cannot be scheduled.
* Operators would have to test and support all share/group combinations.
* Migration becomes more complicated, especially if a group spans pools, since the destination would have to support the same combination of group & share types.
* Replication is similarly affected.
* Retype gets weird if some but not all group members must be migrated.

 We considered reusing share types for groups as well. But share types include a set of public extra specs that may not map well to groups. And by adding group types as a separate object, there can be little confusion about their purpose and use. 

Some have noted that extra specs tend to be treated as an AND operation, where all features must be available for a backend to be chosen. It is conceivable that even if a backend can replicate a group or take consistent snapshots of a group, it might not be able to perform both operations on the same group. But this problem already exists with share types and hasn’t been a serious issue. Creating types is an admin-only operation and the burden remains on the admin to understand the capabilities of the backends in use and to create the share types and share group types appropriately. 

Note that this proposal explicitly does not address pool or backend replication, which is fundamentally different. Actions on shares or share groups are intended for tenants, whereas a pool or backend can contain data from multiple tenants. So pool or backend operations, while serving potentially valuable use cases, are inherently admin-only workflows that would be designed and exercised differently should Manila support them.

=== Share Group APIs ===

It is possible to design a REST API that seamlessly handles both shares and share groups with little duplication of APIs. But at this point in Manila's development, it is arguably too late to radically redesign the API.

It may be possible to overload some of the existing APIs to handle both shares and share groups. For example, POST /shares/{id}/action could accept the ID of a share or share group and just do the right thing. But other APIs, such as GET /shares are less practical to overload, since a single endpoint would be returning objects of different types.

It seems better to merely duplicate a few APIs with group versions as needed. For example:
* POST /shares --> POST /groups
* POST /shares/{share_id}/action --> POST /groups/{group_id}/action
* POST /snapshots --> POST /group_snapshots

== Acknowledgments ==
Most of the analysis and recommendations contained herein apply equally well to Cinder, which is arguably much further down the path of crippling complexity. 

This proposal was developed by cknight with input from Manila community members bswartz, ameade, tbarron, and cfouts. The Cinder conversations about grouping, as well as the groundbreaking contributions of CGs to Cinder, all of which inspired this proposal, were led by xyang.

Manila/design/manila-generic-groups

2016-04-22T20:03:29Z

Clinton Knight: /* Implementation details */

= Manila generic groups =
== Executive summary ==
Manila needs a grouping construct that, like shares, is a 1st-class atomic data type. Our experience with CGs has demonstrated the complexity of adding a grouping capability, yet there are other use cases such as migration, replication, and backup in which some storage controllers could only offer such features on share groups. CGs also highlighted the poor optics of an advanced feature with comparatively little potential for vendor support. And adding new grouping constructs for each new feature is not technically feasible. All of the above may be addressed by generic groups, which we think is a clean extension to the original architecture of Manila.
== An Intractable Matrix ==
Manila is at an exciting time in its development, where all the simple features are available and the community is focused on adding high-value features such as migration, replication, and consistency groups. Experimental code is available for each of these, but little consideration has been given to the obvious need for these complex features to interact not only with each other but also with whatever additional features were added later. And looking a little deeper, it is apparent that each of these has limitations that could be solved by a single architectural enhancement.
=== Consistency groups ===
Consistency groups (CGs) are the only construct in Manila that operates on a group of shares. For some, a CG implies a guarantee of ordered writes by a storage controller, while for others a CG is a mechanism for taking consistent point-in-time snapshots of a set of shares. A group with either attribute may have value by itself, but the CGs implementation doesn’t distinguish between them. 

CGs are a highly specialized construct with dedicated CLI commands, REST APIs, database tables, scheduler filters, and driver APIs that totaled over 9900 lines. None of these are reusable for anything else, and none of the rest of Manila has any awareness of CGs. Even worse, despite all the complexity of the CG feature, only a small minority of storage backends can support them, so the code-to-value ratio is very low and the limited availability of CGs ensures the user experience is inconsistent between clouds.

=== Replication ===
Replication is another high value feature, and the Manila implementation is arguably clean and flexible, but as constituted there is no core ability to replicate groups of shares. It seems reasonable to implement the feature iteratively, beginning with replication of single shares. However, there has been the tacit acknowledgement that some backends would not be able to replicate individual shares, even though those same backends could potentially replicate a group of shares if the group were constituted in a certain way. 

Following the precedent established with CGs, one approach would be to introduce ‘replication groups’ whereby multiple shares could be replicated together. But this would introduce yet another set of APIs, tables, etc. that are too specialized to use for anything else. 

Another approach might be to define CGs as the grouping unit of replication, but this makes little sense because consistency groups (however we define them) and replication are distinct features that shouldn’t be artificially bundled for the sake of expediency of definition or implementation.
=== Migration ===
Just as some backends may only be able to replicate shares in groups, it follows that those backends may also need to migrate shares in groups. And in the case of CGs, it doesn’t make sense to migrate CG members individually; the whole group must be moved, requiring the migration engine to be CG aware. 

Consider the following reasonable use cases:
* Replicate a consistency group
* Snapshot a replication group
* Migrate a replication group
* Retype shares in a consistency group
* Retype shares in a replication group
* Backup a consistency group
* Backup a replication group
* Apply a common policy to a set of shares
 
On the current path, the support matrix has a dimension dedicated solely to features, so each feature must be coded to be interoperable with every other feature. This quickly becomes a support and testing nightmare, and it becomes exponentially more complicated to add more features going forward. 

Fundamentally, the problem is that we are adding features without an underlying architectural framework on which to hang them. To escape the matrix, we must step back and rethink a few things.

== A simple solution ==
So to arrive at a solution, let’s enumerate what we have:
* Primitive objects (shares) with supported operations controlled by share types that vary by backend
* A very specific group object (CGs) with limited support potential by backends
* A number of APIs (CGs, etc.) that many users can’t use at all and appear grafted into the project
 
And what would we prefer:
* A uniform API and user experience that varies as little as possible by backend
* A clean way to group shares so we can do CGs, multi-object replication, migration, etc.
 
How do we get here?
* Universally available groups of primitive objects with supported operations that vary by backend
 
=== Idea #1: Introduce a generic Share Group object ===
Just as Manila has ‘shares’, it should also have ‘share groups’. In its simplest form, unlike CGs, a share group should not guarantee any specialized operation. Instead, it should merely constitute an atomic Manila data type on which nearly any Manila action is available. For example, given a share group, the user should be able to select the group in the Manila UI and invoke features such as snapshot, clone, backup, migrate, replicate, retype, etc. 

In the general case, group actions are handled by the share manager layer (or potentially the common share driver superclass). For example, invoking ‘snapshot’ on a group causes the share manager to take a snapshot of each group member individually. The resulting group snapshot object may be used to create a new group, not unlike how CG snapshots were implemented. 

There are numerous advantages to this approach. Every driver, no matter how unsophisticated, can leverage the manageability goodness of groups. The user experience is uniform, since groups are always available regardless of which backends are present and because nearly every Manila action available on shares is also available on groups. 

Of course, feature-rich backends would like to add their secret sauce, whether it be CGs, group-based replication, or whatever else comes along that might be easier/cheaper/faster to do to groups of shares. That brings us to a related idea.

=== Idea #2: Introduce Share Group types ===
Just as Manila has ‘share types’, it should also have ‘share group types’. Any driver that can perform a group operation in an advantaged way may report that as a group capability, such as:
* Ordered writes
* Consistent snapshots
* Group replication
* Group backup
* Group modification
 
As with share types, the cloud administrator predefines share group types that may contain extra specs corresponding to the group capabilities reported by the backends. The admin also specifies which share type(s) a given group type may contain. When creating a group, the user specifies the group type and share type, and the scheduler then creates the group on one of the backends that match the specified share and share group type. 

Anytime a group action, such as ‘snapshot’, comes into the share manager, the manager checks whether its driver offers an advantaged implementation of that operation. If not, the manager handles the workflow itself as described above. But if so, the manager routes the workflow to its driver for fulfillment. 

The advantages of this approach should be obvious. Development and testing are simplified because there isn’t a need to define and test a different set of group management APIs for each feature, or to test every combination of every feature. Instead of becoming an N-by-N matrix of interacting features, Manila largely becomes an N-by-2 matrix of actions that may be invoked on either individual shares or share groups. Users and admins are already familiar with share types, so introducing share group types would seem a natural and consistent evolution of the same foundational concept.

The matrix would look like:
{| class="wikitable"
|-
! Share action !! Share Group action
|-
| Create (volume type) || Create (volume types, group type)
|-
| Delete || Delete (group, optionally all members)
|-
| Snapshot || Snapshot (may or may not be a CG snapshot)
|-
| Create from snapshot / Clone || Create from group snapshot / Clone group
|-
| Replicate || Replicate
|-
| Backup || Backup
|-
| Retype || Retype
|-
| Migrate || Migrate
|-
| Extend / shrink || N/A
|}

== Manila implementation plan ==
Implementing generic groups in Manila should be a straightforward series of steps:
# Implement generic groups. Because we did CGs, we already know all parts of the codebase that must change to support any kind of group. So the simplest approach is to modify the CG code to morph it into the generic groups feature.
# Add the share group type feature by duplicating and customizing the share type code.
# Enhance the scheduler to place groups according to share group types. Like #1, this is already informed by the CG project.
# Implement group snapshots in the share manager to demonstrate group snapshots in any driver.
# Plumb group snapshots to a CG-capable driver to demonstrate CG functionality in the new framework.
# Update Tempest to cover all of the above.
# Update Manila client to change CGs to share groups.
# Enhance Manila client with share group types.
# Add share group support to Horizon. We never built CG support into Horizon, so this is all-new work that now has much broader appeal and applicability.
# Add additional group actions over time (migrate, replicate, retype, clone, …). At this point, new group capabilities become vertical slices that are simple to add incrementally.
 
Because we implemented CGs just recently as an experimental feature, we have the freedom to replace that code without deprecation or upgrade considerations. Steps 1-8 would get Manila to parity with the CG feature added in Liberty, would require only a few person-weeks of effort, and would better position Manila for long-term evolution and supportability.
=== Implementation details ===
There are few things to note, several of which were already solved during the CG work. 

Groups are first-class objects in Manila, and operations on groups are treated as atomic. To enable support by as many backends as possible, Manila will still maintain DB objects for both group and member snapshots, just as was done with CGs. 

A few actions, such as extend & shrink, are inherently applicable only to individual shares. One could theoretically apply extend to a group, increasing the size of each member, but that seems like an unlikely use case. Any actions in this category must remain available to group members, and other actions such as taking snapshots of group members could be allowed, but other actions such as migration or replication would be available only at the group level and not on its members. 

A group is limited to a single backend. Allowing groups that span backends is theoretically possible, but that would require fanout of operations from the API layer to multiple share managers across the asynchronous event bus, which would lead to complicated synchronization and state management for little operational benefit. 

As was done with Manila CGs, a driver may optionally limit a group to either the confines of a pool or an entire backend. It is known that pools are the unit of data motion (i.e. replication or migration) for some backends, so we think drivers need this flexibility. 

In a departure from the CG implementation, a group type may support multiple share types, but a group may only contain shares of a single type. This restriction could be lifted in a later release, but it avoids numerous challenges:

* With multiple share types in a group, the number of potential share/group combinations becomes much larger and it becomes more likely that a group is requested that cannot be scheduled.
* Operators would have to test and support all share/group combinations.
* Migration becomes more complicated, especially if a group spans pools, since the destination would have to support the same combination of group & share types.
* Replication is similarly affected.
* Retype gets weird if some but not all group members must be migrated.

 We considered reusing share types for groups as well. But share types include a set of public extra specs that may not map well to groups. And by adding group types as a separate object, there can be little confusion about their purpose and use. 

Some have noted that extra specs tend to be treated as an AND operation, where all features must be available for a backend to be chosen. It is conceivable that even if a backend can replicate a group or take consistent snapshots of a group, it might not be able to perform both operations on the same group. But this problem already exists with share types and hasn’t been a serious issue. Creating types is an admin-only operation and the burden remains on the admin to understand the capabilities of the backends in use and to create the share types and share group types appropriately. 

Note that this proposal explicitly does not address pool or backend replication, which is fundamentally different. Actions on shares or share groups are intended for tenants, whereas a pool or backend can contain data from multiple tenants. So pool or backend operations, while serving potentially valuable use cases, are inherently admin-only workflows that would be designed and exercised differently should Manila support them.

=== Share Group APIs ===

It is possible to design a REST API that seamlessly handles both shares and share groups with little duplication of APIs. But at this point in Manila's development, it is arguably too late to radically redesign the API.

It may be possible to overload some of the existing APIs to handle both shares and share groups. For example, POST /shares/{id}/action could accept the ID of a share or share group and just do the right thing. But other APIs, such as GET /shares are less practical to overload, since a single endpoint would be returning objects of different types.

It seems better to merely duplicate a few APIs with group versions as needed. For example:
* POST /shares --> POST /groups
* POST /shares/{share_id}/action --> POST /groups/{group_id}/action
* POST /snapshots --> POST /group_snapshots

== Acknowledgments ==
Most of the analysis and recommendations contained herein apply equally well to Cinder, which is arguably much further down the path of crippling complexity. 

This proposal was developed by cknight with input from Manila community members bswartz, ameade, tbarron, and cfouts. The Cinder conversations about grouping, as well as the groundbreaking contributions of CGs to Cinder, all of which inspired this proposal, were led by xyang.

Manila/design/manila-generic-groups

2016-04-22T19:49:18Z

Clinton Knight: /* Idea #2: Introduce Share Group types */

= Manila generic groups =
== Executive summary ==
Manila needs a grouping construct that, like shares, is a 1st-class atomic data type. Our experience with CGs has demonstrated the complexity of adding a grouping capability, yet there are other use cases such as migration, replication, and backup in which some storage controllers could only offer such features on share groups. CGs also highlighted the poor optics of an advanced feature with comparatively little potential for vendor support. And adding new grouping constructs for each new feature is not technically feasible. All of the above may be addressed by generic groups, which we think is a clean extension to the original architecture of Manila.
== An Intractable Matrix ==
Manila is at an exciting time in its development, where all the simple features are available and the community is focused on adding high-value features such as migration, replication, and consistency groups. Experimental code is available for each of these, but little consideration has been given to the obvious need for these complex features to interact not only with each other but also with whatever additional features were added later. And looking a little deeper, it is apparent that each of these has limitations that could be solved by a single architectural enhancement.
=== Consistency groups ===
Consistency groups (CGs) are the only construct in Manila that operates on a group of shares. For some, a CG implies a guarantee of ordered writes by a storage controller, while for others a CG is a mechanism for taking consistent point-in-time snapshots of a set of shares. A group with either attribute may have value by itself, but the CGs implementation doesn’t distinguish between them. 

CGs are a highly specialized construct with dedicated CLI commands, REST APIs, database tables, scheduler filters, and driver APIs that totaled over 9900 lines. None of these are reusable for anything else, and none of the rest of Manila has any awareness of CGs. Even worse, despite all the complexity of the CG feature, only a small minority of storage backends can support them, so the code-to-value ratio is very low and the limited availability of CGs ensures the user experience is inconsistent between clouds.

=== Replication ===
Replication is another high value feature, and the Manila implementation is arguably clean and flexible, but as constituted there is no core ability to replicate groups of shares. It seems reasonable to implement the feature iteratively, beginning with replication of single shares. However, there has been the tacit acknowledgement that some backends would not be able to replicate individual shares, even though those same backends could potentially replicate a group of shares if the group were constituted in a certain way. 

Following the precedent established with CGs, one approach would be to introduce ‘replication groups’ whereby multiple shares could be replicated together. But this would introduce yet another set of APIs, tables, etc. that are too specialized to use for anything else. 

Another approach might be to define CGs as the grouping unit of replication, but this makes little sense because consistency groups (however we define them) and replication are distinct features that shouldn’t be artificially bundled for the sake of expediency of definition or implementation.
=== Migration ===
Just as some backends may only be able to replicate shares in groups, it follows that those backends may also need to migrate shares in groups. And in the case of CGs, it doesn’t make sense to migrate CG members individually; the whole group must be moved, requiring the migration engine to be CG aware. 

Consider the following reasonable use cases:
* Replicate a consistency group
* Snapshot a replication group
* Migrate a replication group
* Retype shares in a consistency group
* Retype shares in a replication group
* Backup a consistency group
* Backup a replication group
* Apply a common policy to a set of shares
 
On the current path, the support matrix has a dimension dedicated solely to features, so each feature must be coded to be interoperable with every other feature. This quickly becomes a support and testing nightmare, and it becomes exponentially more complicated to add more features going forward. 

Fundamentally, the problem is that we are adding features without an underlying architectural framework on which to hang them. To escape the matrix, we must step back and rethink a few things.

== A simple solution ==
So to arrive at a solution, let’s enumerate what we have:
* Primitive objects (shares) with supported operations controlled by share types that vary by backend
* A very specific group object (CGs) with limited support potential by backends
* A number of APIs (CGs, etc.) that many users can’t use at all and appear grafted into the project
 
And what would we prefer:
* A uniform API and user experience that varies as little as possible by backend
* A clean way to group shares so we can do CGs, multi-object replication, migration, etc.
 
How do we get here?
* Universally available groups of primitive objects with supported operations that vary by backend
 
=== Idea #1: Introduce a generic Share Group object ===
Just as Manila has ‘shares’, it should also have ‘share groups’. In its simplest form, unlike CGs, a share group should not guarantee any specialized operation. Instead, it should merely constitute an atomic Manila data type on which nearly any Manila action is available. For example, given a share group, the user should be able to select the group in the Manila UI and invoke features such as snapshot, clone, backup, migrate, replicate, retype, etc. 

In the general case, group actions are handled by the share manager layer (or potentially the common share driver superclass). For example, invoking ‘snapshot’ on a group causes the share manager to take a snapshot of each group member individually. The resulting group snapshot object may be used to create a new group, not unlike how CG snapshots were implemented. 

There are numerous advantages to this approach. Every driver, no matter how unsophisticated, can leverage the manageability goodness of groups. The user experience is uniform, since groups are always available regardless of which backends are present and because nearly every Manila action available on shares is also available on groups. 

Of course, feature-rich backends would like to add their secret sauce, whether it be CGs, group-based replication, or whatever else comes along that might be easier/cheaper/faster to do to groups of shares. That brings us to a related idea.

=== Idea #2: Introduce Share Group types ===
Just as Manila has ‘share types’, it should also have ‘share group types’. Any driver that can perform a group operation in an advantaged way may report that as a group capability, such as:
* Ordered writes
* Consistent snapshots
* Group replication
* Group backup
* Group modification
 
As with share types, the cloud administrator predefines share group types that may contain extra specs corresponding to the group capabilities reported by the backends. The admin also specifies which share type(s) a given group type may contain. When creating a group, the user specifies the group type and share type, and the scheduler then creates the group on one of the backends that match the specified share and share group type. 

Anytime a group action, such as ‘snapshot’, comes into the share manager, the manager checks whether its driver offers an advantaged implementation of that operation. If not, the manager handles the workflow itself as described above. But if so, the manager routes the workflow to its driver for fulfillment. 

The advantages of this approach should be obvious. Development and testing are simplified because there isn’t a need to define and test a different set of group management APIs for each feature, or to test every combination of every feature. Instead of becoming an N-by-N matrix of interacting features, Manila largely becomes an N-by-2 matrix of actions that may be invoked on either individual shares or share groups. Users and admins are already familiar with share types, so introducing share group types would seem a natural and consistent evolution of the same foundational concept.

The matrix would look like:
{| class="wikitable"
|-
! Share action !! Share Group action
|-
| Create (volume type) || Create (volume types, group type)
|-
| Delete || Delete (group, optionally all members)
|-
| Snapshot || Snapshot (may or may not be a CG snapshot)
|-
| Create from snapshot / Clone || Create from group snapshot / Clone group
|-
| Replicate || Replicate
|-
| Backup || Backup
|-
| Retype || Retype
|-
| Migrate || Migrate
|-
| Extend / shrink || N/A
|}

== Manila implementation plan ==
Implementing generic groups in Manila should be a straightforward series of steps:
# Implement generic groups. Because we did CGs, we already know all parts of the codebase that must change to support any kind of group. So the simplest approach is to modify the CG code to morph it into the generic groups feature.
# Add the share group type feature by duplicating and customizing the share type code.
# Enhance the scheduler to place groups according to share group types. Like #1, this is already informed by the CG project.
# Implement group snapshots in the share manager to demonstrate group snapshots in any driver.
# Plumb group snapshots to a CG-capable driver to demonstrate CG functionality in the new framework.
# Update Tempest to cover all of the above.
# Update Manila client to change CGs to share groups.
# Enhance Manila client with share group types.
# Add share group support to Horizon. We never built CG support into Horizon, so this is all-new work that now has much broader appeal and applicability.
# Add additional group actions over time (migrate, replicate, retype, clone, …). At this point, new group capabilities become vertical slices that are simple to add incrementally.
 
Because we implemented CGs just recently as an experimental feature, we have the freedom to replace that code without deprecation or upgrade considerations. Steps 1-8 would get Manila to parity with the CG feature added in Liberty, would require only a few person-weeks of effort, and would better position Manila for long-term evolution and supportability.
=== Implementation details ===
There are few things to note, several of which were already solved during the CG work. 

Groups are first-class objects in Manila, and operations on groups are treated as atomic. For example, a user will see a group snapshot but not the individual snapshots of group members. To enable support by as many backends as possible, Manila will still maintain DB objects for both group and member snapshots, just as was done with CGs. 

A few actions, such as extend & shrink, are inherently applicable only to individual shares. One could theoretically apply extend to a group, increasing the size of each member, but that seems like an unlikely use case. Any actions in this category must remain available to group members, although most actions such as clone or snapshot would be available only at the group level and not on its members. 

A group is limited to a single backend. Allowing groups that span backends is theoretically possible, but that would require fanout of operations from the API layer to multiple share managers across the asynchronous event bus, which would lead to complicated synchronization and state management for little operational benefit. 

As was done with Manila CGs, a driver may optionally limit a group to either the confines of a pool or an entire backend. It is known that pools are the unit of data motion (i.e. replication or migration) for some backends, so we think drivers need this flexibility. 

We considered reusing share types for groups as well. But share types include a set of public extra specs that may not map well to groups. And by adding group types as a separate object, there can be little confusion about their purpose and use. 

Some have noted that extra specs tend to be treated as an AND operation, where all features must be available for a backend to be chosen. It is conceivable that even if a backend can replicate a group or take consistent snapshots of a group, it might not be able to perform both operations on the same group. But this problem already exists with share types and hasn’t been a serious issue. Creating types is an admin-only operation and the burden remains on the admin to understand the capabilities of the backends in use and to create the share types and share group types appropriately. 

Note that this proposal explicitly does not address pool or backend replication, which is fundamentally different. Actions on shares or share groups are intended for tenants, whereas a pool or backend can contain data from multiple tenants. So pool or backend operations, while serving potentially valuable use cases, are inherently admin-only workflows that would be designed and exercised differently should Manila support them.

=== Share Group APIs ===

It is possible to design a REST API that seamlessly handles both shares and share groups with little duplication of APIs. But at this point in Manila's development, it is arguably too late to radically redesign the API.

It may be possible to overload some of the existing APIs to handle both shares and share groups. For example, POST /shares/{id}/action could accept the ID of a share or share group and just do the right thing. But other APIs, such as GET /shares are less practical to overload, since a single endpoint would be returning objects of different types.

It seems better to merely duplicate a few APIs with group versions as needed. For example:
* POST /shares --> POST /groups
* POST /shares/{share_id}/action --> POST /groups/{group_id}/action
* POST /snapshots --> POST /group_snapshots

== Acknowledgments ==
Most of the analysis and recommendations contained herein apply equally well to Cinder, which is arguably much further down the path of crippling complexity. 

This proposal was developed by cknight with input from Manila community members bswartz, ameade, tbarron, and cfouts. The Cinder conversations about grouping, as well as the groundbreaking contributions of CGs to Cinder, all of which inspired this proposal, were led by xyang.

Manila/design/manila-generic-groups

2016-04-22T19:45:53Z

Clinton Knight: /* Migration */

= Manila generic groups =
== Executive summary ==
Manila needs a grouping construct that, like shares, is a 1st-class atomic data type. Our experience with CGs has demonstrated the complexity of adding a grouping capability, yet there are other use cases such as migration, replication, and backup in which some storage controllers could only offer such features on share groups. CGs also highlighted the poor optics of an advanced feature with comparatively little potential for vendor support. And adding new grouping constructs for each new feature is not technically feasible. All of the above may be addressed by generic groups, which we think is a clean extension to the original architecture of Manila.
== An Intractable Matrix ==
Manila is at an exciting time in its development, where all the simple features are available and the community is focused on adding high-value features such as migration, replication, and consistency groups. Experimental code is available for each of these, but little consideration has been given to the obvious need for these complex features to interact not only with each other but also with whatever additional features were added later. And looking a little deeper, it is apparent that each of these has limitations that could be solved by a single architectural enhancement.
=== Consistency groups ===
Consistency groups (CGs) are the only construct in Manila that operates on a group of shares. For some, a CG implies a guarantee of ordered writes by a storage controller, while for others a CG is a mechanism for taking consistent point-in-time snapshots of a set of shares. A group with either attribute may have value by itself, but the CGs implementation doesn’t distinguish between them. 

CGs are a highly specialized construct with dedicated CLI commands, REST APIs, database tables, scheduler filters, and driver APIs that totaled over 9900 lines. None of these are reusable for anything else, and none of the rest of Manila has any awareness of CGs. Even worse, despite all the complexity of the CG feature, only a small minority of storage backends can support them, so the code-to-value ratio is very low and the limited availability of CGs ensures the user experience is inconsistent between clouds.

=== Replication ===
Replication is another high value feature, and the Manila implementation is arguably clean and flexible, but as constituted there is no core ability to replicate groups of shares. It seems reasonable to implement the feature iteratively, beginning with replication of single shares. However, there has been the tacit acknowledgement that some backends would not be able to replicate individual shares, even though those same backends could potentially replicate a group of shares if the group were constituted in a certain way. 

Following the precedent established with CGs, one approach would be to introduce ‘replication groups’ whereby multiple shares could be replicated together. But this would introduce yet another set of APIs, tables, etc. that are too specialized to use for anything else. 

Another approach might be to define CGs as the grouping unit of replication, but this makes little sense because consistency groups (however we define them) and replication are distinct features that shouldn’t be artificially bundled for the sake of expediency of definition or implementation.
=== Migration ===
Just as some backends may only be able to replicate shares in groups, it follows that those backends may also need to migrate shares in groups. And in the case of CGs, it doesn’t make sense to migrate CG members individually; the whole group must be moved, requiring the migration engine to be CG aware. 

Consider the following reasonable use cases:
* Replicate a consistency group
* Snapshot a replication group
* Migrate a replication group
* Retype shares in a consistency group
* Retype shares in a replication group
* Backup a consistency group
* Backup a replication group
* Apply a common policy to a set of shares
 
On the current path, the support matrix has a dimension dedicated solely to features, so each feature must be coded to be interoperable with every other feature. This quickly becomes a support and testing nightmare, and it becomes exponentially more complicated to add more features going forward. 

Fundamentally, the problem is that we are adding features without an underlying architectural framework on which to hang them. To escape the matrix, we must step back and rethink a few things.

== A simple solution ==
So to arrive at a solution, let’s enumerate what we have:
* Primitive objects (shares) with supported operations controlled by share types that vary by backend
* A very specific group object (CGs) with limited support potential by backends
* A number of APIs (CGs, etc.) that many users can’t use at all and appear grafted into the project
 
And what would we prefer:
* A uniform API and user experience that varies as little as possible by backend
* A clean way to group shares so we can do CGs, multi-object replication, migration, etc.
 
How do we get here?
* Universally available groups of primitive objects with supported operations that vary by backend
 
=== Idea #1: Introduce a generic Share Group object ===
Just as Manila has ‘shares’, it should also have ‘share groups’. In its simplest form, unlike CGs, a share group should not guarantee any specialized operation. Instead, it should merely constitute an atomic Manila data type on which nearly any Manila action is available. For example, given a share group, the user should be able to select the group in the Manila UI and invoke features such as snapshot, clone, backup, migrate, replicate, retype, etc. 

In the general case, group actions are handled by the share manager layer (or potentially the common share driver superclass). For example, invoking ‘snapshot’ on a group causes the share manager to take a snapshot of each group member individually. The resulting group snapshot object may be used to create a new group, not unlike how CG snapshots were implemented. 

There are numerous advantages to this approach. Every driver, no matter how unsophisticated, can leverage the manageability goodness of groups. The user experience is uniform, since groups are always available regardless of which backends are present and because nearly every Manila action available on shares is also available on groups. 

Of course, feature-rich backends would like to add their secret sauce, whether it be CGs, group-based replication, or whatever else comes along that might be easier/cheaper/faster to do to groups of shares. That brings us to a related idea.

=== Idea #2: Introduce Share Group types ===
Just as Manila has ‘share types’, it should also have ‘share group types’. Any driver that can perform a group operation in an advantaged way may report that as a group capability, such as:
* Ordered writes
* Consistent snapshots
* Group replication
* Group backup
* Group modification
 
As with share types, the cloud administrator predefines share group types that may contain extra specs corresponding to the group capabilities reported by the backends. And when creating a group, the user specifies one or more share types as well as a group type, and the scheduler then creates the group on one of the backends that match the specified share and share group types. 

Anytime a group action, such as ‘snapshot’, comes into the share manager, the manager checks whether its driver offers an advantaged implementation of that operation. If not, the manager handles the workflow itself as described above. But if so, the manager routes the workflow to its driver for fulfillment. 

The advantages of this approach should be obvious. Development and testing are simplified because there isn’t a need to define and test a different set of group management APIs for each feature, or to test every combination of every feature. Instead of becoming an N-by-N matrix of interacting features, Manila largely becomes an N-by-2 matrix of actions that may be invoked on either individual shares or share groups. Users and admins are already familiar with share types, so introducing share group types would seem a natural and consistent evolution of the same foundational concept.

The matrix would look like:
{| class="wikitable"
|-
! Share action !! Share Group action
|-
| Create (volume type) || Create (volume types, group type)
|-
| Delete || Delete (group, optionally all members)
|-
| Snapshot || Snapshot (may or may not be a CG snapshot)
|-
| Create from snapshot / Clone || Create from group snapshot / Clone group
|-
| Replicate || Replicate
|-
| Backup || Backup
|-
| Retype || Retype
|-
| Migrate || Migrate
|-
| Extend / shrink || N/A
|}

== Manila implementation plan ==
Implementing generic groups in Manila should be a straightforward series of steps:
# Implement generic groups. Because we did CGs, we already know all parts of the codebase that must change to support any kind of group. So the simplest approach is to modify the CG code to morph it into the generic groups feature.
# Add the share group type feature by duplicating and customizing the share type code.
# Enhance the scheduler to place groups according to share group types. Like #1, this is already informed by the CG project.
# Implement group snapshots in the share manager to demonstrate group snapshots in any driver.
# Plumb group snapshots to a CG-capable driver to demonstrate CG functionality in the new framework.
# Update Tempest to cover all of the above.
# Update Manila client to change CGs to share groups.
# Enhance Manila client with share group types.
# Add share group support to Horizon. We never built CG support into Horizon, so this is all-new work that now has much broader appeal and applicability.
# Add additional group actions over time (migrate, replicate, retype, clone, …). At this point, new group capabilities become vertical slices that are simple to add incrementally.
 
Because we implemented CGs just recently as an experimental feature, we have the freedom to replace that code without deprecation or upgrade considerations. Steps 1-8 would get Manila to parity with the CG feature added in Liberty, would require only a few person-weeks of effort, and would better position Manila for long-term evolution and supportability.
=== Implementation details ===
There are few things to note, several of which were already solved during the CG work. 

Groups are first-class objects in Manila, and operations on groups are treated as atomic. For example, a user will see a group snapshot but not the individual snapshots of group members. To enable support by as many backends as possible, Manila will still maintain DB objects for both group and member snapshots, just as was done with CGs. 

A few actions, such as extend & shrink, are inherently applicable only to individual shares. One could theoretically apply extend to a group, increasing the size of each member, but that seems like an unlikely use case. Any actions in this category must remain available to group members, although most actions such as clone or snapshot would be available only at the group level and not on its members. 

A group is limited to a single backend. Allowing groups that span backends is theoretically possible, but that would require fanout of operations from the API layer to multiple share managers across the asynchronous event bus, which would lead to complicated synchronization and state management for little operational benefit. 

As was done with Manila CGs, a driver may optionally limit a group to either the confines of a pool or an entire backend. It is known that pools are the unit of data motion (i.e. replication or migration) for some backends, so we think drivers need this flexibility. 

We considered reusing share types for groups as well. But share types include a set of public extra specs that may not map well to groups. And by adding group types as a separate object, there can be little confusion about their purpose and use. 

Some have noted that extra specs tend to be treated as an AND operation, where all features must be available for a backend to be chosen. It is conceivable that even if a backend can replicate a group or take consistent snapshots of a group, it might not be able to perform both operations on the same group. But this problem already exists with share types and hasn’t been a serious issue. Creating types is an admin-only operation and the burden remains on the admin to understand the capabilities of the backends in use and to create the share types and share group types appropriately. 

Note that this proposal explicitly does not address pool or backend replication, which is fundamentally different. Actions on shares or share groups are intended for tenants, whereas a pool or backend can contain data from multiple tenants. So pool or backend operations, while serving potentially valuable use cases, are inherently admin-only workflows that would be designed and exercised differently should Manila support them.

=== Share Group APIs ===

It is possible to design a REST API that seamlessly handles both shares and share groups with little duplication of APIs. But at this point in Manila's development, it is arguably too late to radically redesign the API.

It may be possible to overload some of the existing APIs to handle both shares and share groups. For example, POST /shares/{id}/action could accept the ID of a share or share group and just do the right thing. But other APIs, such as GET /shares are less practical to overload, since a single endpoint would be returning objects of different types.

It seems better to merely duplicate a few APIs with group versions as needed. For example:
* POST /shares --> POST /groups
* POST /shares/{share_id}/action --> POST /groups/{group_id}/action
* POST /snapshots --> POST /group_snapshots

== Acknowledgments ==
Most of the analysis and recommendations contained herein apply equally well to Cinder, which is arguably much further down the path of crippling complexity. 

This proposal was developed by cknight with input from Manila community members bswartz, ameade, tbarron, and cfouts. The Cinder conversations about grouping, as well as the groundbreaking contributions of CGs to Cinder, all of which inspired this proposal, were led by xyang.

Manila/design/manila-generic-groups

2016-04-22T19:38:10Z

Clinton Knight: /* Idea #1: Introduce a generic Share Group object */

= Manila generic groups =
== Executive summary ==
Manila needs a grouping construct that, like shares, is a 1st-class atomic data type. Our experience with CGs has demonstrated the complexity of adding a grouping capability, yet there are other use cases such as migration, replication, and backup in which some storage controllers could only offer such features on share groups. CGs also highlighted the poor optics of an advanced feature with comparatively little potential for vendor support. And adding new grouping constructs for each new feature is not technically feasible. All of the above may be addressed by generic groups, which we think is a clean extension to the original architecture of Manila.
== An Intractable Matrix ==
Manila is at an exciting time in its development, where all the simple features are available and the community is focused on adding high-value features such as migration, replication, and consistency groups. Experimental code is available for each of these, but little consideration has been given to the obvious need for these complex features to interact not only with each other but also with whatever additional features were added later. And looking a little deeper, it is apparent that each of these has limitations that could be solved by a single architectural enhancement.
=== Consistency groups ===
Consistency groups (CGs) are the only construct in Manila that operates on a group of shares. For some, a CG implies a guarantee of ordered writes by a storage controller, while for others a CG is a mechanism for taking consistent point-in-time snapshots of a set of shares. A group with either attribute may have value by itself, but the CGs implementation doesn’t distinguish between them. 

CGs are a highly specialized construct with dedicated CLI commands, REST APIs, database tables, scheduler filters, and driver APIs that totaled over 9900 lines. None of these are reusable for anything else, and none of the rest of Manila has any awareness of CGs. Even worse, despite all the complexity of the CG feature, only a small minority of storage backends can support them, so the code-to-value ratio is very low and the limited availability of CGs ensures the user experience is inconsistent between clouds.

=== Replication ===
Replication is another high value feature, and the Manila implementation is arguably clean and flexible, but as constituted there is no core ability to replicate groups of shares. It seems reasonable to implement the feature iteratively, beginning with replication of single shares. However, there has been the tacit acknowledgement that some backends would not be able to replicate individual shares, even though those same backends could potentially replicate a group of shares if the group were constituted in a certain way. 

Following the precedent established with CGs, one approach would be to introduce ‘replication groups’ whereby multiple shares could be replicated together. But this would introduce yet another set of APIs, tables, etc. that are too specialized to use for anything else. 

Another approach might be to define CGs as the grouping unit of replication, but this makes little sense because consistency groups (however we define them) and replication are distinct features that shouldn’t be artificially bundled for the sake of expediency of definition or implementation.
=== Migration ===
Just as some backends may only be able to replicate shares in groups, it follows that those backends may also need to migrate shares in groups. And in the case of CGs, it doesn’t make sense to migrate CG members individually; the whole group must be moved, requiring the replication engine to be CG aware. 

Consider the following reasonable use cases:
* Replicate a consistency group
* Snapshot a replication group
* Migrate a replication group
* Retype shares in a consistency group
* Retype shares in a replication group
* Backup a consistency group
* Backup a replication group
* Apply a common policy to a set of shares
 
On the current path, the support matrix has a dimension dedicated solely to features, so each feature must be coded to be interoperable with every other feature. This quickly becomes a support and testing nightmare, and it becomes exponentially more complicated to add more features going forward. 

Fundamentally, the problem is that we are adding features without an underlying architectural framework on which to hang them. To escape the matrix, we must step back and rethink a few things.

== A simple solution ==
So to arrive at a solution, let’s enumerate what we have:
* Primitive objects (shares) with supported operations controlled by share types that vary by backend
* A very specific group object (CGs) with limited support potential by backends
* A number of APIs (CGs, etc.) that many users can’t use at all and appear grafted into the project
 
And what would we prefer:
* A uniform API and user experience that varies as little as possible by backend
* A clean way to group shares so we can do CGs, multi-object replication, migration, etc.
 
How do we get here?
* Universally available groups of primitive objects with supported operations that vary by backend
 
=== Idea #1: Introduce a generic Share Group object ===
Just as Manila has ‘shares’, it should also have ‘share groups’. In its simplest form, unlike CGs, a share group should not guarantee any specialized operation. Instead, it should merely constitute an atomic Manila data type on which nearly any Manila action is available. For example, given a share group, the user should be able to select the group in the Manila UI and invoke features such as snapshot, clone, backup, migrate, replicate, retype, etc. 

In the general case, group actions are handled by the share manager layer (or potentially the common share driver superclass). For example, invoking ‘snapshot’ on a group causes the share manager to take a snapshot of each group member individually. The resulting group snapshot object may be used to create a new group, not unlike how CG snapshots were implemented. 

There are numerous advantages to this approach. Every driver, no matter how unsophisticated, can leverage the manageability goodness of groups. The user experience is uniform, since groups are always available regardless of which backends are present and because nearly every Manila action available on shares is also available on groups. 

Of course, feature-rich backends would like to add their secret sauce, whether it be CGs, group-based replication, or whatever else comes along that might be easier/cheaper/faster to do to groups of shares. That brings us to a related idea.

=== Idea #2: Introduce Share Group types ===
Just as Manila has ‘share types’, it should also have ‘share group types’. Any driver that can perform a group operation in an advantaged way may report that as a group capability, such as:
* Ordered writes
* Consistent snapshots
* Group replication
* Group backup
* Group modification
 
As with share types, the cloud administrator predefines share group types that may contain extra specs corresponding to the group capabilities reported by the backends. And when creating a group, the user specifies one or more share types as well as a group type, and the scheduler then creates the group on one of the backends that match the specified share and share group types. 

Anytime a group action, such as ‘snapshot’, comes into the share manager, the manager checks whether its driver offers an advantaged implementation of that operation. If not, the manager handles the workflow itself as described above. But if so, the manager routes the workflow to its driver for fulfillment. 

The advantages of this approach should be obvious. Development and testing are simplified because there isn’t a need to define and test a different set of group management APIs for each feature, or to test every combination of every feature. Instead of becoming an N-by-N matrix of interacting features, Manila largely becomes an N-by-2 matrix of actions that may be invoked on either individual shares or share groups. Users and admins are already familiar with share types, so introducing share group types would seem a natural and consistent evolution of the same foundational concept.

The matrix would look like:
{| class="wikitable"
|-
! Share action !! Share Group action
|-
| Create (volume type) || Create (volume types, group type)
|-
| Delete || Delete (group, optionally all members)
|-
| Snapshot || Snapshot (may or may not be a CG snapshot)
|-
| Create from snapshot / Clone || Create from group snapshot / Clone group
|-
| Replicate || Replicate
|-
| Backup || Backup
|-
| Retype || Retype
|-
| Migrate || Migrate
|-
| Extend / shrink || N/A
|}

== Manila implementation plan ==
Implementing generic groups in Manila should be a straightforward series of steps:
# Implement generic groups. Because we did CGs, we already know all parts of the codebase that must change to support any kind of group. So the simplest approach is to modify the CG code to morph it into the generic groups feature.
# Add the share group type feature by duplicating and customizing the share type code.
# Enhance the scheduler to place groups according to share group types. Like #1, this is already informed by the CG project.
# Implement group snapshots in the share manager to demonstrate group snapshots in any driver.
# Plumb group snapshots to a CG-capable driver to demonstrate CG functionality in the new framework.
# Update Tempest to cover all of the above.
# Update Manila client to change CGs to share groups.
# Enhance Manila client with share group types.
# Add share group support to Horizon. We never built CG support into Horizon, so this is all-new work that now has much broader appeal and applicability.
# Add additional group actions over time (migrate, replicate, retype, clone, …). At this point, new group capabilities become vertical slices that are simple to add incrementally.
 
Because we implemented CGs just recently as an experimental feature, we have the freedom to replace that code without deprecation or upgrade considerations. Steps 1-8 would get Manila to parity with the CG feature added in Liberty, would require only a few person-weeks of effort, and would better position Manila for long-term evolution and supportability.
=== Implementation details ===
There are few things to note, several of which were already solved during the CG work. 

Groups are first-class objects in Manila, and operations on groups are treated as atomic. For example, a user will see a group snapshot but not the individual snapshots of group members. To enable support by as many backends as possible, Manila will still maintain DB objects for both group and member snapshots, just as was done with CGs. 

A few actions, such as extend & shrink, are inherently applicable only to individual shares. One could theoretically apply extend to a group, increasing the size of each member, but that seems like an unlikely use case. Any actions in this category must remain available to group members, although most actions such as clone or snapshot would be available only at the group level and not on its members. 

A group is limited to a single backend. Allowing groups that span backends is theoretically possible, but that would require fanout of operations from the API layer to multiple share managers across the asynchronous event bus, which would lead to complicated synchronization and state management for little operational benefit. 

As was done with Manila CGs, a driver may optionally limit a group to either the confines of a pool or an entire backend. It is known that pools are the unit of data motion (i.e. replication or migration) for some backends, so we think drivers need this flexibility. 

We considered reusing share types for groups as well. But share types include a set of public extra specs that may not map well to groups. And by adding group types as a separate object, there can be little confusion about their purpose and use. 

Some have noted that extra specs tend to be treated as an AND operation, where all features must be available for a backend to be chosen. It is conceivable that even if a backend can replicate a group or take consistent snapshots of a group, it might not be able to perform both operations on the same group. But this problem already exists with share types and hasn’t been a serious issue. Creating types is an admin-only operation and the burden remains on the admin to understand the capabilities of the backends in use and to create the share types and share group types appropriately. 

Note that this proposal explicitly does not address pool or backend replication, which is fundamentally different. Actions on shares or share groups are intended for tenants, whereas a pool or backend can contain data from multiple tenants. So pool or backend operations, while serving potentially valuable use cases, are inherently admin-only workflows that would be designed and exercised differently should Manila support them.

=== Share Group APIs ===

It is possible to design a REST API that seamlessly handles both shares and share groups with little duplication of APIs. But at this point in Manila's development, it is arguably too late to radically redesign the API.

It may be possible to overload some of the existing APIs to handle both shares and share groups. For example, POST /shares/{id}/action could accept the ID of a share or share group and just do the right thing. But other APIs, such as GET /shares are less practical to overload, since a single endpoint would be returning objects of different types.

It seems better to merely duplicate a few APIs with group versions as needed. For example:
* POST /shares --> POST /groups
* POST /shares/{share_id}/action --> POST /groups/{group_id}/action
* POST /snapshots --> POST /group_snapshots

== Acknowledgments ==
Most of the analysis and recommendations contained herein apply equally well to Cinder, which is arguably much further down the path of crippling complexity. 

This proposal was developed by cknight with input from Manila community members bswartz, ameade, tbarron, and cfouts. The Cinder conversations about grouping, as well as the groundbreaking contributions of CGs to Cinder, all of which inspired this proposal, were led by xyang.

Manila/design/manila-generic-groups

2016-04-22T19:12:50Z

Clinton Knight: /* Consistency groups */

= Manila generic groups =
== Executive summary ==
Manila needs a grouping construct that, like shares, is a 1st-class atomic data type. Our experience with CGs has demonstrated the complexity of adding a grouping capability, yet there are other use cases such as migration, replication, and backup in which some storage controllers could only offer such features on share groups. CGs also highlighted the poor optics of an advanced feature with comparatively little potential for vendor support. And adding new grouping constructs for each new feature is not technically feasible. All of the above may be addressed by generic groups, which we think is a clean extension to the original architecture of Manila.
== An Intractable Matrix ==
Manila is at an exciting time in its development, where all the simple features are available and the community is focused on adding high-value features such as migration, replication, and consistency groups. Experimental code is available for each of these, but little consideration has been given to the obvious need for these complex features to interact not only with each other but also with whatever additional features were added later. And looking a little deeper, it is apparent that each of these has limitations that could be solved by a single architectural enhancement.
=== Consistency groups ===
Consistency groups (CGs) are the only construct in Manila that operates on a group of shares. For some, a CG implies a guarantee of ordered writes by a storage controller, while for others a CG is a mechanism for taking consistent point-in-time snapshots of a set of shares. A group with either attribute may have value by itself, but the CGs implementation doesn’t distinguish between them. 

CGs are a highly specialized construct with dedicated CLI commands, REST APIs, database tables, scheduler filters, and driver APIs that totaled over 9900 lines. None of these are reusable for anything else, and none of the rest of Manila has any awareness of CGs. Even worse, despite all the complexity of the CG feature, only a small minority of storage backends can support them, so the code-to-value ratio is very low and the limited availability of CGs ensures the user experience is inconsistent between clouds.

=== Replication ===
Replication is another high value feature, and the Manila implementation is arguably clean and flexible, but as constituted there is no core ability to replicate groups of shares. It seems reasonable to implement the feature iteratively, beginning with replication of single shares. However, there has been the tacit acknowledgement that some backends would not be able to replicate individual shares, even though those same backends could potentially replicate a group of shares if the group were constituted in a certain way. 

Following the precedent established with CGs, one approach would be to introduce ‘replication groups’ whereby multiple shares could be replicated together. But this would introduce yet another set of APIs, tables, etc. that are too specialized to use for anything else. 

Another approach might be to define CGs as the grouping unit of replication, but this makes little sense because consistency groups (however we define them) and replication are distinct features that shouldn’t be artificially bundled for the sake of expediency of definition or implementation.
=== Migration ===
Just as some backends may only be able to replicate shares in groups, it follows that those backends may also need to migrate shares in groups. And in the case of CGs, it doesn’t make sense to migrate CG members individually; the whole group must be moved, requiring the replication engine to be CG aware. 

Consider the following reasonable use cases:
* Replicate a consistency group
* Snapshot a replication group
* Migrate a replication group
* Retype shares in a consistency group
* Retype shares in a replication group
* Backup a consistency group
* Backup a replication group
* Apply a common policy to a set of shares
 
On the current path, the support matrix has a dimension dedicated solely to features, so each feature must be coded to be interoperable with every other feature. This quickly becomes a support and testing nightmare, and it becomes exponentially more complicated to add more features going forward. 

Fundamentally, the problem is that we are adding features without an underlying architectural framework on which to hang them. To escape the matrix, we must step back and rethink a few things.

== A simple solution ==
So to arrive at a solution, let’s enumerate what we have:
* Primitive objects (shares) with supported operations controlled by share types that vary by backend
* A very specific group object (CGs) with limited support potential by backends
* A number of APIs (CGs, etc.) that many users can’t use at all and appear grafted into the project
 
And what would we prefer:
* A uniform API and user experience that varies as little as possible by backend
* A clean way to group shares so we can do CGs, multi-object replication, migration, etc.
 
How do we get here?
* Universally available groups of primitive objects with supported operations that vary by backend
 
=== Idea #1: Introduce a generic Share Group object ===
Just as Manila has ‘shares’, it should also have ‘share groups’. In its simplest form, unlike CGs, a share group should not guarantee any specialized operation. Instead, it should merely constitute an atomic Manila data type on which nearly any Manila action is available. For example, given a share group, the user should be able to select the group in the Manila UI and invoke features such as snapshot, clone, backup, migrate, replicate, retype, etc. 

In the general case, group actions are handled by the share manager layer (or potentially the common share driver superclass). For example, invoking ‘snapshot’ on a group causes the share manager to take a snapshot of each group member individually. The resulting group snapshot object may be used to create a new group, not unlike how CG snapshots were implemented. 

There are numerous advantages to this approach. Every driver, no matter how unsophisticated, can leverage the manageability goodness of groups. The user experience is uniform, since groups are always available regardless of which backends are present and because nearly every Manila action available on shares is also available on groups. 

Of course, feature-rich backends would like to add their secret sauce, whether it be CGs, group-based replication, or whatever else comes along that might be easier/cheaper/faster to do to groups of shares. That brings us to related idea.
=== Idea #2: Introduce Share Group types ===
Just as Manila has ‘share types’, it should also have ‘share group types’. Any driver that can perform a group operation in an advantaged way may report that as a group capability, such as:
* Ordered writes
* Consistent snapshots
* Group replication
* Group backup
* Group modification
 
As with share types, the cloud administrator predefines share group types that may contain extra specs corresponding to the group capabilities reported by the backends. And when creating a group, the user specifies one or more share types as well as a group type, and the scheduler then creates the group on one of the backends that match the specified share and share group types. 

Anytime a group action, such as ‘snapshot’, comes into the share manager, the manager checks whether its driver offers an advantaged implementation of that operation. If not, the manager handles the workflow itself as described above. But if so, the manager routes the workflow to its driver for fulfillment. 

The advantages of this approach should be obvious. Development and testing are simplified because there isn’t a need to define and test a different set of group management APIs for each feature, or to test every combination of every feature. Instead of becoming an N-by-N matrix of interacting features, Manila largely becomes an N-by-2 matrix of actions that may be invoked on either individual shares or share groups. Users and admins are already familiar with share types, so introducing share group types would seem a natural and consistent evolution of the same foundational concept.

The matrix would look like:
{| class="wikitable"
|-
! Share action !! Share Group action
|-
| Create (volume type) || Create (volume types, group type)
|-
| Delete || Delete (group, optionally all members)
|-
| Snapshot || Snapshot (may or may not be a CG snapshot)
|-
| Create from snapshot / Clone || Create from group snapshot / Clone group
|-
| Replicate || Replicate
|-
| Backup || Backup
|-
| Retype || Retype
|-
| Migrate || Migrate
|-
| Extend / shrink || N/A
|}

== Manila implementation plan ==
Implementing generic groups in Manila should be a straightforward series of steps:
# Implement generic groups. Because we did CGs, we already know all parts of the codebase that must change to support any kind of group. So the simplest approach is to modify the CG code to morph it into the generic groups feature.
# Add the share group type feature by duplicating and customizing the share type code.
# Enhance the scheduler to place groups according to share group types. Like #1, this is already informed by the CG project.
# Implement group snapshots in the share manager to demonstrate group snapshots in any driver.
# Plumb group snapshots to a CG-capable driver to demonstrate CG functionality in the new framework.
# Update Tempest to cover all of the above.
# Update Manila client to change CGs to share groups.
# Enhance Manila client with share group types.
# Add share group support to Horizon. We never built CG support into Horizon, so this is all-new work that now has much broader appeal and applicability.
# Add additional group actions over time (migrate, replicate, retype, clone, …). At this point, new group capabilities become vertical slices that are simple to add incrementally.
 
Because we implemented CGs just recently as an experimental feature, we have the freedom to replace that code without deprecation or upgrade considerations. Steps 1-8 would get Manila to parity with the CG feature added in Liberty, would require only a few person-weeks of effort, and would better position Manila for long-term evolution and supportability.
=== Implementation details ===
There are few things to note, several of which were already solved during the CG work. 

Groups are first-class objects in Manila, and operations on groups are treated as atomic. For example, a user will see a group snapshot but not the individual snapshots of group members. To enable support by as many backends as possible, Manila will still maintain DB objects for both group and member snapshots, just as was done with CGs. 

A few actions, such as extend & shrink, are inherently applicable only to individual shares. One could theoretically apply extend to a group, increasing the size of each member, but that seems like an unlikely use case. Any actions in this category must remain available to group members, although most actions such as clone or snapshot would be available only at the group level and not on its members. 

A group is limited to a single backend. Allowing groups that span backends is theoretically possible, but that would require fanout of operations from the API layer to multiple share managers across the asynchronous event bus, which would lead to complicated synchronization and state management for little operational benefit. 

As was done with Manila CGs, a driver may optionally limit a group to either the confines of a pool or an entire backend. It is known that pools are the unit of data motion (i.e. replication or migration) for some backends, so we think drivers need this flexibility. 

We considered reusing share types for groups as well. But share types include a set of public extra specs that may not map well to groups. And by adding group types as a separate object, there can be little confusion about their purpose and use. 

Some have noted that extra specs tend to be treated as an AND operation, where all features must be available for a backend to be chosen. It is conceivable that even if a backend can replicate a group or take consistent snapshots of a group, it might not be able to perform both operations on the same group. But this problem already exists with share types and hasn’t been a serious issue. Creating types is an admin-only operation and the burden remains on the admin to understand the capabilities of the backends in use and to create the share types and share group types appropriately. 

Note that this proposal explicitly does not address pool or backend replication, which is fundamentally different. Actions on shares or share groups are intended for tenants, whereas a pool or backend can contain data from multiple tenants. So pool or backend operations, while serving potentially valuable use cases, are inherently admin-only workflows that would be designed and exercised differently should Manila support them.

=== Share Group APIs ===

It is possible to design a REST API that seamlessly handles both shares and share groups with little duplication of APIs. But at this point in Manila's development, it is arguably too late to radically redesign the API.

It may be possible to overload some of the existing APIs to handle both shares and share groups. For example, POST /shares/{id}/action could accept the ID of a share or share group and just do the right thing. But other APIs, such as GET /shares are less practical to overload, since a single endpoint would be returning objects of different types.

It seems better to merely duplicate a few APIs with group versions as needed. For example:
* POST /shares --> POST /groups
* POST /shares/{share_id}/action --> POST /groups/{group_id}/action
* POST /snapshots --> POST /group_snapshots

== Acknowledgments ==
Most of the analysis and recommendations contained herein apply equally well to Cinder, which is arguably much further down the path of crippling complexity. 

This proposal was developed by cknight with input from Manila community members bswartz, ameade, tbarron, and cfouts. The Cinder conversations about grouping, as well as the groundbreaking contributions of CGs to Cinder, all of which inspired this proposal, were led by xyang.

Manila/design/manila-generic-groups

2016-04-19T13:26:14Z

Clinton Knight: /* Idea #2: Introduce Share Group types */

= Manila generic groups =
== Executive summary ==
Manila needs a grouping construct that, like shares, is a 1st-class atomic data type. Our experience with CGs has demonstrated the complexity of adding a grouping capability, yet there are other use cases such as migration, replication, and backup in which some storage controllers could only offer such features on share groups. CGs also highlighted the poor optics of an advanced feature with comparatively little potential for vendor support. And adding new grouping constructs for each new feature is not technically feasible. All of the above may be addressed by generic groups, which we think is a clean extension to the original architecture of Manila.
== An Intractable Matrix ==
Manila is at an exciting time in its development, where all the simple features are available and the community is focused on adding high-value features such as migration, replication, and consistency groups. Experimental code is available for each of these, but little consideration has been given to the obvious need for these complex features to interact not only with each other but also with whatever additional features were added later. And looking a little deeper, it is apparent that each of these has limitations that could be solved by a single architectural enhancement.
=== Consistency groups ===
Consistency groups (CGs) are the only construct in Manila that operates on a group of shares. For some, a CG implies a guarantee of ordered writes by a storage controller, while for others a CG is a mechanism for taking consistent point-in-time snapshots of a set of shares. A group with either attribute may have value by itself, but the CGs implementation doesn’t distinguish between them. 

CGs are a highly specialized construct with dedicated CLI commands, REST APIs, database tables, scheduler filters, and driver APIs. None of these are reusable for anything else, and presently none of the rest of Manila has any awareness of CGs. Even worse, despite all the complexity of the CG feature, only a small minority of storage backends can support them, so the code-to-value ratio is very low. Finally, the limited availability of CGs ensures the user experience is inconsistent between clouds.
=== Replication ===
Replication is another high value feature, and the Manila implementation is arguably clean and flexible, but as constituted there is no core ability to replicate groups of shares. It seems reasonable to implement the feature iteratively, beginning with replication of single shares. However, there has been the tacit acknowledgement that some backends would not be able to replicate individual shares, even though those same backends could potentially replicate a group of shares if the group were constituted in a certain way. 

Following the precedent established with CGs, one approach would be to introduce ‘replication groups’ whereby multiple shares could be replicated together. But this would introduce yet another set of APIs, tables, etc. that are too specialized to use for anything else. 

Another approach might be to define CGs as the grouping unit of replication, but this makes little sense because consistency groups (however we define them) and replication are distinct features that shouldn’t be artificially bundled for the sake of expediency of definition or implementation.
=== Migration ===
Just as some backends may only be able to replicate shares in groups, it follows that those backends may also need to migrate shares in groups. And in the case of CGs, it doesn’t make sense to migrate CG members individually; the whole group must be moved, requiring the replication engine to be CG aware. 

Consider the following reasonable use cases:
* Replicate a consistency group
* Snapshot a replication group
* Migrate a replication group
* Retype shares in a consistency group
* Retype shares in a replication group
* Backup a consistency group
* Backup a replication group
* Apply a common policy to a set of shares
 
On the current path, the support matrix has a dimension dedicated solely to features, so each feature must be coded to be interoperable with every other feature. This quickly becomes a support and testing nightmare, and it becomes exponentially more complicated to add more features going forward. 

Fundamentally, the problem is that we are adding features without an underlying architectural framework on which to hang them. To escape the matrix, we must step back and rethink a few things.

== A simple solution ==
So to arrive at a solution, let’s enumerate what we have:
* Primitive objects (shares) with supported operations controlled by share types that vary by backend
* A very specific group object (CGs) with limited support potential by backends
* A number of APIs (CGs, etc.) that many users can’t use at all and appear grafted into the project
 
And what would we prefer:
* A uniform API and user experience that varies as little as possible by backend
* A clean way to group shares so we can do CGs, multi-object replication, migration, etc.
 
How do we get here?
* Universally available groups of primitive objects with supported operations that vary by backend
 
=== Idea #1: Introduce a generic Share Group object ===
Just as Manila has ‘shares’, it should also have ‘share groups’. In its simplest form, unlike CGs, a share group should not guarantee any specialized operation. Instead, it should merely constitute an atomic Manila data type on which nearly any Manila action is available. For example, given a share group, the user should be able to select the group in the Manila UI and invoke features such as snapshot, clone, backup, migrate, replicate, retype, etc. 

In the general case, group actions are handled by the share manager layer (or potentially the common share driver superclass). For example, invoking ‘snapshot’ on a group causes the share manager to take a snapshot of each group member individually. The resulting group snapshot object may be used to create a new group, not unlike how CG snapshots were implemented. 

There are numerous advantages to this approach. Every driver, no matter how unsophisticated, can leverage the manageability goodness of groups. The user experience is uniform, since groups are always available regardless of which backends are present and because nearly every Manila action available on shares is also available on groups. 

Of course, feature-rich backends would like to add their secret sauce, whether it be CGs, group-based replication, or whatever else comes along that might be easier/cheaper/faster to do to groups of shares. That brings us to related idea.
=== Idea #2: Introduce Share Group types ===
Just as Manila has ‘share types’, it should also have ‘share group types’. Any driver that can perform a group operation in an advantaged way may report that as a group capability, such as:
* Ordered writes
* Consistent snapshots
* Group replication
* Group backup
* Group modification
 
As with share types, the cloud administrator predefines share group types that may contain extra specs corresponding to the group capabilities reported by the backends. And when creating a group, the user specifies one or more share types as well as a group type, and the scheduler then creates the group on one of the backends that match the specified share and share group types. 

Anytime a group action, such as ‘snapshot’, comes into the share manager, the manager checks whether its driver offers an advantaged implementation of that operation. If not, the manager handles the workflow itself as described above. But if so, the manager routes the workflow to its driver for fulfillment. 

The advantages of this approach should be obvious. Development and testing are simplified because there isn’t a need to define and test a different set of group management APIs for each feature, or to test every combination of every feature. Instead of becoming an N-by-N matrix of interacting features, Manila largely becomes an N-by-2 matrix of actions that may be invoked on either individual shares or share groups. Users and admins are already familiar with share types, so introducing share group types would seem a natural and consistent evolution of the same foundational concept.

The matrix would look like:
{| class="wikitable"
|-
! Share action !! Share Group action
|-
| Create (volume type) || Create (volume types, group type)
|-
| Delete || Delete (group, optionally all members)
|-
| Snapshot || Snapshot (may or may not be a CG snapshot)
|-
| Create from snapshot / Clone || Create from group snapshot / Clone group
|-
| Replicate || Replicate
|-
| Backup || Backup
|-
| Retype || Retype
|-
| Migrate || Migrate
|-
| Extend / shrink || N/A
|}

== Manila implementation plan ==
Implementing generic groups in Manila should be a straightforward series of steps:
# Implement generic groups. Because we did CGs, we already know all parts of the codebase that must change to support any kind of group. So the simplest approach is to modify the CG code to morph it into the generic groups feature.
# Add the share group type feature by duplicating and customizing the share type code.
# Enhance the scheduler to place groups according to share group types. Like #1, this is already informed by the CG project.
# Implement group snapshots in the share manager to demonstrate group snapshots in any driver.
# Plumb group snapshots to a CG-capable driver to demonstrate CG functionality in the new framework.
# Update Tempest to cover all of the above.
# Update Manila client to change CGs to share groups.
# Enhance Manila client with share group types.
# Add share group support to Horizon. We never built CG support into Horizon, so this is all-new work that now has much broader appeal and applicability.
# Add additional group actions over time (migrate, replicate, retype, clone, …). At this point, new group capabilities become vertical slices that are simple to add incrementally.
 
Because we implemented CGs just recently as an experimental feature, we have the freedom to replace that code without deprecation or upgrade considerations. Steps 1-8 would get Manila to parity with the CG feature added in Liberty, would require only a few person-weeks of effort, and would better position Manila for long-term evolution and supportability.
=== Implementation details ===
There are few things to note, several of which were already solved during the CG work. 

Groups are first-class objects in Manila, and operations on groups are treated as atomic. For example, a user will see a group snapshot but not the individual snapshots of group members. To enable support by as many backends as possible, Manila will still maintain DB objects for both group and member snapshots, just as was done with CGs. 

A few actions, such as extend & shrink, are inherently applicable only to individual shares. One could theoretically apply extend to a group, increasing the size of each member, but that seems like an unlikely use case. Any actions in this category must remain available to group members, although most actions such as clone or snapshot would be available only at the group level and not on its members. 

A group is limited to a single backend. Allowing groups that span backends is theoretically possible, but that would require fanout of operations from the API layer to multiple share managers across the asynchronous event bus, which would lead to complicated synchronization and state management for little operational benefit. 

As was done with Manila CGs, a driver may optionally limit a group to either the confines of a pool or an entire backend. It is known that pools are the unit of data motion (i.e. replication or migration) for some backends, so we think drivers need this flexibility. 

We considered reusing share types for groups as well. But share types include a set of public extra specs that may not map well to groups. And by adding group types as a separate object, there can be little confusion about their purpose and use. 

Some have noted that extra specs tend to be treated as an AND operation, where all features must be available for a backend to be chosen. It is conceivable that even if a backend can replicate a group or take consistent snapshots of a group, it might not be able to perform both operations on the same group. But this problem already exists with share types and hasn’t been a serious issue. Creating types is an admin-only operation and the burden remains on the admin to understand the capabilities of the backends in use and to create the share types and share group types appropriately. 

Note that this proposal explicitly does not address pool or backend replication, which is fundamentally different. Actions on shares or share groups are intended for tenants, whereas a pool or backend can contain data from multiple tenants. So pool or backend operations, while serving potentially valuable use cases, are inherently admin-only workflows that would be designed and exercised differently should Manila support them.

=== Share Group APIs ===

It is possible to design a REST API that seamlessly handles both shares and share groups with little duplication of APIs. But at this point in Manila's development, it is arguably too late to radically redesign the API.

It may be possible to overload some of the existing APIs to handle both shares and share groups. For example, POST /shares/{id}/action could accept the ID of a share or share group and just do the right thing. But other APIs, such as GET /shares are less practical to overload, since a single endpoint would be returning objects of different types.

It seems better to merely duplicate a few APIs with group versions as needed. For example:
* POST /shares --> POST /groups
* POST /shares/{share_id}/action --> POST /groups/{group_id}/action
* POST /snapshots --> POST /group_snapshots

== Acknowledgments ==
Most of the analysis and recommendations contained herein apply equally well to Cinder, which is arguably much further down the path of crippling complexity. 

This proposal was developed by cknight with input from Manila community members bswartz, ameade, tbarron, and cfouts. The Cinder conversations about grouping, as well as the groundbreaking contributions of CGs to Cinder, all of which inspired this proposal, were led by xyang.

Manila/Meetings

2016-04-07T13:50:17Z

Clinton Knight: /* Next meeting */

Manila/Meetings

2016-03-31T14:53:20Z

Clinton Knight:

Manila/Meetings

2016-03-31T14:51:07Z

Clinton Knight:

Manila/Meetings

2016-02-25T15:06:05Z

Clinton Knight: /* Next meeting */

Manila/Meetings

2016-02-25T15:03:35Z

Clinton Knight: /* Next meeting */

Manila/Meetings

2016-02-25T15:01:53Z

Clinton Knight: /* Next meeting */

Manila/Meetings

2016-02-25T15:00:05Z

Clinton Knight: /* Next meeting */

Manila/Meetings

2016-02-25T14:58:18Z

Clinton Knight: /* Next meeting */

Manila/design/manila-generic-groups

2016-01-13T14:18:09Z

Clinton Knight: /* Share Group APIs */

= Manila generic groups =
== Executive summary ==
Manila needs a grouping construct that, like shares, is a 1st-class atomic data type. Our experience with CGs has demonstrated the complexity of adding a grouping capability, yet there are other use cases such as migration, replication, and backup in which some storage controllers could only offer such features on share groups. CGs also highlighted the poor optics of an advanced feature with comparatively little potential for vendor support. And adding new grouping constructs for each new feature is not technically feasible. All of the above may be addressed by generic groups, which we think is a clean extension to the original architecture of Manila.
== An Intractable Matrix ==
Manila is at an exciting time in its development, where all the simple features are available and the community is focused on adding high-value features such as migration, replication, and consistency groups. Experimental code is available for each of these, but little consideration has been given to the obvious need for these complex features to interact not only with each other but also with whatever additional features were added later. And looking a little deeper, it is apparent that each of these has limitations that could be solved by a single architectural enhancement.
=== Consistency groups ===
Consistency groups (CGs) are the only construct in Manila that operates on a group of shares. For some, a CG implies a guarantee of ordered writes by a storage controller, while for others a CG is a mechanism for taking consistent point-in-time snapshots of a set of shares. A group with either attribute may have value by itself, but the CGs implementation doesn’t distinguish between them. 

CGs are a highly specialized construct with dedicated CLI commands, REST APIs, database tables, scheduler filters, and driver APIs. None of these are reusable for anything else, and presently none of the rest of Manila has any awareness of CGs. Even worse, despite all the complexity of the CG feature, only a small minority of storage backends can support them, so the code-to-value ratio is very low. Finally, the limited availability of CGs ensures the user experience is inconsistent between clouds.
=== Replication ===
Replication is another high value feature, and the Manila implementation is arguably clean and flexible, but as constituted there is no core ability to replicate groups of shares. It seems reasonable to implement the feature iteratively, beginning with replication of single shares. However, there has been the tacit acknowledgement that some backends would not be able to replicate individual shares, even though those same backends could potentially replicate a group of shares if the group were constituted in a certain way. 

Following the precedent established with CGs, one approach would be to introduce ‘replication groups’ whereby multiple shares could be replicated together. But this would introduce yet another set of APIs, tables, etc. that are too specialized to use for anything else. 

Another approach might be to define CGs as the grouping unit of replication, but this makes little sense because consistency groups (however we define them) and replication are distinct features that shouldn’t be artificially bundled for the sake of expediency of definition or implementation.
=== Migration ===
Just as some backends may only be able to replicate shares in groups, it follows that those backends may also need to migrate shares in groups. And in the case of CGs, it doesn’t make sense to migrate CG members individually; the whole group must be moved, requiring the replication engine to be CG aware. 

Consider the following reasonable use cases:
* Replicate a consistency group
* Snapshot a replication group
* Migrate a replication group
* Retype shares in a consistency group
* Retype shares in a replication group
* Backup a consistency group
* Backup a replication group
* Apply a common policy to a set of shares
 
On the current path, the support matrix has a dimension dedicated solely to features, so each feature must be coded to be interoperable with every other feature. This quickly becomes a support and testing nightmare, and it becomes exponentially more complicated to add more features going forward. 

Fundamentally, the problem is that we are adding features without an underlying architectural framework on which to hang them. To escape the matrix, we must step back and rethink a few things.

== A simple solution ==
So to arrive at a solution, let’s enumerate what we have:
* Primitive objects (shares) with supported operations controlled by share types that vary by backend
* A very specific group object (CGs) with limited support potential by backends
* A number of APIs (CGs, etc.) that many users can’t use at all and appear grafted into the project
 
And what would we prefer:
* A uniform API and user experience that varies as little as possible by backend
* A clean way to group shares so we can do CGs, multi-object replication, migration, etc.
 
How do we get here?
* Universally available groups of primitive objects with supported operations that vary by backend
 
=== Idea #1: Introduce a generic Share Group object ===
Just as Manila has ‘shares’, it should also have ‘share groups’. In its simplest form, unlike CGs, a share group should not guarantee any specialized operation. Instead, it should merely constitute an atomic Manila data type on which nearly any Manila action is available. For example, given a share group, the user should be able to select the group in the Manila UI and invoke features such as snapshot, clone, backup, migrate, replicate, retype, etc. 

In the general case, group actions are handled by the share manager layer (or potentially the common share driver superclass). For example, invoking ‘snapshot’ on a group causes the share manager to take a snapshot of each group member individually. The resulting group snapshot object may be used to create a new group, not unlike how CG snapshots were implemented. 

There are numerous advantages to this approach. Every driver, no matter how unsophisticated, can leverage the manageability goodness of groups. The user experience is uniform, since groups are always available regardless of which backends are present and because nearly every Manila action available on shares is also available on groups. 

Of course, feature-rich backends would like to add their secret sauce, whether it be CGs, group-based replication, or whatever else comes along that might be easier/cheaper/faster to do to groups of shares. That brings us to related idea.
=== Idea #2: Introduce Share Group types ===
Just as Manila has ‘share types’, it should also have ‘share group types’. Any driver that can perform a group operation in an advantaged way may report that as a group capability, such as:
* Ordered writes
* Consistent snapshots
* Group replication
* Group migration
* Group backup
* Group modification
 
As with share types, the cloud administrator predefines share group types that may contain extra specs corresponding to the group capabilities reported by the backends. And when creating a group, the user specifies one or more share types as well as a group type, and the scheduler then creates the group on one of the backends that match the specified share and share group types. 

Anytime a group action, such as ‘snapshot’, comes into the share manager, the manager checks whether its driver offers an advantaged implementation of that operation. If not, the manager handles the workflow itself as described above. But if so, the manager routes the workflow to its driver for fulfillment. 

The advantages of this approach should be obvious. Development and testing are simplified because there isn’t a need to define and test a different set of group management APIs for each feature, or to test every combination of every feature. Instead of becoming an N-by-N matrix of interacting features, Manila largely becomes an N-by-2 matrix of actions that may be invoked on either individual shares or share groups. Users and admins are already familiar with share types, so introducing share group types would seem a natural and consistent evolution of the same foundational concept.

The matrix would look like:
{| class="wikitable"
|-
! Share action !! Share Group action
|-
| Create (volume type) || Create (volume types, group type)
|-
| Delete || Delete (group, optionally all members)
|-
| Snapshot || Snapshot (may or may not be a CG snapshot)
|-
| Create from snapshot / Clone || Create from group snapshot / Clone group
|-
| Replicate || Replicate
|-
| Backup || Backup
|-
| Retype || Retype
|-
| Migrate || Migrate
|-
| Extend / shrink || N/A
|}

== Manila implementation plan ==
Implementing generic groups in Manila should be a straightforward series of steps:
# Implement generic groups. Because we did CGs, we already know all parts of the codebase that must change to support any kind of group. So the simplest approach is to modify the CG code to morph it into the generic groups feature.
# Add the share group type feature by duplicating and customizing the share type code.
# Enhance the scheduler to place groups according to share group types. Like #1, this is already informed by the CG project.
# Implement group snapshots in the share manager to demonstrate group snapshots in any driver.
# Plumb group snapshots to a CG-capable driver to demonstrate CG functionality in the new framework.
# Update Tempest to cover all of the above.
# Update Manila client to change CGs to share groups.
# Enhance Manila client with share group types.
# Add share group support to Horizon. We never built CG support into Horizon, so this is all-new work that now has much broader appeal and applicability.
# Add additional group actions over time (migrate, replicate, retype, clone, …). At this point, new group capabilities become vertical slices that are simple to add incrementally.
 
Because we implemented CGs just recently as an experimental feature, we have the freedom to replace that code without deprecation or upgrade considerations. Steps 1-8 would get Manila to parity with the CG feature added in Liberty, would require only a few person-weeks of effort, and would better position Manila for long-term evolution and supportability.
=== Implementation details ===
There are few things to note, several of which were already solved during the CG work. 

Groups are first-class objects in Manila, and operations on groups are treated as atomic. For example, a user will see a group snapshot but not the individual snapshots of group members. To enable support by as many backends as possible, Manila will still maintain DB objects for both group and member snapshots, just as was done with CGs. 

A few actions, such as extend & shrink, are inherently applicable only to individual shares. One could theoretically apply extend to a group, increasing the size of each member, but that seems like an unlikely use case. Any actions in this category must remain available to group members, although most actions such as clone or snapshot would be available only at the group level and not on its members. 

A group is limited to a single backend. Allowing groups that span backends is theoretically possible, but that would require fanout of operations from the API layer to multiple share managers across the asynchronous event bus, which would lead to complicated synchronization and state management for little operational benefit. 

As was done with Manila CGs, a driver may optionally limit a group to either the confines of a pool or an entire backend. It is known that pools are the unit of data motion (i.e. replication or migration) for some backends, so we think drivers need this flexibility. 

We considered reusing share types for groups as well. But share types include a set of public extra specs that may not map well to groups. And by adding group types as a separate object, there can be little confusion about their purpose and use. 

Some have noted that extra specs tend to be treated as an AND operation, where all features must be available for a backend to be chosen. It is conceivable that even if a backend can replicate a group or take consistent snapshots of a group, it might not be able to perform both operations on the same group. But this problem already exists with share types and hasn’t been a serious issue. Creating types is an admin-only operation and the burden remains on the admin to understand the capabilities of the backends in use and to create the share types and share group types appropriately. 

Note that this proposal explicitly does not address pool or backend replication, which is fundamentally different. Actions on shares or share groups are intended for tenants, whereas a pool or backend can contain data from multiple tenants. So pool or backend operations, while serving potentially valuable use cases, are inherently admin-only workflows that would be designed and exercised differently should Manila support them.

=== Share Group APIs ===

It is possible to design a REST API that seamlessly handles both shares and share groups with little duplication of APIs. But at this point in Manila's development, it is arguably too late to radically redesign the API.

It may be possible to overload some of the existing APIs to handle both shares and share groups. For example, POST /shares/{id}/action could accept the ID of a share or share group and just do the right thing. But other APIs, such as GET /shares are less practical to overload, since a single endpoint would be returning objects of different types.

It seems better to merely duplicate a few APIs with group versions as needed. For example:
* POST /shares --> POST /groups
* POST /shares/{share_id}/action --> POST /groups/{group_id}/action
* POST /snapshots --> POST /group_snapshots

== Acknowledgments ==
Most of the analysis and recommendations contained herein apply equally well to Cinder, which is arguably much further down the path of crippling complexity. 

This proposal was developed by cknight with input from Manila community members bswartz, ameade, tbarron, and cfouts. The Cinder conversations about grouping, as well as the groundbreaking contributions of CGs to Cinder, all of which inspired this proposal, were led by xyang.

Manila/design/manila-generic-groups

2016-01-13T14:17:14Z

Clinton Knight:

= Manila generic groups =
== Executive summary ==
Manila needs a grouping construct that, like shares, is a 1st-class atomic data type. Our experience with CGs has demonstrated the complexity of adding a grouping capability, yet there are other use cases such as migration, replication, and backup in which some storage controllers could only offer such features on share groups. CGs also highlighted the poor optics of an advanced feature with comparatively little potential for vendor support. And adding new grouping constructs for each new feature is not technically feasible. All of the above may be addressed by generic groups, which we think is a clean extension to the original architecture of Manila.
== An Intractable Matrix ==
Manila is at an exciting time in its development, where all the simple features are available and the community is focused on adding high-value features such as migration, replication, and consistency groups. Experimental code is available for each of these, but little consideration has been given to the obvious need for these complex features to interact not only with each other but also with whatever additional features were added later. And looking a little deeper, it is apparent that each of these has limitations that could be solved by a single architectural enhancement.
=== Consistency groups ===
Consistency groups (CGs) are the only construct in Manila that operates on a group of shares. For some, a CG implies a guarantee of ordered writes by a storage controller, while for others a CG is a mechanism for taking consistent point-in-time snapshots of a set of shares. A group with either attribute may have value by itself, but the CGs implementation doesn’t distinguish between them. 

CGs are a highly specialized construct with dedicated CLI commands, REST APIs, database tables, scheduler filters, and driver APIs. None of these are reusable for anything else, and presently none of the rest of Manila has any awareness of CGs. Even worse, despite all the complexity of the CG feature, only a small minority of storage backends can support them, so the code-to-value ratio is very low. Finally, the limited availability of CGs ensures the user experience is inconsistent between clouds.
=== Replication ===
Replication is another high value feature, and the Manila implementation is arguably clean and flexible, but as constituted there is no core ability to replicate groups of shares. It seems reasonable to implement the feature iteratively, beginning with replication of single shares. However, there has been the tacit acknowledgement that some backends would not be able to replicate individual shares, even though those same backends could potentially replicate a group of shares if the group were constituted in a certain way. 

Following the precedent established with CGs, one approach would be to introduce ‘replication groups’ whereby multiple shares could be replicated together. But this would introduce yet another set of APIs, tables, etc. that are too specialized to use for anything else. 

Another approach might be to define CGs as the grouping unit of replication, but this makes little sense because consistency groups (however we define them) and replication are distinct features that shouldn’t be artificially bundled for the sake of expediency of definition or implementation.
=== Migration ===
Just as some backends may only be able to replicate shares in groups, it follows that those backends may also need to migrate shares in groups. And in the case of CGs, it doesn’t make sense to migrate CG members individually; the whole group must be moved, requiring the replication engine to be CG aware. 

Consider the following reasonable use cases:
* Replicate a consistency group
* Snapshot a replication group
* Migrate a replication group
* Retype shares in a consistency group
* Retype shares in a replication group
* Backup a consistency group
* Backup a replication group
* Apply a common policy to a set of shares
 
On the current path, the support matrix has a dimension dedicated solely to features, so each feature must be coded to be interoperable with every other feature. This quickly becomes a support and testing nightmare, and it becomes exponentially more complicated to add more features going forward. 

Fundamentally, the problem is that we are adding features without an underlying architectural framework on which to hang them. To escape the matrix, we must step back and rethink a few things.

== A simple solution ==
So to arrive at a solution, let’s enumerate what we have:
* Primitive objects (shares) with supported operations controlled by share types that vary by backend
* A very specific group object (CGs) with limited support potential by backends
* A number of APIs (CGs, etc.) that many users can’t use at all and appear grafted into the project
 
And what would we prefer:
* A uniform API and user experience that varies as little as possible by backend
* A clean way to group shares so we can do CGs, multi-object replication, migration, etc.
 
How do we get here?
* Universally available groups of primitive objects with supported operations that vary by backend
 
=== Idea #1: Introduce a generic Share Group object ===
Just as Manila has ‘shares’, it should also have ‘share groups’. In its simplest form, unlike CGs, a share group should not guarantee any specialized operation. Instead, it should merely constitute an atomic Manila data type on which nearly any Manila action is available. For example, given a share group, the user should be able to select the group in the Manila UI and invoke features such as snapshot, clone, backup, migrate, replicate, retype, etc. 

In the general case, group actions are handled by the share manager layer (or potentially the common share driver superclass). For example, invoking ‘snapshot’ on a group causes the share manager to take a snapshot of each group member individually. The resulting group snapshot object may be used to create a new group, not unlike how CG snapshots were implemented. 

There are numerous advantages to this approach. Every driver, no matter how unsophisticated, can leverage the manageability goodness of groups. The user experience is uniform, since groups are always available regardless of which backends are present and because nearly every Manila action available on shares is also available on groups. 

Of course, feature-rich backends would like to add their secret sauce, whether it be CGs, group-based replication, or whatever else comes along that might be easier/cheaper/faster to do to groups of shares. That brings us to related idea.
=== Idea #2: Introduce Share Group types ===
Just as Manila has ‘share types’, it should also have ‘share group types’. Any driver that can perform a group operation in an advantaged way may report that as a group capability, such as:
* Ordered writes
* Consistent snapshots
* Group replication
* Group migration
* Group backup
* Group modification
 
As with share types, the cloud administrator predefines share group types that may contain extra specs corresponding to the group capabilities reported by the backends. And when creating a group, the user specifies one or more share types as well as a group type, and the scheduler then creates the group on one of the backends that match the specified share and share group types. 

Anytime a group action, such as ‘snapshot’, comes into the share manager, the manager checks whether its driver offers an advantaged implementation of that operation. If not, the manager handles the workflow itself as described above. But if so, the manager routes the workflow to its driver for fulfillment. 

The advantages of this approach should be obvious. Development and testing are simplified because there isn’t a need to define and test a different set of group management APIs for each feature, or to test every combination of every feature. Instead of becoming an N-by-N matrix of interacting features, Manila largely becomes an N-by-2 matrix of actions that may be invoked on either individual shares or share groups. Users and admins are already familiar with share types, so introducing share group types would seem a natural and consistent evolution of the same foundational concept.

The matrix would look like:
{| class="wikitable"
|-
! Share action !! Share Group action
|-
| Create (volume type) || Create (volume types, group type)
|-
| Delete || Delete (group, optionally all members)
|-
| Snapshot || Snapshot (may or may not be a CG snapshot)
|-
| Create from snapshot / Clone || Create from group snapshot / Clone group
|-
| Replicate || Replicate
|-
| Backup || Backup
|-
| Retype || Retype
|-
| Migrate || Migrate
|-
| Extend / shrink || N/A
|}

== Manila implementation plan ==
Implementing generic groups in Manila should be a straightforward series of steps:
# Implement generic groups. Because we did CGs, we already know all parts of the codebase that must change to support any kind of group. So the simplest approach is to modify the CG code to morph it into the generic groups feature.
# Add the share group type feature by duplicating and customizing the share type code.
# Enhance the scheduler to place groups according to share group types. Like #1, this is already informed by the CG project.
# Implement group snapshots in the share manager to demonstrate group snapshots in any driver.
# Plumb group snapshots to a CG-capable driver to demonstrate CG functionality in the new framework.
# Update Tempest to cover all of the above.
# Update Manila client to change CGs to share groups.
# Enhance Manila client with share group types.
# Add share group support to Horizon. We never built CG support into Horizon, so this is all-new work that now has much broader appeal and applicability.
# Add additional group actions over time (migrate, replicate, retype, clone, …). At this point, new group capabilities become vertical slices that are simple to add incrementally.
 
Because we implemented CGs just recently as an experimental feature, we have the freedom to replace that code without deprecation or upgrade considerations. Steps 1-8 would get Manila to parity with the CG feature added in Liberty, would require only a few person-weeks of effort, and would better position Manila for long-term evolution and supportability.
=== Implementation details ===
There are few things to note, several of which were already solved during the CG work. 

Groups are first-class objects in Manila, and operations on groups are treated as atomic. For example, a user will see a group snapshot but not the individual snapshots of group members. To enable support by as many backends as possible, Manila will still maintain DB objects for both group and member snapshots, just as was done with CGs. 

A few actions, such as extend & shrink, are inherently applicable only to individual shares. One could theoretically apply extend to a group, increasing the size of each member, but that seems like an unlikely use case. Any actions in this category must remain available to group members, although most actions such as clone or snapshot would be available only at the group level and not on its members. 

A group is limited to a single backend. Allowing groups that span backends is theoretically possible, but that would require fanout of operations from the API layer to multiple share managers across the asynchronous event bus, which would lead to complicated synchronization and state management for little operational benefit. 

As was done with Manila CGs, a driver may optionally limit a group to either the confines of a pool or an entire backend. It is known that pools are the unit of data motion (i.e. replication or migration) for some backends, so we think drivers need this flexibility. 

We considered reusing share types for groups as well. But share types include a set of public extra specs that may not map well to groups. And by adding group types as a separate object, there can be little confusion about their purpose and use. 

Some have noted that extra specs tend to be treated as an AND operation, where all features must be available for a backend to be chosen. It is conceivable that even if a backend can replicate a group or take consistent snapshots of a group, it might not be able to perform both operations on the same group. But this problem already exists with share types and hasn’t been a serious issue. Creating types is an admin-only operation and the burden remains on the admin to understand the capabilities of the backends in use and to create the share types and share group types appropriately. 

Note that this proposal explicitly does not address pool or backend replication, which is fundamentally different. Actions on shares or share groups are intended for tenants, whereas a pool or backend can contain data from multiple tenants. So pool or backend operations, while serving potentially valuable use cases, are inherently admin-only workflows that would be designed and exercised differently should Manila support them.

=== Share Group APIs ===

It is possible to design a REST API that seamlessly handles both shares and share groups with little duplication of APIs. But at this point in Manila's development, it is arguably too late to radically redesign the API.

I may be possible to overload some of the existing APIs to handle both shares and share groups. For example, POST /shares/{id}/action could accept the ID of a share or share group and just do the right thing. But other APIs, such as GET /shares are less practical to overload, since a single endpoint would be returning objects of different types.

It seems better to merely duplicate a few APIs with group versions as needed. For example:
* POST /shares --> POST /groups
* POST /shares/{share_id}/action --> POST /groups/{group_id}/action
* POST /snapshots --> POST /group_snapshots

== Acknowledgments ==
Most of the analysis and recommendations contained herein apply equally well to Cinder, which is arguably much further down the path of crippling complexity. 

This proposal was developed by cknight with input from Manila community members bswartz, ameade, tbarron, and cfouts. The Cinder conversations about grouping, as well as the groundbreaking contributions of CGs to Cinder, all of which inspired this proposal, were led by xyang.

Manila/design/manila-generic-groups

2015-12-16T21:37:19Z

Clinton Knight: /* Acknowledgments */

= Manila generic groups =
== Executive summary ==
Manila needs a grouping construct that, like shares, is a 1st-class atomic data type. Our experience with CGs has demonstrated the complexity of adding a grouping capability, yet there are other use cases such as migration, replication, and backup in which some storage controllers could only offer such features on share groups. CGs also highlighted the poor optics of an advanced feature with comparatively little potential for vendor support. And adding new grouping constructs for each new feature is not technically feasible. All of the above may be addressed by generic groups, which we think is a clean extension to the original architecture of Manila.
== An Intractable Matrix ==
Manila is at an exciting time in its development, where all the simple features are available and the community is focused on adding high-value features such as migration, replication, and consistency groups. Experimental code is available for each of these, but little consideration has been given to the obvious need for these complex features to interact not only with each other but also with whatever additional features were added later. And looking a little deeper, it is apparent that each of these has limitations that could be solved by a single architectural enhancement.
=== Consistency groups ===
Consistency groups (CGs) are the only construct in Manila that operates on a group of shares. For some, a CG implies a guarantee of ordered writes by a storage controller, while for others a CG is a mechanism for taking consistent point-in-time snapshots of a set of shares. A group with either attribute may have value by itself, but the CGs implementation doesn’t distinguish between them. 

CGs are a highly specialized construct with dedicated CLI commands, REST APIs, database tables, scheduler filters, and driver APIs. None of these are reusable for anything else, and presently none of the rest of Manila has any awareness of CGs. Even worse, despite all the complexity of the CG feature, only a small minority of storage backends can support them, so the code-to-value ratio is very low. Finally, the limited availability of CGs ensures the user experience is inconsistent between clouds.
=== Replication ===
Replication is another high value feature, and the Manila implementation is arguably clean and flexible, but as constituted there is no core ability to replicate groups of shares. It seems reasonable to implement the feature iteratively, beginning with replication of single shares. However, there has been the tacit acknowledgement that some backends would not be able to replicate individual shares, even though those same backends could potentially replicate a group of shares if the group were constituted in a certain way. 

Following the precedent established with CGs, one approach would be to introduce ‘replication groups’ whereby multiple shares could be replicated together. But this would introduce yet another set of APIs, tables, etc. that are too specialized to use for anything else. 

Another approach might be to define CGs as the grouping unit of replication, but this makes little sense because consistency groups (however we define them) and replication are distinct features that shouldn’t be artificially bundled for the sake of expediency of definition or implementation.
=== Migration ===
Just as some backends may only be able to replicate shares in groups, it follows that those backends may also need to migrate shares in groups. And in the case of CGs, it doesn’t make sense to migrate CG members individually; the whole group must be moved, requiring the replication engine to be CG aware. 

Consider the following reasonable use cases:
* Replicate a consistency group
* Snapshot a replication group
* Migrate a replication group
* Retype shares in a consistency group
* Retype shares in a replication group
* Backup a consistency group
* Backup a replication group
* Apply a common policy to a set of shares
 
On the current path, the support matrix has a dimension dedicated solely to features, so each feature must be coded to be interoperable with every other feature. This quickly becomes a support and testing nightmare, and it becomes exponentially more complicated to add more features going forward. 

Fundamentally, the problem is that we are adding features without an underlying architectural framework on which to hang them. To escape the matrix, we must step back and rethink a few things.

== A simple solution ==
So to arrive at a solution, let’s enumerate what we have:
* Primitive objects (shares) with supported operations controlled by share types that vary by backend
* A very specific group object (CGs) with limited support potential by backends
* A number of APIs (CGs, etc.) that many users can’t use at all and appear grafted into the project
 
And what would we prefer:
* A uniform API and user experience that varies as little as possible by backend
* A clean way to group shares so we can do CGs, multi-object replication, migration, etc.
 
How do we get here?
* Universally available groups of primitive objects with supported operations that vary by backend
 
=== Idea #1: Introduce a generic Share Group object ===
Just as Manila has ‘shares’, it should also have ‘share groups’. In its simplest form, unlike CGs, a share group should not guarantee any specialized operation. Instead, it should merely constitute an atomic Manila data type on which nearly any Manila action is available. For example, given a share group, the user should be able to select the group in the Manila UI and invoke features such as snapshot, clone, backup, migrate, replicate, retype, etc. 

In the general case, group actions are handled by the share manager layer (or potentially the common share driver superclass). For example, invoking ‘snapshot’ on a group causes the share manager to take a snapshot of each group member individually. The resulting group snapshot object may be used to create a new group, not unlike how CG snapshots were implemented. 

There are numerous advantages to this approach. Every driver, no matter how unsophisticated, can leverage the manageability goodness of groups. The user experience is uniform, since groups are always available regardless of which backends are present and because nearly every Manila action available on shares is also available on groups. 

Of course, feature-rich backends would like to add their secret sauce, whether it be CGs, group-based replication, or whatever else comes along that might be easier/cheaper/faster to do to groups of shares. That brings us to related idea.
=== Idea #2: Introduce Share Group types ===
Just as Manila has ‘share types’, it should also have ‘share group types’. Any driver that can perform a group operation in an advantaged way may report that as a group capability, such as:
* Ordered writes
* Consistent snapshots
* Group replication
* Group migration
* Group backup
* Group modification
 
As with share types, the cloud administrator predefines share group types that may contain extra specs corresponding to the group capabilities reported by the backends. And when creating a group, the user specifies one or more share types as well as a group type, and the scheduler then creates the group on one of the backends that match the specified share and share group types. 

Anytime a group action, such as ‘snapshot’, comes into the share manager, the manager checks whether its driver offers an advantaged implementation of that operation. If not, the manager handles the workflow itself as described above. But if so, the manager routes the workflow to its driver for fulfillment. 

The advantages of this approach should be obvious. Development and testing are simplified because there isn’t a need to define and test a different set of group management APIs for each feature, or to test every combination of every feature. Instead of becoming an N-by-N matrix of interacting features, Manila largely becomes an N-by-2 matrix of actions that may be invoked on either individual shares or share groups. Users and admins are already familiar with share types, so introducing share group types would seem a natural and consistent evolution of the same foundational concept.

The matrix would look like:
{| class="wikitable"
|-
! Share action !! Share Group action
|-
| Create (volume type) || Create (volume types, group type)
|-
| Delete || Delete (group, optionally all members)
|-
| Snapshot || Snapshot (may or may not be a CG snapshot)
|-
| Create from snapshot / Clone || Create from group snapshot / Clone group
|-
| Replicate || Replicate
|-
| Backup || Backup
|-
| Retype || Retype
|-
| Migrate || Migrate
|-
| Extend / shrink || N/A
|}

== Manila implementation plan ==
Implementing generic groups in Manila should be a straightforward series of steps:
# Implement generic groups. Because we did CGs, we already know all parts of the codebase that must change to support any kind of group. So the simplest approach is to modify the CG code to morph it into the generic groups feature.
# Add the share group type feature by duplicating and customizing the share type code.
# Enhance the scheduler to place groups according to share group types. Like #1, this is already informed by the CG project.
# Implement group snapshots in the share manager to demonstrate group snapshots in any driver.
# Plumb group snapshots to a CG-capable driver to demonstrate CG functionality in the new framework.
# Update Tempest to cover all of the above.
# Update Manila client to change CGs to share groups.
# Enhance Manila client with share group types.
# Add share group support to Horizon. We never built CG support into Horizon, so this is all-new work that now has much broader appeal and applicability.
# Add additional group actions over time (migrate, replicate, retype, clone, …). At this point, new group capabilities become vertical slices that are simple to add incrementally.
 
Because we implemented CGs just recently as an experimental feature, we have the freedom to replace that code without deprecation or upgrade considerations. Steps 1-8 would get Manila to parity with the CG feature added in Liberty, would require only a few person-weeks of effort, and would better position Manila for long-term evolution and supportability.
=== Implementation details ===
There are few things to note, several of which were already solved during the CG work. 

Groups are first-class objects in Manila, and operations on groups are treated as atomic. For example, a user will see a group snapshot but not the individual snapshots of group members. To enable support by as many backends as possible, Manila will still maintain DB objects for both group and member snapshots, just as was done with CGs. 

A few actions, such as extend & shrink, are inherently applicable only to individual shares. One could theoretically apply extend to a group, increasing the size of each member, but that seems like an unlikely use case. Any actions in this category must remain available to group members, although most actions such as clone or snapshot would be available only at the group level and not on its members. 

A group is limited to a single backend. Allowing groups that span backends is theoretically possible, but that would require fanout of operations from the API layer to multiple share managers across the asynchronous event bus, which would lead to complicated synchronization and state management for little operational benefit. 

As was done with Manila CGs, a driver may optionally limit a group to either the confines of a pool or an entire backend. It is known that pools are the unit of data motion (i.e. replication or migration) for some backends, so we think drivers need this flexibility. 

We considered reusing share types for groups as well. But share types include a set of public extra specs that may not map well to groups. And by adding group types as a separate object, there can be little confusion about their purpose and use. 

Some have noted that extra specs tend to be treated as an AND operation, where all features must be available for a backend to be chosen. It is conceivable that even if a backend can replicate a group or take consistent snapshots of a group, it might not be able to perform both operations on the same group. But this problem already exists with share types and hasn’t been a serious issue. Creating types is an admin-only operation and the burden remains on the admin to understand the capabilities of the backends in use and to create the share types and share group types appropriately. 

Note that this proposal explicitly does not address pool or backend replication, which is fundamentally different. Actions on shares or share groups are intended for tenants, whereas a pool or backend can contain data from multiple tenants. So pool or backend operations, while serving potentially valuable use cases, are inherently admin-only workflows that would be designed and exercised differently should Manila support them.

== Acknowledgments ==
Most of the analysis and recommendations contained herein apply equally well to Cinder, which is arguably much further down the path of crippling complexity. 

This proposal was developed by cknight with input from Manila community members bswartz, ameade, tbarron, and cfouts. The Cinder conversations about grouping, as well as the groundbreaking contributions of CGs to Cinder, all of which inspired this proposal, were led by xyang.

Manila/design/manila-generic-groups

2015-12-16T21:36:15Z

Clinton Knight: /* Acknowledgments */

= Manila generic groups =
== Executive summary ==
Manila needs a grouping construct that, like shares, is a 1st-class atomic data type. Our experience with CGs has demonstrated the complexity of adding a grouping capability, yet there are other use cases such as migration, replication, and backup in which some storage controllers could only offer such features on share groups. CGs also highlighted the poor optics of an advanced feature with comparatively little potential for vendor support. And adding new grouping constructs for each new feature is not technically feasible. All of the above may be addressed by generic groups, which we think is a clean extension to the original architecture of Manila.
== An Intractable Matrix ==
Manila is at an exciting time in its development, where all the simple features are available and the community is focused on adding high-value features such as migration, replication, and consistency groups. Experimental code is available for each of these, but little consideration has been given to the obvious need for these complex features to interact not only with each other but also with whatever additional features were added later. And looking a little deeper, it is apparent that each of these has limitations that could be solved by a single architectural enhancement.
=== Consistency groups ===
Consistency groups (CGs) are the only construct in Manila that operates on a group of shares. For some, a CG implies a guarantee of ordered writes by a storage controller, while for others a CG is a mechanism for taking consistent point-in-time snapshots of a set of shares. A group with either attribute may have value by itself, but the CGs implementation doesn’t distinguish between them. 

CGs are a highly specialized construct with dedicated CLI commands, REST APIs, database tables, scheduler filters, and driver APIs. None of these are reusable for anything else, and presently none of the rest of Manila has any awareness of CGs. Even worse, despite all the complexity of the CG feature, only a small minority of storage backends can support them, so the code-to-value ratio is very low. Finally, the limited availability of CGs ensures the user experience is inconsistent between clouds.
=== Replication ===
Replication is another high value feature, and the Manila implementation is arguably clean and flexible, but as constituted there is no core ability to replicate groups of shares. It seems reasonable to implement the feature iteratively, beginning with replication of single shares. However, there has been the tacit acknowledgement that some backends would not be able to replicate individual shares, even though those same backends could potentially replicate a group of shares if the group were constituted in a certain way. 

Following the precedent established with CGs, one approach would be to introduce ‘replication groups’ whereby multiple shares could be replicated together. But this would introduce yet another set of APIs, tables, etc. that are too specialized to use for anything else. 

Another approach might be to define CGs as the grouping unit of replication, but this makes little sense because consistency groups (however we define them) and replication are distinct features that shouldn’t be artificially bundled for the sake of expediency of definition or implementation.
=== Migration ===
Just as some backends may only be able to replicate shares in groups, it follows that those backends may also need to migrate shares in groups. And in the case of CGs, it doesn’t make sense to migrate CG members individually; the whole group must be moved, requiring the replication engine to be CG aware. 

Consider the following reasonable use cases:
* Replicate a consistency group
* Snapshot a replication group
* Migrate a replication group
* Retype shares in a consistency group
* Retype shares in a replication group
* Backup a consistency group
* Backup a replication group
* Apply a common policy to a set of shares
 
On the current path, the support matrix has a dimension dedicated solely to features, so each feature must be coded to be interoperable with every other feature. This quickly becomes a support and testing nightmare, and it becomes exponentially more complicated to add more features going forward. 

Fundamentally, the problem is that we are adding features without an underlying architectural framework on which to hang them. To escape the matrix, we must step back and rethink a few things.

== A simple solution ==
So to arrive at a solution, let’s enumerate what we have:
* Primitive objects (shares) with supported operations controlled by share types that vary by backend
* A very specific group object (CGs) with limited support potential by backends
* A number of APIs (CGs, etc.) that many users can’t use at all and appear grafted into the project
 
And what would we prefer:
* A uniform API and user experience that varies as little as possible by backend
* A clean way to group shares so we can do CGs, multi-object replication, migration, etc.
 
How do we get here?
* Universally available groups of primitive objects with supported operations that vary by backend
 
=== Idea #1: Introduce a generic Share Group object ===
Just as Manila has ‘shares’, it should also have ‘share groups’. In its simplest form, unlike CGs, a share group should not guarantee any specialized operation. Instead, it should merely constitute an atomic Manila data type on which nearly any Manila action is available. For example, given a share group, the user should be able to select the group in the Manila UI and invoke features such as snapshot, clone, backup, migrate, replicate, retype, etc. 

In the general case, group actions are handled by the share manager layer (or potentially the common share driver superclass). For example, invoking ‘snapshot’ on a group causes the share manager to take a snapshot of each group member individually. The resulting group snapshot object may be used to create a new group, not unlike how CG snapshots were implemented. 

There are numerous advantages to this approach. Every driver, no matter how unsophisticated, can leverage the manageability goodness of groups. The user experience is uniform, since groups are always available regardless of which backends are present and because nearly every Manila action available on shares is also available on groups. 

Of course, feature-rich backends would like to add their secret sauce, whether it be CGs, group-based replication, or whatever else comes along that might be easier/cheaper/faster to do to groups of shares. That brings us to related idea.
=== Idea #2: Introduce Share Group types ===
Just as Manila has ‘share types’, it should also have ‘share group types’. Any driver that can perform a group operation in an advantaged way may report that as a group capability, such as:
* Ordered writes
* Consistent snapshots
* Group replication
* Group migration
* Group backup
* Group modification
 
As with share types, the cloud administrator predefines share group types that may contain extra specs corresponding to the group capabilities reported by the backends. And when creating a group, the user specifies one or more share types as well as a group type, and the scheduler then creates the group on one of the backends that match the specified share and share group types. 

Anytime a group action, such as ‘snapshot’, comes into the share manager, the manager checks whether its driver offers an advantaged implementation of that operation. If not, the manager handles the workflow itself as described above. But if so, the manager routes the workflow to its driver for fulfillment. 

The advantages of this approach should be obvious. Development and testing are simplified because there isn’t a need to define and test a different set of group management APIs for each feature, or to test every combination of every feature. Instead of becoming an N-by-N matrix of interacting features, Manila largely becomes an N-by-2 matrix of actions that may be invoked on either individual shares or share groups. Users and admins are already familiar with share types, so introducing share group types would seem a natural and consistent evolution of the same foundational concept.

The matrix would look like:
{| class="wikitable"
|-
! Share action !! Share Group action
|-
| Create (volume type) || Create (volume types, group type)
|-
| Delete || Delete (group, optionally all members)
|-
| Snapshot || Snapshot (may or may not be a CG snapshot)
|-
| Create from snapshot / Clone || Create from group snapshot / Clone group
|-
| Replicate || Replicate
|-
| Backup || Backup
|-
| Retype || Retype
|-
| Migrate || Migrate
|-
| Extend / shrink || N/A
|}

== Manila implementation plan ==
Implementing generic groups in Manila should be a straightforward series of steps:
# Implement generic groups. Because we did CGs, we already know all parts of the codebase that must change to support any kind of group. So the simplest approach is to modify the CG code to morph it into the generic groups feature.
# Add the share group type feature by duplicating and customizing the share type code.
# Enhance the scheduler to place groups according to share group types. Like #1, this is already informed by the CG project.
# Implement group snapshots in the share manager to demonstrate group snapshots in any driver.
# Plumb group snapshots to a CG-capable driver to demonstrate CG functionality in the new framework.
# Update Tempest to cover all of the above.
# Update Manila client to change CGs to share groups.
# Enhance Manila client with share group types.
# Add share group support to Horizon. We never built CG support into Horizon, so this is all-new work that now has much broader appeal and applicability.
# Add additional group actions over time (migrate, replicate, retype, clone, …). At this point, new group capabilities become vertical slices that are simple to add incrementally.
 
Because we implemented CGs just recently as an experimental feature, we have the freedom to replace that code without deprecation or upgrade considerations. Steps 1-8 would get Manila to parity with the CG feature added in Liberty, would require only a few person-weeks of effort, and would better position Manila for long-term evolution and supportability.
=== Implementation details ===
There are few things to note, several of which were already solved during the CG work. 

Groups are first-class objects in Manila, and operations on groups are treated as atomic. For example, a user will see a group snapshot but not the individual snapshots of group members. To enable support by as many backends as possible, Manila will still maintain DB objects for both group and member snapshots, just as was done with CGs. 

A few actions, such as extend & shrink, are inherently applicable only to individual shares. One could theoretically apply extend to a group, increasing the size of each member, but that seems like an unlikely use case. Any actions in this category must remain available to group members, although most actions such as clone or snapshot would be available only at the group level and not on its members. 

A group is limited to a single backend. Allowing groups that span backends is theoretically possible, but that would require fanout of operations from the API layer to multiple share managers across the asynchronous event bus, which would lead to complicated synchronization and state management for little operational benefit. 

As was done with Manila CGs, a driver may optionally limit a group to either the confines of a pool or an entire backend. It is known that pools are the unit of data motion (i.e. replication or migration) for some backends, so we think drivers need this flexibility. 

We considered reusing share types for groups as well. But share types include a set of public extra specs that may not map well to groups. And by adding group types as a separate object, there can be little confusion about their purpose and use. 

Some have noted that extra specs tend to be treated as an AND operation, where all features must be available for a backend to be chosen. It is conceivable that even if a backend can replicate a group or take consistent snapshots of a group, it might not be able to perform both operations on the same group. But this problem already exists with share types and hasn’t been a serious issue. Creating types is an admin-only operation and the burden remains on the admin to understand the capabilities of the backends in use and to create the share types and share group types appropriately. 

Note that this proposal explicitly does not address pool or backend replication, which is fundamentally different. Actions on shares or share groups are intended for tenants, whereas a pool or backend can contain data from multiple tenants. So pool or backend operations, while serving potentially valuable use cases, are inherently admin-only workflows that would be designed and exercised differently should Manila support them.

== Acknowledgments ==
Most of the analysis and recommendations contained herein apply equally well to Cinder, which is arguably much further down the path of crippling complexity. 

This proposal was developed by cknight with input from Manila community members bswartz, ameade, tbarron, and cfouts. The ongoing Cinder conversations about grouping, as well as the groundbreaking contributions of CGs to Cinder, all of which inspired this proposal, were led by xyang.

Manila/design/manila-generic-groups

2015-12-16T21:31:24Z

Clinton Knight: /* Idea #2: Introduce Share Group types */

= Manila generic groups =
== Executive summary ==
Manila needs a grouping construct that, like shares, is a 1st-class atomic data type. Our experience with CGs has demonstrated the complexity of adding a grouping capability, yet there are other use cases such as migration, replication, and backup in which some storage controllers could only offer such features on share groups. CGs also highlighted the poor optics of an advanced feature with comparatively little potential for vendor support. And adding new grouping constructs for each new feature is not technically feasible. All of the above may be addressed by generic groups, which we think is a clean extension to the original architecture of Manila.
== An Intractable Matrix ==
Manila is at an exciting time in its development, where all the simple features are available and the community is focused on adding high-value features such as migration, replication, and consistency groups. Experimental code is available for each of these, but little consideration has been given to the obvious need for these complex features to interact not only with each other but also with whatever additional features were added later. And looking a little deeper, it is apparent that each of these has limitations that could be solved by a single architectural enhancement.
=== Consistency groups ===
Consistency groups (CGs) are the only construct in Manila that operates on a group of shares. For some, a CG implies a guarantee of ordered writes by a storage controller, while for others a CG is a mechanism for taking consistent point-in-time snapshots of a set of shares. A group with either attribute may have value by itself, but the CGs implementation doesn’t distinguish between them. 

CGs are a highly specialized construct with dedicated CLI commands, REST APIs, database tables, scheduler filters, and driver APIs. None of these are reusable for anything else, and presently none of the rest of Manila has any awareness of CGs. Even worse, despite all the complexity of the CG feature, only a small minority of storage backends can support them, so the code-to-value ratio is very low. Finally, the limited availability of CGs ensures the user experience is inconsistent between clouds.
=== Replication ===
Replication is another high value feature, and the Manila implementation is arguably clean and flexible, but as constituted there is no core ability to replicate groups of shares. It seems reasonable to implement the feature iteratively, beginning with replication of single shares. However, there has been the tacit acknowledgement that some backends would not be able to replicate individual shares, even though those same backends could potentially replicate a group of shares if the group were constituted in a certain way. 

Following the precedent established with CGs, one approach would be to introduce ‘replication groups’ whereby multiple shares could be replicated together. But this would introduce yet another set of APIs, tables, etc. that are too specialized to use for anything else. 

Another approach might be to define CGs as the grouping unit of replication, but this makes little sense because consistency groups (however we define them) and replication are distinct features that shouldn’t be artificially bundled for the sake of expediency of definition or implementation.
=== Migration ===
Just as some backends may only be able to replicate shares in groups, it follows that those backends may also need to migrate shares in groups. And in the case of CGs, it doesn’t make sense to migrate CG members individually; the whole group must be moved, requiring the replication engine to be CG aware. 

Consider the following reasonable use cases:
* Replicate a consistency group
* Snapshot a replication group
* Migrate a replication group
* Retype shares in a consistency group
* Retype shares in a replication group
* Backup a consistency group
* Backup a replication group
* Apply a common policy to a set of shares
 
On the current path, the support matrix has a dimension dedicated solely to features, so each feature must be coded to be interoperable with every other feature. This quickly becomes a support and testing nightmare, and it becomes exponentially more complicated to add more features going forward. 

Fundamentally, the problem is that we are adding features without an underlying architectural framework on which to hang them. To escape the matrix, we must step back and rethink a few things.

== A simple solution ==
So to arrive at a solution, let’s enumerate what we have:
* Primitive objects (shares) with supported operations controlled by share types that vary by backend
* A very specific group object (CGs) with limited support potential by backends
* A number of APIs (CGs, etc.) that many users can’t use at all and appear grafted into the project
 
And what would we prefer:
* A uniform API and user experience that varies as little as possible by backend
* A clean way to group shares so we can do CGs, multi-object replication, migration, etc.
 
How do we get here?
* Universally available groups of primitive objects with supported operations that vary by backend
 
=== Idea #1: Introduce a generic Share Group object ===
Just as Manila has ‘shares’, it should also have ‘share groups’. In its simplest form, unlike CGs, a share group should not guarantee any specialized operation. Instead, it should merely constitute an atomic Manila data type on which nearly any Manila action is available. For example, given a share group, the user should be able to select the group in the Manila UI and invoke features such as snapshot, clone, backup, migrate, replicate, retype, etc. 

In the general case, group actions are handled by the share manager layer (or potentially the common share driver superclass). For example, invoking ‘snapshot’ on a group causes the share manager to take a snapshot of each group member individually. The resulting group snapshot object may be used to create a new group, not unlike how CG snapshots were implemented. 

There are numerous advantages to this approach. Every driver, no matter how unsophisticated, can leverage the manageability goodness of groups. The user experience is uniform, since groups are always available regardless of which backends are present and because nearly every Manila action available on shares is also available on groups. 

Of course, feature-rich backends would like to add their secret sauce, whether it be CGs, group-based replication, or whatever else comes along that might be easier/cheaper/faster to do to groups of shares. That brings us to related idea.
=== Idea #2: Introduce Share Group types ===
Just as Manila has ‘share types’, it should also have ‘share group types’. Any driver that can perform a group operation in an advantaged way may report that as a group capability, such as:
* Ordered writes
* Consistent snapshots
* Group replication
* Group migration
* Group backup
* Group modification
 
As with share types, the cloud administrator predefines share group types that may contain extra specs corresponding to the group capabilities reported by the backends. And when creating a group, the user specifies one or more share types as well as a group type, and the scheduler then creates the group on one of the backends that match the specified share and share group types. 

Anytime a group action, such as ‘snapshot’, comes into the share manager, the manager checks whether its driver offers an advantaged implementation of that operation. If not, the manager handles the workflow itself as described above. But if so, the manager routes the workflow to its driver for fulfillment. 

The advantages of this approach should be obvious. Development and testing are simplified because there isn’t a need to define and test a different set of group management APIs for each feature, or to test every combination of every feature. Instead of becoming an N-by-N matrix of interacting features, Manila largely becomes an N-by-2 matrix of actions that may be invoked on either individual shares or share groups. Users and admins are already familiar with share types, so introducing share group types would seem a natural and consistent evolution of the same foundational concept.

The matrix would look like:
{| class="wikitable"
|-
! Share action !! Share Group action
|-
| Create (volume type) || Create (volume types, group type)
|-
| Delete || Delete (group, optionally all members)
|-
| Snapshot || Snapshot (may or may not be a CG snapshot)
|-
| Create from snapshot / Clone || Create from group snapshot / Clone group
|-
| Replicate || Replicate
|-
| Backup || Backup
|-
| Retype || Retype
|-
| Migrate || Migrate
|-
| Extend / shrink || N/A
|}

== Manila implementation plan ==
Implementing generic groups in Manila should be a straightforward series of steps:
# Implement generic groups. Because we did CGs, we already know all parts of the codebase that must change to support any kind of group. So the simplest approach is to modify the CG code to morph it into the generic groups feature.
# Add the share group type feature by duplicating and customizing the share type code.
# Enhance the scheduler to place groups according to share group types. Like #1, this is already informed by the CG project.
# Implement group snapshots in the share manager to demonstrate group snapshots in any driver.
# Plumb group snapshots to a CG-capable driver to demonstrate CG functionality in the new framework.
# Update Tempest to cover all of the above.
# Update Manila client to change CGs to share groups.
# Enhance Manila client with share group types.
# Add share group support to Horizon. We never built CG support into Horizon, so this is all-new work that now has much broader appeal and applicability.
# Add additional group actions over time (migrate, replicate, retype, clone, …). At this point, new group capabilities become vertical slices that are simple to add incrementally.
 
Because we implemented CGs just recently as an experimental feature, we have the freedom to replace that code without deprecation or upgrade considerations. Steps 1-8 would get Manila to parity with the CG feature added in Liberty, would require only a few person-weeks of effort, and would better position Manila for long-term evolution and supportability.
=== Implementation details ===
There are few things to note, several of which were already solved during the CG work. 

Groups are first-class objects in Manila, and operations on groups are treated as atomic. For example, a user will see a group snapshot but not the individual snapshots of group members. To enable support by as many backends as possible, Manila will still maintain DB objects for both group and member snapshots, just as was done with CGs. 

A few actions, such as extend & shrink, are inherently applicable only to individual shares. One could theoretically apply extend to a group, increasing the size of each member, but that seems like an unlikely use case. Any actions in this category must remain available to group members, although most actions such as clone or snapshot would be available only at the group level and not on its members. 

A group is limited to a single backend. Allowing groups that span backends is theoretically possible, but that would require fanout of operations from the API layer to multiple share managers across the asynchronous event bus, which would lead to complicated synchronization and state management for little operational benefit. 

As was done with Manila CGs, a driver may optionally limit a group to either the confines of a pool or an entire backend. It is known that pools are the unit of data motion (i.e. replication or migration) for some backends, so we think drivers need this flexibility. 

We considered reusing share types for groups as well. But share types include a set of public extra specs that may not map well to groups. And by adding group types as a separate object, there can be little confusion about their purpose and use. 

Some have noted that extra specs tend to be treated as an AND operation, where all features must be available for a backend to be chosen. It is conceivable that even if a backend can replicate a group or take consistent snapshots of a group, it might not be able to perform both operations on the same group. But this problem already exists with share types and hasn’t been a serious issue. Creating types is an admin-only operation and the burden remains on the admin to understand the capabilities of the backends in use and to create the share types and share group types appropriately. 

Note that this proposal explicitly does not address pool or backend replication, which is fundamentally different. Actions on shares or share groups are intended for tenants, whereas a pool or backend can contain data from multiple tenants. So pool or backend operations, while serving potentially valuable use cases, are inherently admin-only workflows that would be designed and exercised differently should Manila support them.

== Acknowledgments ==
Most of the analysis and recommendations contained herein apply equally well to Cinder, which is arguably much further down the path of crippling complexity. 

This proposal was developed by cknight with input from Manila community members bswartz, ameade, tbarron, and cfouts.

Manila/design/manila-generic-groups

2015-12-16T21:29:11Z

Clinton Knight: /* Idea #2: Introduce Share Group types */

= Manila generic groups =
== Executive summary ==
Manila needs a grouping construct that, like shares, is a 1st-class atomic data type. Our experience with CGs has demonstrated the complexity of adding a grouping capability, yet there are other use cases such as migration, replication, and backup in which some storage controllers could only offer such features on share groups. CGs also highlighted the poor optics of an advanced feature with comparatively little potential for vendor support. And adding new grouping constructs for each new feature is not technically feasible. All of the above may be addressed by generic groups, which we think is a clean extension to the original architecture of Manila.
== An Intractable Matrix ==
Manila is at an exciting time in its development, where all the simple features are available and the community is focused on adding high-value features such as migration, replication, and consistency groups. Experimental code is available for each of these, but little consideration has been given to the obvious need for these complex features to interact not only with each other but also with whatever additional features were added later. And looking a little deeper, it is apparent that each of these has limitations that could be solved by a single architectural enhancement.
=== Consistency groups ===
Consistency groups (CGs) are the only construct in Manila that operates on a group of shares. For some, a CG implies a guarantee of ordered writes by a storage controller, while for others a CG is a mechanism for taking consistent point-in-time snapshots of a set of shares. A group with either attribute may have value by itself, but the CGs implementation doesn’t distinguish between them. 

CGs are a highly specialized construct with dedicated CLI commands, REST APIs, database tables, scheduler filters, and driver APIs. None of these are reusable for anything else, and presently none of the rest of Manila has any awareness of CGs. Even worse, despite all the complexity of the CG feature, only a small minority of storage backends can support them, so the code-to-value ratio is very low. Finally, the limited availability of CGs ensures the user experience is inconsistent between clouds.
=== Replication ===
Replication is another high value feature, and the Manila implementation is arguably clean and flexible, but as constituted there is no core ability to replicate groups of shares. It seems reasonable to implement the feature iteratively, beginning with replication of single shares. However, there has been the tacit acknowledgement that some backends would not be able to replicate individual shares, even though those same backends could potentially replicate a group of shares if the group were constituted in a certain way. 

Following the precedent established with CGs, one approach would be to introduce ‘replication groups’ whereby multiple shares could be replicated together. But this would introduce yet another set of APIs, tables, etc. that are too specialized to use for anything else. 

Another approach might be to define CGs as the grouping unit of replication, but this makes little sense because consistency groups (however we define them) and replication are distinct features that shouldn’t be artificially bundled for the sake of expediency of definition or implementation.
=== Migration ===
Just as some backends may only be able to replicate shares in groups, it follows that those backends may also need to migrate shares in groups. And in the case of CGs, it doesn’t make sense to migrate CG members individually; the whole group must be moved, requiring the replication engine to be CG aware. 

Consider the following reasonable use cases:
* Replicate a consistency group
* Snapshot a replication group
* Migrate a replication group
* Retype shares in a consistency group
* Retype shares in a replication group
* Backup a consistency group
* Backup a replication group
* Apply a common policy to a set of shares
 
On the current path, the support matrix has a dimension dedicated solely to features, so each feature must be coded to be interoperable with every other feature. This quickly becomes a support and testing nightmare, and it becomes exponentially more complicated to add more features going forward. 

Fundamentally, the problem is that we are adding features without an underlying architectural framework on which to hang them. To escape the matrix, we must step back and rethink a few things.

== A simple solution ==
So to arrive at a solution, let’s enumerate what we have:
* Primitive objects (shares) with supported operations controlled by share types that vary by backend
* A very specific group object (CGs) with limited support potential by backends
* A number of APIs (CGs, etc.) that many users can’t use at all and appear grafted into the project
 
And what would we prefer:
* A uniform API and user experience that varies as little as possible by backend
* A clean way to group shares so we can do CGs, multi-object replication, migration, etc.
 
How do we get here?
* Universally available groups of primitive objects with supported operations that vary by backend
 
=== Idea #1: Introduce a generic Share Group object ===
Just as Manila has ‘shares’, it should also have ‘share groups’. In its simplest form, unlike CGs, a share group should not guarantee any specialized operation. Instead, it should merely constitute an atomic Manila data type on which nearly any Manila action is available. For example, given a share group, the user should be able to select the group in the Manila UI and invoke features such as snapshot, clone, backup, migrate, replicate, retype, etc. 

In the general case, group actions are handled by the share manager layer (or potentially the common share driver superclass). For example, invoking ‘snapshot’ on a group causes the share manager to take a snapshot of each group member individually. The resulting group snapshot object may be used to create a new group, not unlike how CG snapshots were implemented. 

There are numerous advantages to this approach. Every driver, no matter how unsophisticated, can leverage the manageability goodness of groups. The user experience is uniform, since groups are always available regardless of which backends are present and because nearly every Manila action available on shares is also available on groups. 

Of course, feature-rich backends would like to add their secret sauce, whether it be CGs, group-based replication, or whatever else comes along that might be easier/cheaper/faster to do to groups of shares. That brings us to related idea.
=== Idea #2: Introduce Share Group types ===
Just as Manila has ‘share types’, it should also have ‘share group types’. Any driver that can perform a group operation in an advantaged way may report that as a group capability, such as:
* Ordered writes
* Consistent snapshots
* Group replication
* Group migration
* Group backup
* Group modification
 
As with share types, the cloud administrator predefines share group types that may contain extra specs corresponding to the group capabilities reported by the backends. And when creating a group, the user specifies one or more share types as well as a group type, and the scheduler then creates the group on one of the backends that match the specified share and share group types. 

Anytime a group action, such as ‘snapshot’, comes into the share manager, the manager checks whether its driver offers an advantaged implementation of that operation. If not, the manager handles the workflow itself as described above. But if so, the manager routes the workflow to its driver for fulfillment. 

The advantages of this approach should be obvious. Manila can add a single set of group APIs that can service all current and future group operations, including CG snapshots, replication, migration, retype, backup, etc. Development and testing are simplified because there isn’t a need to define and test a different set of group APIs for each feature, or to test every combination of every feature. Users and admins are already familiar with share types, so introducing share group types would seem a natural and consistent evolution of the same foundational concept. Instead of becoming an N-by-N matrix of interacting features, Manila largely becomes an N-by-2 matrix of actions that may be invoked on either individual shares or share groups.

The matrix would look like:
{| class="wikitable"
|-
! Share action !! Share Group action
|-
| Create (volume type) || Create (volume types, group type)
|-
| Delete || Delete (group, optionally all members)
|-
| Snapshot || Snapshot (may or may not be a CG snapshot)
|-
| Create from snapshot / Clone || Create from group snapshot / Clone group
|-
| Replicate || Replicate
|-
| Backup || Backup
|-
| Retype || Retype
|-
| Migrate || Migrate
|-
| Extend / shrink || N/A
|}

== Manila implementation plan ==
Implementing generic groups in Manila should be a straightforward series of steps:
# Implement generic groups. Because we did CGs, we already know all parts of the codebase that must change to support any kind of group. So the simplest approach is to modify the CG code to morph it into the generic groups feature.
# Add the share group type feature by duplicating and customizing the share type code.
# Enhance the scheduler to place groups according to share group types. Like #1, this is already informed by the CG project.
# Implement group snapshots in the share manager to demonstrate group snapshots in any driver.
# Plumb group snapshots to a CG-capable driver to demonstrate CG functionality in the new framework.
# Update Tempest to cover all of the above.
# Update Manila client to change CGs to share groups.
# Enhance Manila client with share group types.
# Add share group support to Horizon. We never built CG support into Horizon, so this is all-new work that now has much broader appeal and applicability.
# Add additional group actions over time (migrate, replicate, retype, clone, …). At this point, new group capabilities become vertical slices that are simple to add incrementally.
 
Because we implemented CGs just recently as an experimental feature, we have the freedom to replace that code without deprecation or upgrade considerations. Steps 1-8 would get Manila to parity with the CG feature added in Liberty, would require only a few person-weeks of effort, and would better position Manila for long-term evolution and supportability.
=== Implementation details ===
There are few things to note, several of which were already solved during the CG work. 

Groups are first-class objects in Manila, and operations on groups are treated as atomic. For example, a user will see a group snapshot but not the individual snapshots of group members. To enable support by as many backends as possible, Manila will still maintain DB objects for both group and member snapshots, just as was done with CGs. 

A few actions, such as extend & shrink, are inherently applicable only to individual shares. One could theoretically apply extend to a group, increasing the size of each member, but that seems like an unlikely use case. Any actions in this category must remain available to group members, although most actions such as clone or snapshot would be available only at the group level and not on its members. 

A group is limited to a single backend. Allowing groups that span backends is theoretically possible, but that would require fanout of operations from the API layer to multiple share managers across the asynchronous event bus, which would lead to complicated synchronization and state management for little operational benefit. 

As was done with Manila CGs, a driver may optionally limit a group to either the confines of a pool or an entire backend. It is known that pools are the unit of data motion (i.e. replication or migration) for some backends, so we think drivers need this flexibility. 

We considered reusing share types for groups as well. But share types include a set of public extra specs that may not map well to groups. And by adding group types as a separate object, there can be little confusion about their purpose and use. 

Some have noted that extra specs tend to be treated as an AND operation, where all features must be available for a backend to be chosen. It is conceivable that even if a backend can replicate a group or take consistent snapshots of a group, it might not be able to perform both operations on the same group. But this problem already exists with share types and hasn’t been a serious issue. Creating types is an admin-only operation and the burden remains on the admin to understand the capabilities of the backends in use and to create the share types and share group types appropriately. 

Note that this proposal explicitly does not address pool or backend replication, which is fundamentally different. Actions on shares or share groups are intended for tenants, whereas a pool or backend can contain data from multiple tenants. So pool or backend operations, while serving potentially valuable use cases, are inherently admin-only workflows that would be designed and exercised differently should Manila support them.

== Acknowledgments ==
Most of the analysis and recommendations contained herein apply equally well to Cinder, which is arguably much further down the path of crippling complexity. 

This proposal was developed by cknight with input from Manila community members bswartz, ameade, tbarron, and cfouts.

Manila/design/manila-generic-groups

2015-12-16T20:49:57Z

Clinton Knight: /* Manila implementation plan */

= Manila generic groups =
== Executive summary ==
Manila needs a grouping construct that, like shares, is a 1st-class atomic data type. Our experience with CGs has demonstrated the complexity of adding a grouping capability, yet there are other use cases such as migration, replication, and backup in which some storage controllers could only offer such features on share groups. CGs also highlighted the poor optics of an advanced feature with comparatively little potential for vendor support. And adding new grouping constructs for each new feature is not technically feasible. All of the above may be addressed by generic groups, which we think is a clean extension to the original architecture of Manila.
== An Intractable Matrix ==
Manila is at an exciting time in its development, where all the simple features are available and the community is focused on adding high-value features such as migration, replication, and consistency groups. Experimental code is available for each of these, but little consideration has been given to the obvious need for these complex features to interact not only with each other but also with whatever additional features were added later. And looking a little deeper, it is apparent that each of these has limitations that could be solved by a single architectural enhancement.
=== Consistency groups ===
Consistency groups (CGs) are the only construct in Manila that operates on a group of shares. For some, a CG implies a guarantee of ordered writes by a storage controller, while for others a CG is a mechanism for taking consistent point-in-time snapshots of a set of shares. A group with either attribute may have value by itself, but the CGs implementation doesn’t distinguish between them. 

CGs are a highly specialized construct with dedicated CLI commands, REST APIs, database tables, scheduler filters, and driver APIs. None of these are reusable for anything else, and presently none of the rest of Manila has any awareness of CGs. Even worse, despite all the complexity of the CG feature, only a small minority of storage backends can support them, so the code-to-value ratio is very low. Finally, the limited availability of CGs ensures the user experience is inconsistent between clouds.
=== Replication ===
Replication is another high value feature, and the Manila implementation is arguably clean and flexible, but as constituted there is no core ability to replicate groups of shares. It seems reasonable to implement the feature iteratively, beginning with replication of single shares. However, there has been the tacit acknowledgement that some backends would not be able to replicate individual shares, even though those same backends could potentially replicate a group of shares if the group were constituted in a certain way. 

Following the precedent established with CGs, one approach would be to introduce ‘replication groups’ whereby multiple shares could be replicated together. But this would introduce yet another set of APIs, tables, etc. that are too specialized to use for anything else. 

Another approach might be to define CGs as the grouping unit of replication, but this makes little sense because consistency groups (however we define them) and replication are distinct features that shouldn’t be artificially bundled for the sake of expediency of definition or implementation.
=== Migration ===
Just as some backends may only be able to replicate shares in groups, it follows that those backends may also need to migrate shares in groups. And in the case of CGs, it doesn’t make sense to migrate CG members individually; the whole group must be moved, requiring the replication engine to be CG aware. 

Consider the following reasonable use cases:
* Replicate a consistency group
* Snapshot a replication group
* Migrate a replication group
* Retype shares in a consistency group
* Retype shares in a replication group
* Backup a consistency group
* Backup a replication group
* Apply a common policy to a set of shares
 
On the current path, the support matrix has a dimension dedicated solely to features, so each feature must be coded to be interoperable with every other feature. This quickly becomes a support and testing nightmare, and it becomes exponentially more complicated to add more features going forward. 

Fundamentally, the problem is that we are adding features without an underlying architectural framework on which to hang them. To escape the matrix, we must step back and rethink a few things.

== A simple solution ==
So to arrive at a solution, let’s enumerate what we have:
* Primitive objects (shares) with supported operations controlled by share types that vary by backend
* A very specific group object (CGs) with limited support potential by backends
* A number of APIs (CGs, etc.) that many users can’t use at all and appear grafted into the project
 
And what would we prefer:
* A uniform API and user experience that varies as little as possible by backend
* A clean way to group shares so we can do CGs, multi-object replication, migration, etc.
 
How do we get here?
* Universally available groups of primitive objects with supported operations that vary by backend
 
=== Idea #1: Introduce a generic Share Group object ===
Just as Manila has ‘shares’, it should also have ‘share groups’. In its simplest form, unlike CGs, a share group should not guarantee any specialized operation. Instead, it should merely constitute an atomic Manila data type on which nearly any Manila action is available. For example, given a share group, the user should be able to select the group in the Manila UI and invoke features such as snapshot, clone, backup, migrate, replicate, retype, etc. 

In the general case, group actions are handled by the share manager layer (or potentially the common share driver superclass). For example, invoking ‘snapshot’ on a group causes the share manager to take a snapshot of each group member individually. The resulting group snapshot object may be used to create a new group, not unlike how CG snapshots were implemented. 

There are numerous advantages to this approach. Every driver, no matter how unsophisticated, can leverage the manageability goodness of groups. The user experience is uniform, since groups are always available regardless of which backends are present and because nearly every Manila action available on shares is also available on groups. 

Of course, feature-rich backends would like to add their secret sauce, whether it be CGs, group-based replication, or whatever else comes along that might be easier/cheaper/faster to do to groups of shares. That brings us to related idea.
=== Idea #2: Introduce Share Group types ===
Just as Manila has ‘share types’, it should also have ‘share group types’. Any driver that can perform a group operation in an advantaged way may report that as a group capability, such as:
* Ordered writes
* Consistent snapshots
* Group replication
* Group migration
* Group backup
* Group modification
 
As with share types, the cloud administrator predefines share group types that may contain extra specs corresponding to the group capabilities reported by the backends. And when creating a group, the user specifies one or more share types as well as a group type, and the scheduler then creates the group on one of the backends that match the specified share and share group types. 

Anytime a group action, such as ‘snapshot’, comes into the share manager, the manager checks whether its driver offers an advantaged implementation of that operation. If not, the manager handles the workflow itself as described above. But if so, the manager routes the workflow to its driver for fulfillment. 

The advantages of this approach should be obvious. Manila can add a single set of group APIs that can service all current and future group operations, including CG snapshots, replication, migration, retype, backup, etc. Development and testing are simplified because there isn’t a need to define and test a different set of group APIs for each feature, or to test every combination of every feature. Users and admins are already familiar with share types, so introducing share group types would seem a natural and consistent evolution of the same foundational concept. Instead of becoming an N-by-N matrix of interacting features, Manila largely becomes an N-by-2 matrix of actions that may be invoked on either individual shares or share groups.
== Manila implementation plan ==
Implementing generic groups in Manila should be a straightforward series of steps:
# Implement generic groups. Because we did CGs, we already know all parts of the codebase that must change to support any kind of group. So the simplest approach is to modify the CG code to morph it into the generic groups feature.
# Add the share group type feature by duplicating and customizing the share type code.
# Enhance the scheduler to place groups according to share group types. Like #1, this is already informed by the CG project.
# Implement group snapshots in the share manager to demonstrate group snapshots in any driver.
# Plumb group snapshots to a CG-capable driver to demonstrate CG functionality in the new framework.
# Update Tempest to cover all of the above.
# Update Manila client to change CGs to share groups.
# Enhance Manila client with share group types.
# Add share group support to Horizon. We never built CG support into Horizon, so this is all-new work that now has much broader appeal and applicability.
# Add additional group actions over time (migrate, replicate, retype, clone, …). At this point, new group capabilities become vertical slices that are simple to add incrementally.
 
Because we implemented CGs just recently as an experimental feature, we have the freedom to replace that code without deprecation or upgrade considerations. Steps 1-8 would get Manila to parity with the CG feature added in Liberty, would require only a few person-weeks of effort, and would better position Manila for long-term evolution and supportability.
=== Implementation details ===
There are few things to note, several of which were already solved during the CG work. 

Groups are first-class objects in Manila, and operations on groups are treated as atomic. For example, a user will see a group snapshot but not the individual snapshots of group members. To enable support by as many backends as possible, Manila will still maintain DB objects for both group and member snapshots, just as was done with CGs. 

A few actions, such as extend & shrink, are inherently applicable only to individual shares. One could theoretically apply extend to a group, increasing the size of each member, but that seems like an unlikely use case. Any actions in this category must remain available to group members, although most actions such as clone or snapshot would be available only at the group level and not on its members. 

A group is limited to a single backend. Allowing groups that span backends is theoretically possible, but that would require fanout of operations from the API layer to multiple share managers across the asynchronous event bus, which would lead to complicated synchronization and state management for little operational benefit. 

As was done with Manila CGs, a driver may optionally limit a group to either the confines of a pool or an entire backend. It is known that pools are the unit of data motion (i.e. replication or migration) for some backends, so we think drivers need this flexibility. 

We considered reusing share types for groups as well. But share types include a set of public extra specs that may not map well to groups. And by adding group types as a separate object, there can be little confusion about their purpose and use. 

Some have noted that extra specs tend to be treated as an AND operation, where all features must be available for a backend to be chosen. It is conceivable that even if a backend can replicate a group or take consistent snapshots of a group, it might not be able to perform both operations on the same group. But this problem already exists with share types and hasn’t been a serious issue. Creating types is an admin-only operation and the burden remains on the admin to understand the capabilities of the backends in use and to create the share types and share group types appropriately. 

Note that this proposal explicitly does not address pool or backend replication, which is fundamentally different. Actions on shares or share groups are intended for tenants, whereas a pool or backend can contain data from multiple tenants. So pool or backend operations, while serving potentially valuable use cases, are inherently admin-only workflows that would be designed and exercised differently should Manila support them.

== Acknowledgments ==
Most of the analysis and recommendations contained herein apply equally well to Cinder, which is arguably much further down the path of crippling complexity. 

This proposal was developed by cknight with input from Manila community members bswartz, ameade, tbarron, and cfouts.

Manila/design/manila-generic-groups

2015-12-16T20:48:39Z

Clinton Knight:

= Manila generic groups =
== Executive summary ==
Manila needs a grouping construct that, like shares, is a 1st-class atomic data type. Our experience with CGs has demonstrated the complexity of adding a grouping capability, yet there are other use cases such as migration, replication, and backup in which some storage controllers could only offer such features on share groups. CGs also highlighted the poor optics of an advanced feature with comparatively little potential for vendor support. And adding new grouping constructs for each new feature is not technically feasible. All of the above may be addressed by generic groups, which we think is a clean extension to the original architecture of Manila.
== An Intractable Matrix ==
Manila is at an exciting time in its development, where all the simple features are available and the community is focused on adding high-value features such as migration, replication, and consistency groups. Experimental code is available for each of these, but little consideration has been given to the obvious need for these complex features to interact not only with each other but also with whatever additional features were added later. And looking a little deeper, it is apparent that each of these has limitations that could be solved by a single architectural enhancement.
=== Consistency groups ===
Consistency groups (CGs) are the only construct in Manila that operates on a group of shares. For some, a CG implies a guarantee of ordered writes by a storage controller, while for others a CG is a mechanism for taking consistent point-in-time snapshots of a set of shares. A group with either attribute may have value by itself, but the CGs implementation doesn’t distinguish between them. 

CGs are a highly specialized construct with dedicated CLI commands, REST APIs, database tables, scheduler filters, and driver APIs. None of these are reusable for anything else, and presently none of the rest of Manila has any awareness of CGs. Even worse, despite all the complexity of the CG feature, only a small minority of storage backends can support them, so the code-to-value ratio is very low. Finally, the limited availability of CGs ensures the user experience is inconsistent between clouds.
=== Replication ===
Replication is another high value feature, and the Manila implementation is arguably clean and flexible, but as constituted there is no core ability to replicate groups of shares. It seems reasonable to implement the feature iteratively, beginning with replication of single shares. However, there has been the tacit acknowledgement that some backends would not be able to replicate individual shares, even though those same backends could potentially replicate a group of shares if the group were constituted in a certain way. 

Following the precedent established with CGs, one approach would be to introduce ‘replication groups’ whereby multiple shares could be replicated together. But this would introduce yet another set of APIs, tables, etc. that are too specialized to use for anything else. 

Another approach might be to define CGs as the grouping unit of replication, but this makes little sense because consistency groups (however we define them) and replication are distinct features that shouldn’t be artificially bundled for the sake of expediency of definition or implementation.
=== Migration ===
Just as some backends may only be able to replicate shares in groups, it follows that those backends may also need to migrate shares in groups. And in the case of CGs, it doesn’t make sense to migrate CG members individually; the whole group must be moved, requiring the replication engine to be CG aware. 

Consider the following reasonable use cases:
* Replicate a consistency group
* Snapshot a replication group
* Migrate a replication group
* Retype shares in a consistency group
* Retype shares in a replication group
* Backup a consistency group
* Backup a replication group
* Apply a common policy to a set of shares
 
On the current path, the support matrix has a dimension dedicated solely to features, so each feature must be coded to be interoperable with every other feature. This quickly becomes a support and testing nightmare, and it becomes exponentially more complicated to add more features going forward. 

Fundamentally, the problem is that we are adding features without an underlying architectural framework on which to hang them. To escape the matrix, we must step back and rethink a few things.

== A simple solution ==
So to arrive at a solution, let’s enumerate what we have:
* Primitive objects (shares) with supported operations controlled by share types that vary by backend
* A very specific group object (CGs) with limited support potential by backends
* A number of APIs (CGs, etc.) that many users can’t use at all and appear grafted into the project
 
And what would we prefer:
* A uniform API and user experience that varies as little as possible by backend
* A clean way to group shares so we can do CGs, multi-object replication, migration, etc.
 
How do we get here?
* Universally available groups of primitive objects with supported operations that vary by backend
 
=== Idea #1: Introduce a generic Share Group object ===
Just as Manila has ‘shares’, it should also have ‘share groups’. In its simplest form, unlike CGs, a share group should not guarantee any specialized operation. Instead, it should merely constitute an atomic Manila data type on which nearly any Manila action is available. For example, given a share group, the user should be able to select the group in the Manila UI and invoke features such as snapshot, clone, backup, migrate, replicate, retype, etc. 

In the general case, group actions are handled by the share manager layer (or potentially the common share driver superclass). For example, invoking ‘snapshot’ on a group causes the share manager to take a snapshot of each group member individually. The resulting group snapshot object may be used to create a new group, not unlike how CG snapshots were implemented. 

There are numerous advantages to this approach. Every driver, no matter how unsophisticated, can leverage the manageability goodness of groups. The user experience is uniform, since groups are always available regardless of which backends are present and because nearly every Manila action available on shares is also available on groups. 

Of course, feature-rich backends would like to add their secret sauce, whether it be CGs, group-based replication, or whatever else comes along that might be easier/cheaper/faster to do to groups of shares. That brings us to related idea.
=== Idea #2: Introduce Share Group types ===
Just as Manila has ‘share types’, it should also have ‘share group types’. Any driver that can perform a group operation in an advantaged way may report that as a group capability, such as:
* Ordered writes
* Consistent snapshots
* Group replication
* Group migration
* Group backup
* Group modification
 
As with share types, the cloud administrator predefines share group types that may contain extra specs corresponding to the group capabilities reported by the backends. And when creating a group, the user specifies one or more share types as well as a group type, and the scheduler then creates the group on one of the backends that match the specified share and share group types. 

Anytime a group action, such as ‘snapshot’, comes into the share manager, the manager checks whether its driver offers an advantaged implementation of that operation. If not, the manager handles the workflow itself as described above. But if so, the manager routes the workflow to its driver for fulfillment. 

The advantages of this approach should be obvious. Manila can add a single set of group APIs that can service all current and future group operations, including CG snapshots, replication, migration, retype, backup, etc. Development and testing are simplified because there isn’t a need to define and test a different set of group APIs for each feature, or to test every combination of every feature. Users and admins are already familiar with share types, so introducing share group types would seem a natural and consistent evolution of the same foundational concept. Instead of becoming an N-by-N matrix of interacting features, Manila largely becomes an N-by-2 matrix of actions that may be invoked on either individual shares or share groups.
== Manila implementation plan ==
Implementing generic groups in Manila should be a straightforward series of steps:
# Implement generic groups. Because we did CGs, we already know all parts of the codebase that must change to support any kind of group. So the simplest approach is to modify the CG code to morph it into the generic groups feature.
# Add the share group type feature by duplicating and customizing the share type code.
# Enhance the scheduler to place groups according to share group types. Like #1, this is already informed by the CG project.
# Implement group snapshots in the share manager to demonstrate group snapshots in any driver.
# Plumb group snapshots to a CG-capable driver to demonstrate CG functionality in the new framework.
# Update Tempest to cover all of the above.
# Update Manila client to change CGs to share groups.
# Enhance Manila client with share group types.
# Add share group support to Horizon. We never built CG support into Horizon, so this is all-new work that now has much broader appeal and applicability.
# Add additional group actions over time (migrate, replicate, retype, clone, …). At this point, new group capabilities become vertical slices that are simple to add incrementally. 

Because we implemented CGs just recently as an experimental feature, we have the freedom to replace that code without deprecation or upgrade considerations. Steps 1-8 would get Manila to parity with the CG feature added in Liberty, would require only a few person-weeks of effort, and would better position Manila for long-term evolution and supportability.
=== Implementation details ===
There are few things to note, several of which were already solved during the CG work. 

Groups are first-class objects in Manila, and operations on groups are treated as atomic. For example, a user will see a group snapshot but not the individual snapshots of group members. To enable support by as many backends as possible, Manila will still maintain DB objects for both group and member snapshots, just as was done with CGs. 

A few actions, such as extend & shrink, are inherently applicable only to individual shares. One could theoretically apply extend to a group, increasing the size of each member, but that seems like an unlikely use case. Any actions in this category must remain available to group members, although most actions such as clone or snapshot would be available only at the group level and not on its members. 

A group is limited to a single backend. Allowing groups that span backends is theoretically possible, but that would require fanout of operations from the API layer to multiple share managers across the asynchronous event bus, which would lead to complicated synchronization and state management for little operational benefit. 

As was done with Manila CGs, a driver may optionally limit a group to either the confines of a pool or an entire backend. It is known that pools are the unit of data motion (i.e. replication or migration) for some backends, so we think drivers need this flexibility. 

We considered reusing share types for groups as well. But share types include a set of public extra specs that may not map well to groups. And by adding group types as a separate object, there can be little confusion about their purpose and use. 

Some have noted that extra specs tend to be treated as an AND operation, where all features must be available for a backend to be chosen. It is conceivable that even if a backend can replicate a group or take consistent snapshots of a group, it might not be able to perform both operations on the same group. But this problem already exists with share types and hasn’t been a serious issue. Creating types is an admin-only operation and the burden remains on the admin to understand the capabilities of the backends in use and to create the share types and share group types appropriately. 

Note that this proposal explicitly does not address pool or backend replication, which is fundamentally different. Actions on shares or share groups are intended for tenants, whereas a pool or backend can contain data from multiple tenants. So pool or backend operations, while serving potentially valuable use cases, are inherently admin-only workflows that would be designed and exercised differently should Manila support them.
== Acknowledgments ==
Most of the analysis and recommendations contained herein apply equally well to Cinder, which is arguably much further down the path of crippling complexity. 

This proposal was developed by cknight with input from Manila community members bswartz, ameade, tbarron, and cfouts.

Manila/design/manila-generic-groups

2015-12-16T20:47:19Z

Clinton Knight:

= Manila generic groups =
== Executive summary ==
Manila needs a grouping construct that, like shares, is a 1st-class atomic data type. Our experience with CGs has demonstrated the complexity of adding a grouping capability, yet there are other use cases such as migration, replication, and backup in which some storage controllers could only offer such features on share groups. CGs also highlighted the poor optics of an advanced feature with comparatively little potential for vendor support. And adding new grouping constructs for each new feature is not technically feasible. All of the above may be addressed by generic groups, which we think is a clean extension to the original architecture of Manila.
== An Intractable Matrix ==
Manila is at an exciting time in its development, where all the simple features are available and the community is focused on adding high-value features such as migration, replication, and consistency groups. Experimental code is available for each of these, but little consideration has been given to the obvious need for these complex features to interact not only with each other but also with whatever additional features were added later. And looking a little deeper, it is apparent that each of these has limitations that could be solved by a single architectural enhancement.
=== Consistency groups ===
Consistency groups (CGs) are the only construct in Manila that operates on a group of shares. For some, a CG implies a guarantee of ordered writes by a storage controller, while for others a CG is a mechanism for taking consistent point-in-time snapshots of a set of shares. A group with either attribute may have value by itself, but the CGs implementation doesn’t distinguish between them. 

CGs are a highly specialized construct with dedicated CLI commands, REST APIs, database tables, scheduler filters, and driver APIs. None of these are reusable for anything else, and presently none of the rest of Manila has any awareness of CGs. Even worse, despite all the complexity of the CG feature, only a small minority of storage backends can support them, so the code-to-value ratio is very low. Finally, the limited availability of CGs ensures the user experience is inconsistent between clouds.
=== Replication ===
Replication is another high value feature, and the Manila implementation is arguably clean and flexible, but as constituted there is no core ability to replicate groups of shares. It seems reasonable to implement the feature iteratively, beginning with replication of single shares. However, there has been the tacit acknowledgement that some backends would not be able to replicate individual shares, even though those same backends could potentially replicate a group of shares if the group were constituted in a certain way. 

Following the precedent established with CGs, one approach would be to introduce ‘replication groups’ whereby multiple shares could be replicated together. But this would introduce yet another set of APIs, tables, etc. that are too specialized to use for anything else. 

Another approach might be to define CGs as the grouping unit of replication, but this makes little sense because consistency groups (however we define them) and replication are distinct features that shouldn’t be artificially bundled for the sake of expediency of definition or implementation.
=== Migration ===
Just as some backends may only be able to replicate shares in groups, it follows that those backends may also need to migrate shares in groups. And in the case of CGs, it doesn’t make sense to migrate CG members individually; the whole group must be moved, requiring the replication engine to be CG aware. 

Consider the following reasonable use cases:
* Replicate a consistency group
* Snapshot a replication group
* Migrate a replication group
* Retype shares in a consistency group
* Retype shares in a replication group
* Backup a consistency group
* Backup a replication group
* Apply a common policy to a set of shares
 
On the current path, the support matrix has a dimension dedicated solely to features, so each feature must be coded to be interoperable with every other feature. This quickly becomes a support and testing nightmare, and it becomes exponentially more complicated to add more features going forward. 
Fundamentally, the problem is that we are adding features without an underlying architectural framework on which to hang them. To escape the matrix, we must step back and rethink a few things.

== A simple solution ==
So to arrive at a solution, let’s enumerate what we have:
* Primitive objects (shares) with supported operations controlled by share types that vary by backend
* A very specific group object (CGs) with limited support potential by backends
* A number of APIs (CGs, etc.) that many users can’t use at all and appear grafted into the project
 
And what would we prefer:
* A uniform API and user experience that varies as little as possible by backend
* A clean way to group shares so we can do CGs, multi-object replication, migration, etc.
 
How do we get here?
* Universally available groups of primitive objects with supported operations that vary by backend
 
=== Idea #1: Introduce a generic Share Group object ===
Just as Manila has ‘shares’, it should also have ‘share groups’. In its simplest form, unlike CGs, a share group should not guarantee any specialized operation. Instead, it should merely constitute an atomic Manila data type on which nearly any Manila action is available. For example, given a share group, the user should be able to select the group in the Manila UI and invoke features such as snapshot, clone, backup, migrate, replicate, retype, etc.
 
In the general case, group actions are handled by the share manager layer (or potentially the common share driver superclass). For example, invoking ‘snapshot’ on a group causes the share manager to take a snapshot of each group member individually. The resulting group snapshot object may be used to create a new group, not unlike how CG snapshots were implemented.
 
There are numerous advantages to this approach. Every driver, no matter how unsophisticated, can leverage the manageability goodness of groups. The user experience is uniform, since groups are always available regardless of which backends are present and because nearly every Manila action available on shares is also available on groups.
 
Of course, feature-rich backends would like to add their secret sauce, whether it be CGs, group-based replication, or whatever else comes along that might be easier/cheaper/faster to do to groups of shares. That brings us to related idea.
=== Idea #2: Introduce Share Group types ===
Just as Manila has ‘share types’, it should also have ‘share group types’. Any driver that can perform a group operation in an advantaged way may report that as a group capability, such as:
* Ordered writes
* Consistent snapshots
* Group replication
* Group migration
* Group backup
* Group modification
 
As with share types, the cloud administrator predefines share group types that may contain extra specs corresponding to the group capabilities reported by the backends. And when creating a group, the user specifies one or more share types as well as a group type, and the scheduler then creates the group on one of the backends that match the specified share and share group types.
 
Anytime a group action, such as ‘snapshot’, comes into the share manager, the manager checks whether its driver offers an advantaged implementation of that operation. If not, the manager handles the workflow itself as described above. But if so, the manager routes the workflow to its driver for fulfillment.
 
The advantages of this approach should be obvious. Manila can add a single set of group APIs that can service all current and future group operations, including CG snapshots, replication, migration, retype, backup, etc. Development and testing are simplified because there isn’t a need to define and test a different set of group APIs for each feature, or to test every combination of every feature. Users and admins are already familiar with share types, so introducing share group types would seem a natural and consistent evolution of the same foundational concept. Instead of becoming an N-by-N matrix of interacting features, Manila largely becomes an N-by-2 matrix of actions that may be invoked on either individual shares or share groups.
== Manila implementation plan ==
Implementing generic groups in Manila should be a straightforward series of steps:
# Implement generic groups. Because we did CGs, we already know all parts of the codebase that must change to support any kind of group. So the simplest approach is to modify the CG code to morph it into the generic groups feature.
# Add the share group type feature by duplicating and customizing the share type code.
# Enhance the scheduler to place groups according to share group types. Like #1, this is already informed by the CG project.
# Implement group snapshots in the share manager to demonstrate group snapshots in any driver.
# Plumb group snapshots to a CG-capable driver to demonstrate CG functionality in the new framework.
# Update Tempest to cover all of the above.
# Update Manila client to change CGs to share groups.
# Enhance Manila client with share group types.
# Add share group support to Horizon. We never built CG support into Horizon, so this is all-new work that now has much broader appeal and applicability.
# Add additional group actions over time (migrate, replicate, retype, clone, …). At this point, new group capabilities become vertical slices that are simple to add incrementally.
 
Because we implemented CGs just recently as an experimental feature, we have the freedom to replace that code without deprecation or upgrade considerations. Steps 1-8 would get Manila to parity with the CG feature added in Liberty, would require only a few person-weeks of effort, and would better position Manila for long-term evolution and supportability.
=== Implementation details ===
There are few things to note, several of which were already solved during the CG work.
 
Groups are first-class objects in Manila, and operations on groups are treated as atomic. For example, a user will see a group snapshot but not the individual snapshots of group members. To enable support by as many backends as possible, Manila will still maintain DB objects for both group and member snapshots, just as was done with CGs.
 
A few actions, such as extend & shrink, are inherently applicable only to individual shares. One could theoretically apply extend to a group, increasing the size of each member, but that seems like an unlikely use case. Any actions in this category must remain available to group members, although most actions such as clone or snapshot would be available only at the group level and not on its members.
 
A group is limited to a single backend. Allowing groups that span backends is theoretically possible, but that would require fanout of operations from the API layer to multiple share managers across the asynchronous event bus, which would lead to complicated synchronization and state management for little operational benefit.
 
As was done with Manila CGs, a driver may optionally limit a group to either the confines of a pool or an entire backend. It is known that pools are the unit of data motion (i.e. replication or migration) for some backends, so we think drivers need this flexibility.
 
We considered reusing share types for groups as well. But share types include a set of public extra specs that may not map well to groups. And by adding group types as a separate object, there can be little confusion about their purpose and use.
 
Some have noted that extra specs tend to be treated as an AND operation, where all features must be available for a backend to be chosen. It is conceivable that even if a backend can replicate a group or take consistent snapshots of a group, it might not be able to perform both operations on the same group. But this problem already exists with share types and hasn’t been a serious issue. Creating types is an admin-only operation and the burden remains on the admin to understand the capabilities of the backends in use and to create the share types and share group types appropriately.
 
Note that this proposal explicitly does not address pool or backend replication, which is fundamentally different. Actions on shares or share groups are intended for tenants, whereas a pool or backend can contain data from multiple tenants. So pool or backend operations, while serving potentially valuable use cases, are inherently admin-only workflows that would be designed and exercised differently should Manila support them.
== Acknowledgments ==
Most of the analysis and recommendations contained herein apply equally well to Cinder, which is arguably much further down the path of crippling complexity.
 
This proposal was developed by cknight with input from Manila community members bswartz, ameade, tbarron, and cfouts.

Manila/design/manila-generic-groups

2015-12-16T20:46:34Z

Clinton Knight:

= Manila generic groups =
== Executive summary ==
Manila needs a grouping construct that, like shares, is a 1st-class atomic data type. Our experience with CGs has demonstrated the complexity of adding a grouping capability, yet there are other use cases such as migration, replication, and backup in which some storage controllers could only offer such features on share groups. CGs also highlighted the poor optics of an advanced feature with comparatively little potential for vendor support. And adding new grouping constructs for each new feature is not technically feasible. All of the above may be addressed by generic groups, which we think is a clean extension to the original architecture of Manila.
== An Intractable Matrix ==
Manila is at an exciting time in its development, where all the simple features are available and the community is focused on adding high-value features such as migration, replication, and consistency groups. Experimental code is available for each of these, but little consideration has been given to the obvious need for these complex features to interact not only with each other but also with whatever additional features were added later. And looking a little deeper, it is apparent that each of these has limitations that could be solved by a single architectural enhancement.
=== Consistency groups ===
Consistency groups (CGs) are the only construct in Manila that operates on a group of shares. For some, a CG implies a guarantee of ordered writes by a storage controller, while for others a CG is a mechanism for taking consistent point-in-time snapshots of a set of shares. A group with either attribute may have value by itself, but the CGs implementation doesn’t distinguish between them. 

CGs are a highly specialized construct with dedicated CLI commands, REST APIs, database tables, scheduler filters, and driver APIs. None of these are reusable for anything else, and presently none of the rest of Manila has any awareness of CGs. Even worse, despite all the complexity of the CG feature, only a small minority of storage backends can support them, so the code-to-value ratio is very low. Finally, the limited availability of CGs ensures the user experience is inconsistent between clouds.
=== Replication ===
Replication is another high value feature, and the Manila implementation is arguably clean and flexible, but as constituted there is no core ability to replicate groups of shares. It seems reasonable to implement the feature iteratively, beginning with replication of single shares. However, there has been the tacit acknowledgement that some backends would not be able to replicate individual shares, even though those same backends could potentially replicate a group of shares if the group were constituted in a certain way.
 
Following the precedent established with CGs, one approach would be to introduce ‘replication groups’ whereby multiple shares could be replicated together. But this would introduce yet another set of APIs, tables, etc. that are too specialized to use for anything else.
 
Another approach might be to define CGs as the grouping unit of replication, but this makes little sense because consistency groups (however we define them) and replication are distinct features that shouldn’t be artificially bundled for the sake of expediency of definition or implementation.
=== Migration ===
Just as some backends may only be able to replicate shares in groups, it follows that those backends may also need to migrate shares in groups. And in the case of CGs, it doesn’t make sense to migrate CG members individually; the whole group must be moved, requiring the replication engine to be CG aware.
 
Consider the following reasonable use cases:
* Replicate a consistency group
* Snapshot a replication group
* Migrate a replication group
* Retype shares in a consistency group
* Retype shares in a replication group
* Backup a consistency group
* Backup a replication group
* Apply a common policy to a set of shares
 
On the current path, the support matrix has a dimension dedicated solely to features, so each feature must be coded to be interoperable with every other feature. This quickly becomes a support and testing nightmare, and it becomes exponentially more complicated to add more features going forward. Fundamentally, the problem is that we are adding features without an underlying architectural framework on which to hang them. To escape the matrix, we must step back and rethink a few things.

== A simple solution ==
So to arrive at a solution, let’s enumerate what we have:
* Primitive objects (shares) with supported operations controlled by share types that vary by backend
* A very specific group object (CGs) with limited support potential by backends
* A number of APIs (CGs, etc.) that many users can’t use at all and appear grafted into the project
 
And what would we prefer:
* A uniform API and user experience that varies as little as possible by backend
* A clean way to group shares so we can do CGs, multi-object replication, migration, etc.
 
How do we get here?
* Universally available groups of primitive objects with supported operations that vary by backend
 
=== Idea #1: Introduce a generic Share Group object ===
Just as Manila has ‘shares’, it should also have ‘share groups’. In its simplest form, unlike CGs, a share group should not guarantee any specialized operation. Instead, it should merely constitute an atomic Manila data type on which nearly any Manila action is available. For example, given a share group, the user should be able to select the group in the Manila UI and invoke features such as snapshot, clone, backup, migrate, replicate, retype, etc.
 
In the general case, group actions are handled by the share manager layer (or potentially the common share driver superclass). For example, invoking ‘snapshot’ on a group causes the share manager to take a snapshot of each group member individually. The resulting group snapshot object may be used to create a new group, not unlike how CG snapshots were implemented.
 
There are numerous advantages to this approach. Every driver, no matter how unsophisticated, can leverage the manageability goodness of groups. The user experience is uniform, since groups are always available regardless of which backends are present and because nearly every Manila action available on shares is also available on groups.
 
Of course, feature-rich backends would like to add their secret sauce, whether it be CGs, group-based replication, or whatever else comes along that might be easier/cheaper/faster to do to groups of shares. That brings us to related idea.
=== Idea #2: Introduce Share Group types ===
Just as Manila has ‘share types’, it should also have ‘share group types’. Any driver that can perform a group operation in an advantaged way may report that as a group capability, such as:
* Ordered writes
* Consistent snapshots
* Group replication
* Group migration
* Group backup
* Group modification
 
As with share types, the cloud administrator predefines share group types that may contain extra specs corresponding to the group capabilities reported by the backends. And when creating a group, the user specifies one or more share types as well as a group type, and the scheduler then creates the group on one of the backends that match the specified share and share group types.
 
Anytime a group action, such as ‘snapshot’, comes into the share manager, the manager checks whether its driver offers an advantaged implementation of that operation. If not, the manager handles the workflow itself as described above. But if so, the manager routes the workflow to its driver for fulfillment.
 
The advantages of this approach should be obvious. Manila can add a single set of group APIs that can service all current and future group operations, including CG snapshots, replication, migration, retype, backup, etc. Development and testing are simplified because there isn’t a need to define and test a different set of group APIs for each feature, or to test every combination of every feature. Users and admins are already familiar with share types, so introducing share group types would seem a natural and consistent evolution of the same foundational concept. Instead of becoming an N-by-N matrix of interacting features, Manila largely becomes an N-by-2 matrix of actions that may be invoked on either individual shares or share groups.
== Manila implementation plan ==
Implementing generic groups in Manila should be a straightforward series of steps:
# Implement generic groups. Because we did CGs, we already know all parts of the codebase that must change to support any kind of group. So the simplest approach is to modify the CG code to morph it into the generic groups feature.
# Add the share group type feature by duplicating and customizing the share type code.
# Enhance the scheduler to place groups according to share group types. Like #1, this is already informed by the CG project.
# Implement group snapshots in the share manager to demonstrate group snapshots in any driver.
# Plumb group snapshots to a CG-capable driver to demonstrate CG functionality in the new framework.
# Update Tempest to cover all of the above.
# Update Manila client to change CGs to share groups.
# Enhance Manila client with share group types.
# Add share group support to Horizon. We never built CG support into Horizon, so this is all-new work that now has much broader appeal and applicability.
# Add additional group actions over time (migrate, replicate, retype, clone, …). At this point, new group capabilities become vertical slices that are simple to add incrementally.
 
Because we implemented CGs just recently as an experimental feature, we have the freedom to replace that code without deprecation or upgrade considerations. Steps 1-8 would get Manila to parity with the CG feature added in Liberty, would require only a few person-weeks of effort, and would better position Manila for long-term evolution and supportability.
=== Implementation details ===
There are few things to note, several of which were already solved during the CG work.
 
Groups are first-class objects in Manila, and operations on groups are treated as atomic. For example, a user will see a group snapshot but not the individual snapshots of group members. To enable support by as many backends as possible, Manila will still maintain DB objects for both group and member snapshots, just as was done with CGs.
 
A few actions, such as extend & shrink, are inherently applicable only to individual shares. One could theoretically apply extend to a group, increasing the size of each member, but that seems like an unlikely use case. Any actions in this category must remain available to group members, although most actions such as clone or snapshot would be available only at the group level and not on its members.
 
A group is limited to a single backend. Allowing groups that span backends is theoretically possible, but that would require fanout of operations from the API layer to multiple share managers across the asynchronous event bus, which would lead to complicated synchronization and state management for little operational benefit.
 
As was done with Manila CGs, a driver may optionally limit a group to either the confines of a pool or an entire backend. It is known that pools are the unit of data motion (i.e. replication or migration) for some backends, so we think drivers need this flexibility.
 
We considered reusing share types for groups as well. But share types include a set of public extra specs that may not map well to groups. And by adding group types as a separate object, there can be little confusion about their purpose and use.
 
Some have noted that extra specs tend to be treated as an AND operation, where all features must be available for a backend to be chosen. It is conceivable that even if a backend can replicate a group or take consistent snapshots of a group, it might not be able to perform both operations on the same group. But this problem already exists with share types and hasn’t been a serious issue. Creating types is an admin-only operation and the burden remains on the admin to understand the capabilities of the backends in use and to create the share types and share group types appropriately.
 
Note that this proposal explicitly does not address pool or backend replication, which is fundamentally different. Actions on shares or share groups are intended for tenants, whereas a pool or backend can contain data from multiple tenants. So pool or backend operations, while serving potentially valuable use cases, are inherently admin-only workflows that would be designed and exercised differently should Manila support them.
== Acknowledgments ==
Most of the analysis and recommendations contained herein apply equally well to Cinder, which is arguably much further down the path of crippling complexity.
 
This proposal was developed by cknight with input from Manila community members bswartz, ameade, tbarron, and cfouts.

Manila/design/manila-generic-groups

2015-12-16T20:45:52Z

Clinton Knight: /* Migration */

= Manila generic groups =
== Executive summary ==
Manila needs a grouping construct that, like shares, is a 1st-class atomic data type. Our experience with CGs has demonstrated the complexity of adding a grouping capability, yet there are other use cases such as migration, replication, and backup in which some storage controllers could only offer such features on share groups. CGs also highlighted the poor optics of an advanced feature with comparatively little potential for vendor support. And adding new grouping constructs for each new feature is not technically feasible. All of the above may be addressed by generic groups, which we think is a clean extension to the original architecture of Manila.
== An Intractable Matrix ==
Manila is at an exciting time in its development, where all the simple features are available and the community is focused on adding high-value features such as migration, replication, and consistency groups. Experimental code is available for each of these, but little consideration has been given to the obvious need for these complex features to interact not only with each other but also with whatever additional features were added later. And looking a little deeper, it is apparent that each of these has limitations that could be solved by a single architectural enhancement.
=== Consistency groups ===
Consistency groups (CGs) are the only construct in Manila that operates on a group of shares. For some, a CG implies a guarantee of ordered writes by a storage controller, while for others a CG is a mechanism for taking consistent point-in-time snapshots of a set of shares. A group with either attribute may have value by itself, but the CGs implementation doesn’t distinguish between them.
 
CGs are a highly specialized construct with dedicated CLI commands, REST APIs, database tables, scheduler filters, and driver APIs. None of these are reusable for anything else, and presently none of the rest of Manila has any awareness of CGs. Even worse, despite all the complexity of the CG feature, only a small minority of storage backends can support them, so the code-to-value ratio is very low. Finally, the limited availability of CGs ensures the user experience is inconsistent between clouds.
=== Replication ===
Replication is another high value feature, and the Manila implementation is arguably clean and flexible, but as constituted there is no core ability to replicate groups of shares. It seems reasonable to implement the feature iteratively, beginning with replication of single shares. However, there has been the tacit acknowledgement that some backends would not be able to replicate individual shares, even though those same backends could potentially replicate a group of shares if the group were constituted in a certain way.
 
Following the precedent established with CGs, one approach would be to introduce ‘replication groups’ whereby multiple shares could be replicated together. But this would introduce yet another set of APIs, tables, etc. that are too specialized to use for anything else.
 
Another approach might be to define CGs as the grouping unit of replication, but this makes little sense because consistency groups (however we define them) and replication are distinct features that shouldn’t be artificially bundled for the sake of expediency of definition or implementation.
=== Migration ===
Just as some backends may only be able to replicate shares in groups, it follows that those backends may also need to migrate shares in groups. And in the case of CGs, it doesn’t make sense to migrate CG members individually; the whole group must be moved, requiring the replication engine to be CG aware.
 
Consider the following reasonable use cases:
* Replicate a consistency group
* Snapshot a replication group
* Migrate a replication group
* Retype shares in a consistency group
* Retype shares in a replication group
* Backup a consistency group
* Backup a replication group
* Apply a common policy to a set of shares
 
On the current path, the support matrix has a dimension dedicated solely to features, so each feature must be coded to be interoperable with every other feature. This quickly becomes a support and testing nightmare, and it becomes exponentially more complicated to add more features going forward. Fundamentally, the problem is that we are adding features without an underlying architectural framework on which to hang them. To escape the matrix, we must step back and rethink a few things.

== A simple solution ==
So to arrive at a solution, let’s enumerate what we have:
* Primitive objects (shares) with supported operations controlled by share types that vary by backend
* A very specific group object (CGs) with limited support potential by backends
* A number of APIs (CGs, etc.) that many users can’t use at all and appear grafted into the project
 
And what would we prefer:
* A uniform API and user experience that varies as little as possible by backend
* A clean way to group shares so we can do CGs, multi-object replication, migration, etc.
 
How do we get here?
* Universally available groups of primitive objects with supported operations that vary by backend
 
=== Idea #1: Introduce a generic Share Group object ===
Just as Manila has ‘shares’, it should also have ‘share groups’. In its simplest form, unlike CGs, a share group should not guarantee any specialized operation. Instead, it should merely constitute an atomic Manila data type on which nearly any Manila action is available. For example, given a share group, the user should be able to select the group in the Manila UI and invoke features such as snapshot, clone, backup, migrate, replicate, retype, etc.
 
In the general case, group actions are handled by the share manager layer (or potentially the common share driver superclass). For example, invoking ‘snapshot’ on a group causes the share manager to take a snapshot of each group member individually. The resulting group snapshot object may be used to create a new group, not unlike how CG snapshots were implemented.
 
There are numerous advantages to this approach. Every driver, no matter how unsophisticated, can leverage the manageability goodness of groups. The user experience is uniform, since groups are always available regardless of which backends are present and because nearly every Manila action available on shares is also available on groups.
 
Of course, feature-rich backends would like to add their secret sauce, whether it be CGs, group-based replication, or whatever else comes along that might be easier/cheaper/faster to do to groups of shares. That brings us to related idea.
=== Idea #2: Introduce Share Group types ===
Just as Manila has ‘share types’, it should also have ‘share group types’. Any driver that can perform a group operation in an advantaged way may report that as a group capability, such as:
* Ordered writes
* Consistent snapshots
* Group replication
* Group migration
* Group backup
* Group modification
 
As with share types, the cloud administrator predefines share group types that may contain extra specs corresponding to the group capabilities reported by the backends. And when creating a group, the user specifies one or more share types as well as a group type, and the scheduler then creates the group on one of the backends that match the specified share and share group types.
 
Anytime a group action, such as ‘snapshot’, comes into the share manager, the manager checks whether its driver offers an advantaged implementation of that operation. If not, the manager handles the workflow itself as described above. But if so, the manager routes the workflow to its driver for fulfillment.
 
The advantages of this approach should be obvious. Manila can add a single set of group APIs that can service all current and future group operations, including CG snapshots, replication, migration, retype, backup, etc. Development and testing are simplified because there isn’t a need to define and test a different set of group APIs for each feature, or to test every combination of every feature. Users and admins are already familiar with share types, so introducing share group types would seem a natural and consistent evolution of the same foundational concept. Instead of becoming an N-by-N matrix of interacting features, Manila largely becomes an N-by-2 matrix of actions that may be invoked on either individual shares or share groups.
== Manila implementation plan ==
Implementing generic groups in Manila should be a straightforward series of steps:
# Implement generic groups. Because we did CGs, we already know all parts of the codebase that must change to support any kind of group. So the simplest approach is to modify the CG code to morph it into the generic groups feature.
# Add the share group type feature by duplicating and customizing the share type code.
# Enhance the scheduler to place groups according to share group types. Like #1, this is already informed by the CG project.
# Implement group snapshots in the share manager to demonstrate group snapshots in any driver.
# Plumb group snapshots to a CG-capable driver to demonstrate CG functionality in the new framework.
# Update Tempest to cover all of the above.
# Update Manila client to change CGs to share groups.
# Enhance Manila client with share group types.
# Add share group support to Horizon. We never built CG support into Horizon, so this is all-new work that now has much broader appeal and applicability.
# Add additional group actions over time (migrate, replicate, retype, clone, …). At this point, new group capabilities become vertical slices that are simple to add incrementally.
 
Because we implemented CGs just recently as an experimental feature, we have the freedom to replace that code without deprecation or upgrade considerations. Steps 1-8 would get Manila to parity with the CG feature added in Liberty, would require only a few person-weeks of effort, and would better position Manila for long-term evolution and supportability.
=== Implementation details ===
There are few things to note, several of which were already solved during the CG work.
 
Groups are first-class objects in Manila, and operations on groups are treated as atomic. For example, a user will see a group snapshot but not the individual snapshots of group members. To enable support by as many backends as possible, Manila will still maintain DB objects for both group and member snapshots, just as was done with CGs.
 
A few actions, such as extend & shrink, are inherently applicable only to individual shares. One could theoretically apply extend to a group, increasing the size of each member, but that seems like an unlikely use case. Any actions in this category must remain available to group members, although most actions such as clone or snapshot would be available only at the group level and not on its members.
 
A group is limited to a single backend. Allowing groups that span backends is theoretically possible, but that would require fanout of operations from the API layer to multiple share managers across the asynchronous event bus, which would lead to complicated synchronization and state management for little operational benefit.
 
As was done with Manila CGs, a driver may optionally limit a group to either the confines of a pool or an entire backend. It is known that pools are the unit of data motion (i.e. replication or migration) for some backends, so we think drivers need this flexibility.
 
We considered reusing share types for groups as well. But share types include a set of public extra specs that may not map well to groups. And by adding group types as a separate object, there can be little confusion about their purpose and use.
 
Some have noted that extra specs tend to be treated as an AND operation, where all features must be available for a backend to be chosen. It is conceivable that even if a backend can replicate a group or take consistent snapshots of a group, it might not be able to perform both operations on the same group. But this problem already exists with share types and hasn’t been a serious issue. Creating types is an admin-only operation and the burden remains on the admin to understand the capabilities of the backends in use and to create the share types and share group types appropriately.
 
Note that this proposal explicitly does not address pool or backend replication, which is fundamentally different. Actions on shares or share groups are intended for tenants, whereas a pool or backend can contain data from multiple tenants. So pool or backend operations, while serving potentially valuable use cases, are inherently admin-only workflows that would be designed and exercised differently should Manila support them.
== Acknowledgments ==
Most of the analysis and recommendations contained herein apply equally well to Cinder, which is arguably much further down the path of crippling complexity.
 
This proposal was developed by cknight with input from Manila community members bswartz, ameade, tbarron, and cfouts.

Manila/design/manila-generic-groups

2015-12-16T20:45:19Z

Clinton Knight: Created page with "= Manila generic groups = == Executive summary == Manila needs a grouping construct that, like shares, is a 1st-class atomic data type. Our experience with CGs has demonstrat..."

= Manila generic groups =
== Executive summary ==
Manila needs a grouping construct that, like shares, is a 1st-class atomic data type. Our experience with CGs has demonstrated the complexity of adding a grouping capability, yet there are other use cases such as migration, replication, and backup in which some storage controllers could only offer such features on share groups. CGs also highlighted the poor optics of an advanced feature with comparatively little potential for vendor support. And adding new grouping constructs for each new feature is not technically feasible. All of the above may be addressed by generic groups, which we think is a clean extension to the original architecture of Manila.
== An Intractable Matrix ==
Manila is at an exciting time in its development, where all the simple features are available and the community is focused on adding high-value features such as migration, replication, and consistency groups. Experimental code is available for each of these, but little consideration has been given to the obvious need for these complex features to interact not only with each other but also with whatever additional features were added later. And looking a little deeper, it is apparent that each of these has limitations that could be solved by a single architectural enhancement.
=== Consistency groups ===
Consistency groups (CGs) are the only construct in Manila that operates on a group of shares. For some, a CG implies a guarantee of ordered writes by a storage controller, while for others a CG is a mechanism for taking consistent point-in-time snapshots of a set of shares. A group with either attribute may have value by itself, but the CGs implementation doesn’t distinguish between them.
 
CGs are a highly specialized construct with dedicated CLI commands, REST APIs, database tables, scheduler filters, and driver APIs. None of these are reusable for anything else, and presently none of the rest of Manila has any awareness of CGs. Even worse, despite all the complexity of the CG feature, only a small minority of storage backends can support them, so the code-to-value ratio is very low. Finally, the limited availability of CGs ensures the user experience is inconsistent between clouds.
=== Replication ===
Replication is another high value feature, and the Manila implementation is arguably clean and flexible, but as constituted there is no core ability to replicate groups of shares. It seems reasonable to implement the feature iteratively, beginning with replication of single shares. However, there has been the tacit acknowledgement that some backends would not be able to replicate individual shares, even though those same backends could potentially replicate a group of shares if the group were constituted in a certain way.
 
Following the precedent established with CGs, one approach would be to introduce ‘replication groups’ whereby multiple shares could be replicated together. But this would introduce yet another set of APIs, tables, etc. that are too specialized to use for anything else.
 
Another approach might be to define CGs as the grouping unit of replication, but this makes little sense because consistency groups (however we define them) and replication are distinct features that shouldn’t be artificially bundled for the sake of expediency of definition or implementation.
=== Migration ===
Just as some backends may only be able to replicate shares in groups, it follows that those backends may also need to migrate shares in groups. And in the case of CGs, it doesn’t make sense to migrate CG members individually; the whole group must be moved, requiring the replication engine to be CG aware.
 
Consider the following reasonable use cases:
* Replicate a consistency group
* Snapshot a replication group
* Migrate a replication group
* Retype shares in a consistency group
* Retype shares in a replication group
* Backup a consistency group
* Backup a replication group
* Apply a common policy to a set of shares
 
On the current path, the support matrix has a dimension dedicated solely to features, so each feature must be coded to be interoperable with every other feature. This quickly becomes a support and testing nightmare, and it becomes exponentially more complicated to add more features going forward.
 
Fundamentally, the problem is that we are adding features without an underlying architectural framework on which to hang them. To escape the matrix, we must step back and rethink a few things.
== A simple solution ==
So to arrive at a solution, let’s enumerate what we have:
* Primitive objects (shares) with supported operations controlled by share types that vary by backend
* A very specific group object (CGs) with limited support potential by backends
* A number of APIs (CGs, etc.) that many users can’t use at all and appear grafted into the project
 
And what would we prefer:
* A uniform API and user experience that varies as little as possible by backend
* A clean way to group shares so we can do CGs, multi-object replication, migration, etc.
 
How do we get here?
* Universally available groups of primitive objects with supported operations that vary by backend
 
=== Idea #1: Introduce a generic Share Group object ===
Just as Manila has ‘shares’, it should also have ‘share groups’. In its simplest form, unlike CGs, a share group should not guarantee any specialized operation. Instead, it should merely constitute an atomic Manila data type on which nearly any Manila action is available. For example, given a share group, the user should be able to select the group in the Manila UI and invoke features such as snapshot, clone, backup, migrate, replicate, retype, etc.
 
In the general case, group actions are handled by the share manager layer (or potentially the common share driver superclass). For example, invoking ‘snapshot’ on a group causes the share manager to take a snapshot of each group member individually. The resulting group snapshot object may be used to create a new group, not unlike how CG snapshots were implemented.
 
There are numerous advantages to this approach. Every driver, no matter how unsophisticated, can leverage the manageability goodness of groups. The user experience is uniform, since groups are always available regardless of which backends are present and because nearly every Manila action available on shares is also available on groups.
 
Of course, feature-rich backends would like to add their secret sauce, whether it be CGs, group-based replication, or whatever else comes along that might be easier/cheaper/faster to do to groups of shares. That brings us to related idea.
=== Idea #2: Introduce Share Group types ===
Just as Manila has ‘share types’, it should also have ‘share group types’. Any driver that can perform a group operation in an advantaged way may report that as a group capability, such as:
* Ordered writes
* Consistent snapshots
* Group replication
* Group migration
* Group backup
* Group modification
 
As with share types, the cloud administrator predefines share group types that may contain extra specs corresponding to the group capabilities reported by the backends. And when creating a group, the user specifies one or more share types as well as a group type, and the scheduler then creates the group on one of the backends that match the specified share and share group types.
 
Anytime a group action, such as ‘snapshot’, comes into the share manager, the manager checks whether its driver offers an advantaged implementation of that operation. If not, the manager handles the workflow itself as described above. But if so, the manager routes the workflow to its driver for fulfillment.
 
The advantages of this approach should be obvious. Manila can add a single set of group APIs that can service all current and future group operations, including CG snapshots, replication, migration, retype, backup, etc. Development and testing are simplified because there isn’t a need to define and test a different set of group APIs for each feature, or to test every combination of every feature. Users and admins are already familiar with share types, so introducing share group types would seem a natural and consistent evolution of the same foundational concept. Instead of becoming an N-by-N matrix of interacting features, Manila largely becomes an N-by-2 matrix of actions that may be invoked on either individual shares or share groups.
== Manila implementation plan ==
Implementing generic groups in Manila should be a straightforward series of steps:
# Implement generic groups. Because we did CGs, we already know all parts of the codebase that must change to support any kind of group. So the simplest approach is to modify the CG code to morph it into the generic groups feature.
# Add the share group type feature by duplicating and customizing the share type code.
# Enhance the scheduler to place groups according to share group types. Like #1, this is already informed by the CG project.
# Implement group snapshots in the share manager to demonstrate group snapshots in any driver.
# Plumb group snapshots to a CG-capable driver to demonstrate CG functionality in the new framework.
# Update Tempest to cover all of the above.
# Update Manila client to change CGs to share groups.
# Enhance Manila client with share group types.
# Add share group support to Horizon. We never built CG support into Horizon, so this is all-new work that now has much broader appeal and applicability.
# Add additional group actions over time (migrate, replicate, retype, clone, …). At this point, new group capabilities become vertical slices that are simple to add incrementally.
 
Because we implemented CGs just recently as an experimental feature, we have the freedom to replace that code without deprecation or upgrade considerations. Steps 1-8 would get Manila to parity with the CG feature added in Liberty, would require only a few person-weeks of effort, and would better position Manila for long-term evolution and supportability.
=== Implementation details ===
There are few things to note, several of which were already solved during the CG work.
 
Groups are first-class objects in Manila, and operations on groups are treated as atomic. For example, a user will see a group snapshot but not the individual snapshots of group members. To enable support by as many backends as possible, Manila will still maintain DB objects for both group and member snapshots, just as was done with CGs.
 
A few actions, such as extend & shrink, are inherently applicable only to individual shares. One could theoretically apply extend to a group, increasing the size of each member, but that seems like an unlikely use case. Any actions in this category must remain available to group members, although most actions such as clone or snapshot would be available only at the group level and not on its members.
 
A group is limited to a single backend. Allowing groups that span backends is theoretically possible, but that would require fanout of operations from the API layer to multiple share managers across the asynchronous event bus, which would lead to complicated synchronization and state management for little operational benefit.
 
As was done with Manila CGs, a driver may optionally limit a group to either the confines of a pool or an entire backend. It is known that pools are the unit of data motion (i.e. replication or migration) for some backends, so we think drivers need this flexibility.
 
We considered reusing share types for groups as well. But share types include a set of public extra specs that may not map well to groups. And by adding group types as a separate object, there can be little confusion about their purpose and use.
 
Some have noted that extra specs tend to be treated as an AND operation, where all features must be available for a backend to be chosen. It is conceivable that even if a backend can replicate a group or take consistent snapshots of a group, it might not be able to perform both operations on the same group. But this problem already exists with share types and hasn’t been a serious issue. Creating types is an admin-only operation and the burden remains on the admin to understand the capabilities of the backends in use and to create the share types and share group types appropriately.
 
Note that this proposal explicitly does not address pool or backend replication, which is fundamentally different. Actions on shares or share groups are intended for tenants, whereas a pool or backend can contain data from multiple tenants. So pool or backend operations, while serving potentially valuable use cases, are inherently admin-only workflows that would be designed and exercised differently should Manila support them.
== Acknowledgments ==
Most of the analysis and recommendations contained herein apply equally well to Cinder, which is arguably much further down the path of crippling complexity.
 
This proposal was developed by cknight with input from Manila community members bswartz, ameade, tbarron, and cfouts.

Manila/Meetings

2015-12-10T14:59:42Z

Clinton Knight: /* Next meeting */

Manila/Meetings

2015-12-03T04:58:23Z

Clinton Knight: /* Next meeting */

Manila/Meetings

2015-12-01T21:05:53Z

Clinton Knight: /* Next meeting */

Design Summit/Mitaka/Etherpads

2015-10-15T14:39:31Z

Clinton Knight: /* Manila */

[[Category:Summit]]
[[Category:Liberty]]
[[Category:Etherpad]]

<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
__TOC__
</div>
== Event intro/closure ==
* Tue 11:15: Design Summit 101 [https://etherpad.openstack.org/p/mitaka-design-summit-101]
* Fri 12:30: Design Summit feedback [https://etherpad.openstack.org/p/mitaka-design-summit-feedback]

== Cinder ==
* Wed 14.00: Why fight it, Cinder could/should be the next ViPR [https://etherpad.openstack.org/p/mitaka-cinder-direction]
* Wed 14.50: Availability zones in Cinder [https://etherpad.openstack.org/p/mitaka-cinder-az]
* Thur 9.00: Experimental APIs and Microversions [https://etherpad.openstack.org/p/mitaka-cinder-experimental-apis]
* Thur 9.50: Cinder Nova Interaction [https://etherpad.openstack.org/p/mitaka-cinder-nova-interaction]
* Thur 13.50: Cinder driver interface [https://etherpad.openstack.org/p/mitaka-cinder-driver-interface]
* Thur 14.40: API Microversions [https://etherpad.openstack.org/p/mitaka-cinder-api-microversions]
* Thur 15.30: ABC work [https://etherpad.openstack.org/p/mitaka-cinder-abc-work]
* Thur 15.30: Driver deadlines [https://etherpad.openstack.org/p/mitaka-cinder-driver-deadlines]
* Thur 16.30: C-Vol Active/Active HA [https://etherpad.openstack.org/p/mitaka-cinder-cvol-aa]
* Thur 17.20: Volume manager locks [https://etherpad.openstack.org/p/mitaka-cinder-volmgr-locks]
* Fri: Contributor Meetup [https://etherpad.openstack.org/p/mitaka-cinder-contributor-meetup]

== Congress ==
* Wed 2:00: [https://etherpad.openstack.org/p/congress-mitaka-arch Distributed architecture and additional features for Mitaka]
* Wed 2:50: [https://etherpad.openstack.org/p/congress-mitaka-integrations Integration with other projects: congress gating (murano, nova, neutron, etc.), keystone]
* Wed 3:40: [https://etherpad.openstack.org/p/congress-mitaka-external Discussions with external teams: OPNFV, Monasca]

== Cross-Project workshops ==

All sessions are on Tuesday 2015-10-27

* 11:15
** Cycle themes [https://etherpad.openstack.org/p/mitaka-crossproject-themes]
* 12:05
** Supporting DefCore and Interoperability Testing [https://etherpad.openstack.org/p/mitaka-crossproject-defcore]
** Tags today and tomorrow [https://etherpad.openstack.org/p/mitaka-crossproject-next-tags]
* 14:50
** Role Assignments for Service users [https://etherpad.openstack.org/p/mitaka-cross-project-role-assignment-service-user]
* 15:40
** Documenting the OpenStack way [https://etherpad.openstack.org/p/mitaka-crossproject-doc-the-way]
* 16:40
** Troubleshooting cross-project comms [https://etherpad.openstack.org/p/mitaka-crossproject-comms]
* 17:30
** Serving extreme use cases [https://etherpad.openstack.org/p/mitaka-crossproject-extreme-usecases]

== Ceilometer ==
* Wednesday, 2015-08-28
** 11:15 - [https://etherpad.openstack.org/p/mitaka-telemetry-alarms alams]
** 12:05 - [https://etherpad.openstack.org/p/mitaka-telemetry-ui visualising data]
** 14:50 - [https://etherpad.openstack.org/p/mitaka-telemetry-upgrades rolling upgrades]
** 15:40 - [https://etherpad.openstack.org/p/mitaka-telemetry-split componentisation]

* Thursday, 2015-08-29
** 09:00 - [https://etherpad.openstack.org/p/mitaka-telemetry-testing functional and integration testing]
** 09ː50 - [https://etherpad.openstack.org/p/mitaka-telemetry-bi business intelligence]
** 11ː00 - [https://etherpad.openstack.org/p/mitaka-telemetry-polling refined polling]
** 11ː50 - [https://etherpad.openstack.org/p/mitaka-telemetry-cross-project project data ownership]
** 13ː50 - [https://etherpad.openstack.org/p/mitaka-telemetry-alarms event alarms]

* Friday, 2015-08-30
** 09:00 - [https://etherpad.openstack.org/p/mitaka-telemetry-contributors-meetup contributors meetup]

== Designate ==

* Wed 11:15: Roadmap https://etherpad.openstack.org/p/mitaka-designate-summit-roadmap
* Wed 12:05: Alias Records https://etherpad.openstack.org/p/mitaka-designate-summit-alias
* Wed 14:00: Batch API Actions https://etherpad.openstack.org/p/mitaka-designate-summit-batch-api
* Wed 14:50: Embedable Services https://etherpad.openstack.org/p/mitaka-designate-summit-embeddable-services
* Wed 16:40: Incremental Zone Transfer (IFXR) https://etherpad.openstack.org/p/mitaka-designate-summit-ifxr
* Fri 14:00: Contributors Meetup https://etherpad.openstack.org/p/mitaka-designate-summit-meetup

== Nova ==

* Wed 11:15: REST API https://etherpad.openstack.org/p/mitaka-nova-api
* Wed 12:05: Upgrade https://etherpad.openstack.org/p/mitaka-nova-upgrade

* Wed 14:00: Unconference https://etherpad.openstack.org/p/mitaka-nova-unconference
* Wed 14:50: OS VIF lib https://etherpad.openstack.org/p/mitaka-nova-os-vif-lib
* Wed 15:40: Resources and Flavors https://etherpad.openstack.org/p/mitaka-nova-resource-modeling
* Wed 16:40: Resources and Flavors (continued) https://etherpad.openstack.org/p/mitaka-nova-resource-modeling
* Wed 17:30: SR-IOV https://etherpad.openstack.org/p/mitaka-nova-sr-iov

* Thurs 09:00: Cells v2 https://etherpad.openstack.org/p/mitaka-nova-cells
* Thurs 9:50: see Cinder track
* Thurs 11:00: Scheduler https://etherpad.openstack.org/p/mitaka-nova-scheduler
* Thurs 11:50: see Ironic track

* Thurs 13:50: Unconference https://etherpad.openstack.org/p/mitaka-nova-unconference
* Thurs 14:40: Error handling https://etherpad.openstack.org/p/mitaka-nova-error-handling
* Thurs 15:30: Cross Service issues: Server locking, token refresh, Instance users https://etherpad.openstack.org/p/mitaka-nova-service-users
* Thurs 16:30: Mitaka Priorities https://etherpad.openstack.org/p/mitaka-nova-priorities
* Thurs 17:20: Unconference https://etherpad.openstack.org/p/mitaka-nova-unconference

* Fri: 09:00 and 14:00: Nova contributors meetup https://etherpad.openstack.org/p/mitaka-nova-summit-meetup

== Manila ==
* Wed 11:15 - 11:55: (WS) Migration Improvements https://etherpad.openstack.org/p/mitaka-manila-migration-improvements
* Wed 12:05 - 12:45: (WS) Access Allow/Deny Driver Interface https://etherpad.openstack.org/p/mitaka-manila-allow-deny
* Thu 11:00 - 11:40: (FB) Share Replication https://etherpad.openstack.org/p/mitaka-manila-replication
* Thu 11:50 - 12:30: (FB) Alternative Snapshot Semantics https://etherpad.openstack.org/p/mitaka-manila-snapshot-semantics
* Thu 14:40 - 15:20: (WS) Export Location Metadata https://etherpad.openstack.org/p/mitaka-manila-export-location-metadata
* Thu 15:30 - 16:10: (WS) Interactions Between New Features https://etherpad.openstack.org/p/mitaka-manila-feature-interactions
* Fri 09:00 - 12:30: (CM) Contributor Meetup https://etherpad.openstack.org/p/mitaka-manila-contributor-meetup

== Release management ==
* Thu 15:30: Mitaka process changes [https://etherpad.openstack.org/p/mitaka-relmgt-process-changes]
* Thu 16:30: Work session: the Mitaka plan [https://etherpad.openstack.org/p/mitaka-relmgt-plan]

== Puppet OpenStack ==
* General etherpadː https://etherpad.openstack.org/p/HND-puppet
* Wed 2 pm: Code design sessionː https://etherpad.openstack.org/p/HND-puppet-code
* Wed 2.50 pm: Code design sessionː https://etherpad.openstack.org/p/HND-puppet-code
* Thu 1.50 pm: Community feedbackː https://etherpad.openstack.org/p/HND-puppet-community
* Thu 2.40 pm: CI and documentationː https://etherpad.openstack.org/p/HND-puppet-ci and https://etherpad.openstack.org/p/HND-puppet-doc
* Thu 4.30 pm: Code design sessionː https://etherpad.openstack.org/p/HND-puppet-code

== Ironic ==
* The everything etherpad: https://etherpad.openstack.org/p/summit-mitaka-ironic
* Wednesday fishbowl DS3 2:00-2:40 https://etherpad.openstack.org/p/summit-mitaka-ironic-third-party-ci
* Wednesday fishbowl DS3 2:50-3:30 https://etherpad.openstack.org/p/summit-mitaka-ironic-group-management
* Thursday workroom DS10 9:00-9:40 https://etherpad.openstack.org/p/summit-mitaka-ironic-notifications-bus
* Thursday workroom DS10 9:50-10:30 https://etherpad.openstack.org/p/summit-mitaka-ironic-driver-composition
* Thursday fishbowl DS3 11:00-11:40 https://etherpad.openstack.org/p/summit-mitaka-ironic-driver-api
* Thursday fishbowl DS3 11:50-12:30 https://etherpad.openstack.org/p/summit-mitaka-ironic-nova-driver (joint session with Nova)
* Thursday workroom DS11 4:30-5:10 https://etherpad.openstack.org/p/summit-mitaka-ironic-lock-manager
* Thursday workroom DS11 5:20-6:00 https://etherpad.openstack.org/p/summit-mitaka-ironic-gate-improvements
* Friday workroom DS15 9:00-12:30 https://etherpad.openstack.org/p/summit-mitaka-ironic-contributors-meetup (shared space with Infra)

== Heat ==
* The everything etherpad: https://etherpad.openstack.org/p/mitaka-heat-sessions
'''Wed'''
* 11:15 - 11:55: [https://etherpad.openstack.org/p/mitaka-heat-documentation (W) Documentation improvements]
* 12:05 - 12:45: [https://etherpad.openstack.org/p/mitaka-heat-tests (W) Heat tests]
* 14:00 - 14:40: [https://etherpad.openstack.org/p/mitaka-heat-convergence-migration (W) Tool to migrate stacks to/from convergence]
* 14:50 - 15:30: [https://etherpad.openstack.org/p/mitaka-heat-composition-improvements (F) Composition improvements ]
* 15:40 - 16:20: [https://etherpad.openstack.org/p/mitaka-heat-user-ops (F) User/ops session for summit ]
'''Thu'''
* 09:00 - 09:40: [https://etherpad.openstack.org/p/mitaka-heat-autoscaling (W) AutoScaling/Group architecture/roadmap]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/mitaka-heat-break-stack-barrier (W) Breaking the stack barrier]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/mitaka-heat-large-stacks (F) Issues from deploying very large stacks]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/mitaka-heat-openstackclient (F) Complete heat support in python-openstackclient ]
* 13:50 - 14:30: [https://etherpad.openstack.org/p/mitaka-heat-hooks-notifications (W) Hooks & Notifications ]
* 14:40 - 15:20: [https://etherpad.openstack.org/p/mitaka-heat-convergence-ph1 (W) Convergence Phase 1 results]
* 15:30 - 16:10: [https://etherpad.openstack.org/p/mitaka-heat-convergence-ph2 (W) Convergence Phase 2 start]
'''Fri'''
* 09:00 - 12:30: [https://etherpad.openstack.org/p/mitaka-heat-summit-meetup Contributor Meetup]

== OpenStack Chef ==
* general discussion: https://etherpad.openstack.org/p/mitaka-openstack-chef-general
* defining the refactoring process: https://etherpad.openstack.org/p/mitaka-openstack-chef-refactoring

Manila/Meetings

2015-09-03T14:47:25Z

Clinton Knight: /* Next meeting */

Manila/Meetings

2015-09-03T14:46:15Z

Clinton Knight: /* Next meeting */

Manila/Meetings

2015-09-03T14:29:25Z

Clinton Knight: /* Next meeting */

Manila/Meetings

2015-09-03T14:28:16Z

Clinton Knight: /* Next meeting */

Design Summit/Liberty/Etherpads

2015-05-14T15:38:13Z

Clinton Knight: /* Manila */

[[Category:Summit]]
[[Category:Liberty]]
[[Category:Etherpad]]

<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
__TOC__
</div>
==Cross-Project workshops==
Tuesday:

* 11:15 - 11:55
** [https://etherpad.openstack.org/p/liberty-cross-project-python3 Moving apps to Python 3]
** [https://etherpad.openstack.org/p/liberty-cross-project-user-notifications Async status updates]
* 12:05 - 12:45
** Improving UX across all projects
** Functional Testing Show & Tell
* 2:00 - 2:40
** OpenStack SDK
** Modern JavaScript
* 2:50 - 3:30
** [https://etherpad.openstack.org/p/liberty-cross-project-in-team-scaling In-team scaling]
** Service Catalog Standardization
* 3:40 - 4:20
** [https://etherpad.openstack.org/p/liberty-cross-project-api-wg API Working Group]
** Towards one Network Stack: Part 1 - Neutron Gaps and Concerns
* 4:40 - 5:20
** [https://etherpad.openstack.org/p/liberty-cross-project-release-models OpenStack release model(s)]
** Unified Policy File
* 5:30 - 6:10
** OpenStack Documentation
** [https://etherpad.openstack.org/p/liberty-cross-project-managing-concurrency Managing concurrency]

==Barbican==
==Ceilometer==
wednesday:
* 0900 - 0940: [https://etherpad.openstack.org/p/ceilo-multi-identity componentisation / multi identity]
* 0950 - 1030: [https://etherpad.openstack.org/p/event_alarm event alarms]
* 1100 - 1140: [https://etherpad.openstack.org/p/liberty-ceilometer-pipeline-config pipeline configuration]
* 1150 - 1230: [https://etherpad.openstack.org/p/ceilo-declarative-notifications declarative notification meters]
* 1440 - 1520: [https://etherpad.openstack.org/p/YVR-ops-ceilometer ops followup]
* 1530 - 1610: [https://etherpad.openstack.org/p/ceilo-multi-identity componentization carry over]

 thursday:
* 0900 - 0940: [https://etherpad.openstack.org/p/liberty-ceilometer-meter-deprecation meter deprecation]
* 0950 - 1030: [https://etherpad.openstack.org/p/liberty-ceilometer-meter-event samples/events integration]
* 1100 - 1140: [https://etherpad.openstack.org/p/liberty-ceilometer-versioned-objects versioned objects]
* 1150 - 1230: ops followup

 friday:
* 0900 - 1200: [https://etherpad.openstack.org/p/liberty-ceilometer-contributors-meetup contributor meetup]

==Cinder==
==Designate==
==Documentation==

* Wednesday • 1:50pm - 2:30pm https://etherpad.openstack.org/p/Documentation__RST_Migration
* Wednesday • 3:30pm - 4:10pm https://etherpad.openstack.org/p/Documentation__Ops_Work_Session
* Thursday • 11:50am - 12:30pm https://etherpad.openstack.org/p/Documentation__API_Work_Session
* Thursday • 1:30pm - 2:10pm https://etherpad.openstack.org/p/Documentation__Blueprint_Work_Session
* Thursday • 2:20pm - 3:00pm https://etherpad.openstack.org/p/Documentation__Team_Structure_1
* Thursday • 3:10pm - 3:50pm https://etherpad.openstack.org/p/Documentation__Team_Structure_2
* Thursday • 4:10pm - 4:50pm https://etherpad.openstack.org/p/Documentation__Info_Architecture_1
* Thursday • 5:00pm - 5:40pm https://etherpad.openstack.org/p/Documentation__Info_Architecture_2
* Friday • 9:00am - 12:20pm https://etherpad.openstack.org/p/Documentation__Contributors_Meetup

==Glance==
==Heat==
'''Wed'''
* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-heat-liberty-convergence-1 (W) Finishing off Convergence phase 1]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-heat-liberty-convergence-2 (W) Convergence phase 2 planning]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-heat-liberty-testing (W) Functional and integration testing, identify gaps and suggest improvements]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-heat-liberty-lifecycle-ops (W) Better support for lifecycle operations on (tripleo) heat stacks (upgrades etc)]
* 13:50 - 14:30: [https://etherpad.openstack.org/p/YVR-heat-liberty-client-usability (F) Heat client usability]
* 14:40 - 15:20: [https://etherpad.openstack.org/p/YVR-heat-liberty-template-format (F) Heat template format improvements]
* 15:30 - 16:10: [https://etherpad.openstack.org/p/YVR-heat-liberty-rest-api (F) REST API design and roadmap]
* 16:30 - 17:10: [https://etherpad.openstack.org/p/YVR-heat-liberty-senlin-autoscaling (F) Senlin Autoscaling Project - Deep Dive]
* 17:20 - 18:00: [https://etherpad.openstack.org/p/YVR-heat-liberty-containers (F) Orchestrating containers with Heat]

'''Thu'''
* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-heat-liberty-alembic (W) Migrating from sqlalchemy-migrations to alembic]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-heat-liberty-versioned-objects (W) Future of Versioned objects in heat]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-heat-liberty-autoscaling (W) Autoscaling improvements]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-heat-liberty-deprecation (W) Our deprecation process]
* 16:10 - 16:50: [https://etherpad.openstack.org/p/YVR-heat-liberty-contrib-plugins (W) Moving most contrib plugins in-tree]
* 17:00 - 17:40: [https://etherpad.openstack.org/p/YVR-heat-liberty-resource-registering (W) Conditional registering of resources]

'''Fri'''
* 09:00-12:20: [https://etherpad.openstack.org/p/YVR-heat-contributor-meetup Contributor Meetup]

==Horizon==
==Infrastructure==
* Wed 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-infra-irc-meetings (W) Work session on Gerrit-driven IRC meetings]
* Wed 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-infra-activity-tracking (W) Work session on activity tracking]
* Thu 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-infra-task-tracking (F) Task tracking]

==Ironic==
==Keystone==
==Manila==
'''Wed'''
* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-manila-liberty-mount-automation (F) Mount automation]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-manila-liberty-share-migration (F) Share migration]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-manila-liberty-share-replication (F) Share replication]
* 16:30 - 17:10: [https://etherpad.openstack.org/p/YVR-manila-liberty-versioned-objects-and-apis (W) Versioned objects and APIs]
* 17:20 - 18:00: [https://etherpad.openstack.org/p/YVR-manila-liberty-fault-indication (W) Fault indication]

'''Thu'''
* 16:10 - 16:50: [https://etherpad.openstack.org/p/YVR-manila-liberty-thin-provisioning (W) Thin provisioning and consistency groups]
* 17:00 - 17:40: [https://etherpad.openstack.org/p/YVR-manila-service-image (W) Snapshot-share relationship and Manila service image]

==Neutron==

'''Wed'''
* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-neutron-liberty-development Neutron Liberty Development]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-neutron-use-case-discussion Neutron Use Case Discussion]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-neutron-vlan-trunk VLAN Aware VMs]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-neutron-lightning-talks Neutron Lightning Talks]
* 13:50 - 14:30: [https://etherpad.openstack.org/p/YVR-neutron-octavia Octavia]
* 14:40 - 15:20: [https://etherpad.openstack.org/p/YVR-neutron-opendaylight OpenDaylight]
* 15:30 - 16:10: [https://etherpad.openstack.org/p/YVR-neutron-ovn OVN]
* 16:30 - 17:10: [https://etherpad.openstack.org/p/YVR-neutron-ironic Ironic and Neutron integration]
* 17:20 - 18:00: [https://etherpad.openstack.org/p/YVR-neutron-get-me-a-network Get Me a Network!]

'''Thu'''
* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-neutron-lbaas-use-cases Neutron LBaaS Use Cases]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-neutron-l3 Neutron L3]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-neutron-qos QoS]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-neutron-third-party-ci-liberty Third Party CI in Liberty and Beyond]
* 13:30 - 14:10: [https://etherpad.openstack.org/p/YVR-neutron-testing-in-liberty Testing In Liberty]
* 14:20 - 15:00: [https://etherpad.openstack.org/p/YVR-neutron-RBAC Neutron RBAC]
* 15:10 - 15:50: [https://etherpad.openstack.org/p/YVR-neutron-sg-fwaas-future-direction SG and FWaaS Future Directions]
* 16:10 - 16:50: [https://etherpad.openstack.org/p/YVR-neutron-nfv-enhancements Neutron NFV Enhancements]

'''Fri'''
* 09:00-12:20: [https://etherpad.openstack.org/p/YVR-neutron-contributor-meetup Contributor Meetup]

==Nova==

'''Wednesday:'''

* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-nova-scheduler-in-liberty Scheduler in Liberty]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-nova-scalling-out-scheduler-for-cells Scaling out scheduler for cells ]

* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-nova-cells-v2 Cells v2]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-nova-resource-tracker Resource Tracker, Clustered Hypervisors and NFV]

* 13:50 - 14:30: [https://etherpad.openstack.org/p/YVR-nova-spec-blueprint-unconference Nova Spec/Blueprint Unconference]
* 14:40 - 15:20: [https://etherpad.openstack.org/p/YVR-nova-database-internals Database (part 1)]
* 15:30 - 16:10: [https://etherpad.openstack.org/p/YVR-nova-functional-testing-feature-classification Functional Testing and Feature Classification]

* 16:30 - 17:10: [https://etherpad.openstack.org/p/YVR-nova-instance-ha-evacuate-resize Dealing with compute host failure: Instance HA, Evacuate, Resize]
* 17:20 - 18:00: [https://etherpad.openstack.org/p/YVR-nova-api-2.1-in-liberty Nova API v2.1 in Liberty]

 
'''Thursday:'''

* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-nova-api-2.0-3rd-party Future of Nova API v2.0 and 3rd Party APIs]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-nova-quotas-and-database Quotas and Database (part 2)]

* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-nova-flavors-and-image-properties Flavors and Image Properties]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-nova-error-handling Error Handling]

* 13:30 - 14:10: [https://etherpad.openstack.org/p/YVR-nova-spec-blueprint-unconference Nova Spec/Blueprint Unconference]
* 14:20 - 15:00: [https://etherpad.openstack.org/p/YVR-nova-liberty-priorities Liberty Priorities (part 1)]
* 15:10 - 15:50: [https://etherpad.openstack.org/p/YVR-nova-liberty-priorities Liberty Priorities (part 2)]

* 16:10 - 16:50: [https://etherpad.openstack.org/p/YVR-nova-liberty-process Liberty Process and Scaling out Reviews]
* 17:00 - 17:40: [https://etherpad.openstack.org/p/YVR-nova-network Future of Nova's networking and nova-network]

 
'''Friday:'''

* 09:00 - 12:20 and 13:20 - 16:40: [https://etherpad.openstack.org/p/YVR-nova-contributor-meetup Nova Contributor Meetup]

==Oslo==

'''Wednesday:'''

* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-oslo-versioned-objects-intro (F) Get to know your objects and learn how to version them (an introduction to oslo.versionedobjects)!]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-oslo-taskflow-plans (F) Give me liberty, or give me taskflow (come learn about taskflow liberty plans)!]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-oslo-rootwrap-plans (W) Give me liberty, or give me wraps (the future of oslo.rootwrap) ]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-oslo-functional-testing (W) The cost of liberty is less than the price of functional testing]
* 13:50 - 14:30: [https://etherpad.openstack.org/p/YVR-oslo-graduation-schedule (W) Give me more oslo in liberty or else! Schedule & new libraries]
* 14:40 - 15:20: [https://etherpad.openstack.org/p/YVR-oslo-messaging-zmq-status (F) State of zmq in oslo.messaging]
* 15:30 - 16:10: [https://etherpad.openstack.org/p/YVR-oslo-optional-dependencies (W) Emancipate/liberate your optional dependencies - Optional Dependencies]
* 16:30 - 17:10: [https://etherpad.openstack.org/p/YVR-oslo-release-process-review (W) Reviewing our release processes]
* 17:20 - 18:00: [https://etherpad.openstack.org/p/YVR-oslo-config-filter (W) Configuration Filters in oslo.config]

 
'''Thursday:'''

* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-oslo-strategy-discussion (F) The oslo liberty proclamation (and associated strategy discussion)]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-oslo-config-plans (F) Enfranchise oslo.config, let's discuss alternative data sources in oslo.config]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-oslo-log-plans (W) Life, Liberty, and the pursuit of oslo.log changes]
* 13:30 - 14:10: [https://etherpad.openstack.org/p/YVR-oslo-versioned-objects-deep-dive (W) Deep dive on oslo.versionedobjects, bring your wet suits.]
* 14:20 - 15:00: [https://etherpad.openstack.org/p/YVR-oslo-db-plans (F) For a people who are free, and who mean to remain so, a well organized and armed 'oslo.db' is their best security]
* 15:10 - 15:50: [https://etherpad.openstack.org/p/YVR-oslo-asyncio (F) Event loops, coroutines, yield from, futures, a discussion on asyncio (and triollus?)]
* 16:10 - 16:50: [https://etherpad.openstack.org/p/YVR-oslo-messaging-plans (F) Ping pong, oslo.messaging plans for liberty.]
* 17:00 - 17:40: [https://etherpad.openstack.org/p/YVR-oslo-tech-debt-deprecation (F) How to clean up your tech-debt; let's discuss best practices on how to deprecate things in oslo libraries]

 

==QA==

'''Wed'''

* 1150-1230 - [https://etherpad.openstack.org/p/YVR-QA-testing-beyond-the-gate Testing beyond the gate]
* 1350-1430 - [https://etherpad.openstack.org/p/YVR-QA-Devstack-Roadmap Devstack Roadmap]
* 1440-1520 - [https://etherpad.openstack.org/p/YVR-QA-Tempest-service-clients Work Session: Tempest service clients]
* 1630-1710 - [https://etherpad.openstack.org/p/YVR-QA-in-the-big-tent QA in the Big Tent]

'''Thurs.'''
* 0900-0940 - [https://etherpad.openstack.org/p/YVR-QA-idempotent-id Work Session: Idempotent ID]
* 1330-1410 - [https://etherpad.openstack.org/p/YVR-QA-Tempest-as-a-system-program Work Session: Tempest as System Program]
* 1420-1500 - [https://etherpad.openstack.org/p/YVR-QA-Tempest-CLI Work Session: Tempest CLI and Tempest Cleanup]
* 1610-1650 - [https://etherpad.openstack.org/p/YVR-QA-Tempest-Scope-Revisited Tempest Scope Revisited]
* 1700-1740 - [https://etherpad.openstack.org/p/YVR-QA-Liberty-Priorities Liberty Priorities]

==Release Management==
* Wed 17:20-18:00: [https://etherpad.openstack.org/p/YVR-relmgt-stable-branch Stable branch]
* Thu 11:00-11:40 [https://etherpad.openstack.org/p/YVR-relmgt-library-releases Work session: Library releases]

==Sahara==

'''Wed'''

* 11:00am (W) HA http://etherpad.openstack.org/p/sahara-liberty-ha
* 11:50am (W) API v2 https://etherpad.openstack.org/p/sahara-liberty-api-v2

''Thu'''

* 1:30pm (F) UI https://etherpad.openstack.org/p/sahara-liberty-ui
* 2:20pm (F) Security https://etherpad.openstack.org/p/sahara-liberty-security
* 3:10pm (W) EDP https://etherpad.openstack.org/p/sahara-liberty-edp
* 4:10pm (W) Resources Visibility/ACL https://etherpad.openstack.org/p/sahara-liberty-resources
* 5:00pm (W) Plugins https://etherpad.openstack.org/p/sahara-liberty-plugins

''Fri'''

* 9am - 4:40pm Meetup https://etherpad.openstack.org/p/sahara-liberty-meetup

==Security==
* Wed 13:50-14:30: [https://etherpad.openstack.org/p/YVR-security-vmt Vulnerability management]

==Swift==
==TripleO==

==Trove==
==Zaqar==
==Murano==
'''Thu'''

* 2:20pm (F) [https://etherpad.openstack.org/p/murano-liberty-now-n-future Murano: Now, Liberty and Future]
* 4:10pm (W) [https://etherpad.openstack.org/p/murano-liberty-work-session-1 Images & Murano]
* 5:00pm (W) [https://etherpad.openstack.org/p/murano-liberty-work-session-2 Authentication & Agent]

'''Fri'''

* 9am - 4:40pm [https://etherpad.openstack.org/p/murano-liberty-contributors-meetup Contributors Meetup]

==Event intro/closure==
* Tue 11:15-11:55: [https://etherpad.openstack.org/p/YVR-design-summit-101 Design Summit 101]
* Fri 16:50-17:30: [https://etherpad.openstack.org/p/YVR-design-summit-feedback Design Summit feedback session]

==Ops==
Tuesday:
* https://etherpad.openstack.org/p/YVR-ops-101
* https://etherpad.openstack.org/p/YVR-ops-federation
* https://etherpad.openstack.org/p/YVR-ops-rabbitmq
* https://etherpad.openstack.org/p/YVR-ops-logging
* https://etherpad.openstack.org/p/YVR-ops-arch-show-tell
* https://etherpad.openstack.org/p/YVR-ops-ceilometer
* https://etherpad.openstack.org/p/YVR-ops-billing
* https://etherpad.openstack.org/p/YVR-ops-cinder
* https://etherpad.openstack.org/p/YVR-ops-legacy-apps
* https://etherpad.openstack.org/p/YVR-ops-user-committee
* https://etherpad.openstack.org/p/YVR-ops-hypervisor-tuning
* https://etherpad.openstack.org/p/YVR-ops-security
* https://etherpad.openstack.org/p/YVR-ops-deployment
* https://etherpad.openstack.org/p/YVR-ops-database
* https://etherpad.openstack.org/p/YVR-ops-evangelism
* https://etherpad.openstack.org/p/YVR-ops-multi-site
* https://etherpad.openstack.org/p/YVR-ops-nova
* https://etherpad.openstack.org/p/YVR-ops-customer-onboarding
* https://etherpad.openstack.org/p/YVR-ops-containers
* https://etherpad.openstack.org/p/YVR-ops-neutron

Wednesday:
* https://etherpad.openstack.org/p/YVR-ops-telco
* https://etherpad.openstack.org/p/YVR-ops-puppet
* https://etherpad.openstack.org/p/YVR-ops-chef
* https://etherpad.openstack.org/p/YVR-ops-hpc
* https://etherpad.openstack.org/p/YVR-ops-tools
* https://etherpad.openstack.org/p/YVR-ops-ansible
* https://etherpad.openstack.org/p/YVR-ops-ceph
* https://etherpad.openstack.org/p/YVR-ops-tags
* https://etherpad.openstack.org/p/YVR-ops-large-deployments
* https://etherpad.openstack.org/p/YVR-ops-burning-issues
* https://etherpad.openstack.org/p/YVR-ops-docs
* https://etherpad.openstack.org/p/YVR-ops-tech-choices
* https://etherpad.openstack.org/p/YVR-ops-cmdb
* https://etherpad.openstack.org/p/YVR-ops-data-plane-transitions
* https://etherpad.openstack.org/p/YVR-ops-upgrades
* https://etherpad.openstack.org/p/YVR-ops-packaging
* https://etherpad.openstack.org/p/YVR-ops-nova-network

Design Summit/Liberty/Etherpads

2015-05-14T15:37:44Z

Clinton Knight: /* Manila */

[[Category:Summit]]
[[Category:Liberty]]
[[Category:Etherpad]]

<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
__TOC__
</div>
==Cross-Project workshops==
Tuesday:

* 11:15 - 11:55
** [https://etherpad.openstack.org/p/liberty-cross-project-python3 Moving apps to Python 3]
** [https://etherpad.openstack.org/p/liberty-cross-project-user-notifications Async status updates]
* 12:05 - 12:45
** Improving UX across all projects
** Functional Testing Show & Tell
* 2:00 - 2:40
** OpenStack SDK
** Modern JavaScript
* 2:50 - 3:30
** [https://etherpad.openstack.org/p/liberty-cross-project-in-team-scaling In-team scaling]
** Service Catalog Standardization
* 3:40 - 4:20
** [https://etherpad.openstack.org/p/liberty-cross-project-api-wg API Working Group]
** Towards one Network Stack: Part 1 - Neutron Gaps and Concerns
* 4:40 - 5:20
** [https://etherpad.openstack.org/p/liberty-cross-project-release-models OpenStack release model(s)]
** Unified Policy File
* 5:30 - 6:10
** OpenStack Documentation
** [https://etherpad.openstack.org/p/liberty-cross-project-managing-concurrency Managing concurrency]

==Barbican==
==Ceilometer==
wednesday:
* 0900 - 0940: [https://etherpad.openstack.org/p/ceilo-multi-identity componentisation / multi identity]
* 0950 - 1030: [https://etherpad.openstack.org/p/event_alarm event alarms]
* 1100 - 1140: [https://etherpad.openstack.org/p/liberty-ceilometer-pipeline-config pipeline configuration]
* 1150 - 1230: [https://etherpad.openstack.org/p/ceilo-declarative-notifications declarative notification meters]
* 1440 - 1520: [https://etherpad.openstack.org/p/YVR-ops-ceilometer ops followup]
* 1530 - 1610: [https://etherpad.openstack.org/p/ceilo-multi-identity componentization carry over]

 thursday:
* 0900 - 0940: [https://etherpad.openstack.org/p/liberty-ceilometer-meter-deprecation meter deprecation]
* 0950 - 1030: [https://etherpad.openstack.org/p/liberty-ceilometer-meter-event samples/events integration]
* 1100 - 1140: [https://etherpad.openstack.org/p/liberty-ceilometer-versioned-objects versioned objects]
* 1150 - 1230: ops followup

 friday:
* 0900 - 1200: [https://etherpad.openstack.org/p/liberty-ceilometer-contributors-meetup contributor meetup]

==Cinder==
==Designate==
==Documentation==

* Wednesday • 1:50pm - 2:30pm https://etherpad.openstack.org/p/Documentation__RST_Migration
* Wednesday • 3:30pm - 4:10pm https://etherpad.openstack.org/p/Documentation__Ops_Work_Session
* Thursday • 11:50am - 12:30pm https://etherpad.openstack.org/p/Documentation__API_Work_Session
* Thursday • 1:30pm - 2:10pm https://etherpad.openstack.org/p/Documentation__Blueprint_Work_Session
* Thursday • 2:20pm - 3:00pm https://etherpad.openstack.org/p/Documentation__Team_Structure_1
* Thursday • 3:10pm - 3:50pm https://etherpad.openstack.org/p/Documentation__Team_Structure_2
* Thursday • 4:10pm - 4:50pm https://etherpad.openstack.org/p/Documentation__Info_Architecture_1
* Thursday • 5:00pm - 5:40pm https://etherpad.openstack.org/p/Documentation__Info_Architecture_2
* Friday • 9:00am - 12:20pm https://etherpad.openstack.org/p/Documentation__Contributors_Meetup

==Glance==
==Heat==
'''Wed'''
* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-heat-liberty-convergence-1 (W) Finishing off Convergence phase 1]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-heat-liberty-convergence-2 (W) Convergence phase 2 planning]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-heat-liberty-testing (W) Functional and integration testing, identify gaps and suggest improvements]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-heat-liberty-lifecycle-ops (W) Better support for lifecycle operations on (tripleo) heat stacks (upgrades etc)]
* 13:50 - 14:30: [https://etherpad.openstack.org/p/YVR-heat-liberty-client-usability (F) Heat client usability]
* 14:40 - 15:20: [https://etherpad.openstack.org/p/YVR-heat-liberty-template-format (F) Heat template format improvements]
* 15:30 - 16:10: [https://etherpad.openstack.org/p/YVR-heat-liberty-rest-api (F) REST API design and roadmap]
* 16:30 - 17:10: [https://etherpad.openstack.org/p/YVR-heat-liberty-senlin-autoscaling (F) Senlin Autoscaling Project - Deep Dive]
* 17:20 - 18:00: [https://etherpad.openstack.org/p/YVR-heat-liberty-containers (F) Orchestrating containers with Heat]

'''Thu'''
* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-heat-liberty-alembic (W) Migrating from sqlalchemy-migrations to alembic]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-heat-liberty-versioned-objects (W) Future of Versioned objects in heat]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-heat-liberty-autoscaling (W) Autoscaling improvements]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-heat-liberty-deprecation (W) Our deprecation process]
* 16:10 - 16:50: [https://etherpad.openstack.org/p/YVR-heat-liberty-contrib-plugins (W) Moving most contrib plugins in-tree]
* 17:00 - 17:40: [https://etherpad.openstack.org/p/YVR-heat-liberty-resource-registering (W) Conditional registering of resources]

'''Fri'''
* 09:00-12:20: [https://etherpad.openstack.org/p/YVR-heat-contributor-meetup Contributor Meetup]

==Horizon==
==Infrastructure==
* Wed 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-infra-irc-meetings (W) Work session on Gerrit-driven IRC meetings]
* Wed 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-infra-activity-tracking (W) Work session on activity tracking]
* Thu 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-infra-task-tracking (F) Task tracking]

==Ironic==
==Keystone==
==Manila==
'''Wed'''
* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-manila-liberty-mount-automation (F) Mount automation]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-manila-liberty-share-migration (F) Share migration]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-manila-liberty-share-replication (F) Share replication]
* 16:30 - 17:10: [https://etherpad.openstack.org/p/YVR-manila-liberty-versioned-objects-and-apis (W) Versioned objects and APIs]
* 17:20 - 18:00: [https://etherpad.openstack.org/p/YVR-manila-liberty-fault-indication (W) Fault indication]

'''Thu'''
* 16:10 - 16:50: [https://etherpad.openstack.org/p/YVR-manila-liberty-thin-provisioning (W) Thin provisioning and consistency groups]
* 17:00 - 17:40: [https://etherpad.openstack.org/p/YVR-manila-liberty-snapshot-share-relationship-manila-service-image (W) Snapshot-share relationship and Manila service image]

==Neutron==

'''Wed'''
* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-neutron-liberty-development Neutron Liberty Development]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-neutron-use-case-discussion Neutron Use Case Discussion]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-neutron-vlan-trunk VLAN Aware VMs]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-neutron-lightning-talks Neutron Lightning Talks]
* 13:50 - 14:30: [https://etherpad.openstack.org/p/YVR-neutron-octavia Octavia]
* 14:40 - 15:20: [https://etherpad.openstack.org/p/YVR-neutron-opendaylight OpenDaylight]
* 15:30 - 16:10: [https://etherpad.openstack.org/p/YVR-neutron-ovn OVN]
* 16:30 - 17:10: [https://etherpad.openstack.org/p/YVR-neutron-ironic Ironic and Neutron integration]
* 17:20 - 18:00: [https://etherpad.openstack.org/p/YVR-neutron-get-me-a-network Get Me a Network!]

'''Thu'''
* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-neutron-lbaas-use-cases Neutron LBaaS Use Cases]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-neutron-l3 Neutron L3]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-neutron-qos QoS]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-neutron-third-party-ci-liberty Third Party CI in Liberty and Beyond]
* 13:30 - 14:10: [https://etherpad.openstack.org/p/YVR-neutron-testing-in-liberty Testing In Liberty]
* 14:20 - 15:00: [https://etherpad.openstack.org/p/YVR-neutron-RBAC Neutron RBAC]
* 15:10 - 15:50: [https://etherpad.openstack.org/p/YVR-neutron-sg-fwaas-future-direction SG and FWaaS Future Directions]
* 16:10 - 16:50: [https://etherpad.openstack.org/p/YVR-neutron-nfv-enhancements Neutron NFV Enhancements]

'''Fri'''
* 09:00-12:20: [https://etherpad.openstack.org/p/YVR-neutron-contributor-meetup Contributor Meetup]

==Nova==

'''Wednesday:'''

* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-nova-scheduler-in-liberty Scheduler in Liberty]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-nova-scalling-out-scheduler-for-cells Scaling out scheduler for cells ]

* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-nova-cells-v2 Cells v2]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-nova-resource-tracker Resource Tracker, Clustered Hypervisors and NFV]

* 13:50 - 14:30: [https://etherpad.openstack.org/p/YVR-nova-spec-blueprint-unconference Nova Spec/Blueprint Unconference]
* 14:40 - 15:20: [https://etherpad.openstack.org/p/YVR-nova-database-internals Database (part 1)]
* 15:30 - 16:10: [https://etherpad.openstack.org/p/YVR-nova-functional-testing-feature-classification Functional Testing and Feature Classification]

* 16:30 - 17:10: [https://etherpad.openstack.org/p/YVR-nova-instance-ha-evacuate-resize Dealing with compute host failure: Instance HA, Evacuate, Resize]
* 17:20 - 18:00: [https://etherpad.openstack.org/p/YVR-nova-api-2.1-in-liberty Nova API v2.1 in Liberty]

 
'''Thursday:'''

* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-nova-api-2.0-3rd-party Future of Nova API v2.0 and 3rd Party APIs]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-nova-quotas-and-database Quotas and Database (part 2)]

* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-nova-flavors-and-image-properties Flavors and Image Properties]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-nova-error-handling Error Handling]

* 13:30 - 14:10: [https://etherpad.openstack.org/p/YVR-nova-spec-blueprint-unconference Nova Spec/Blueprint Unconference]
* 14:20 - 15:00: [https://etherpad.openstack.org/p/YVR-nova-liberty-priorities Liberty Priorities (part 1)]
* 15:10 - 15:50: [https://etherpad.openstack.org/p/YVR-nova-liberty-priorities Liberty Priorities (part 2)]

* 16:10 - 16:50: [https://etherpad.openstack.org/p/YVR-nova-liberty-process Liberty Process and Scaling out Reviews]
* 17:00 - 17:40: [https://etherpad.openstack.org/p/YVR-nova-network Future of Nova's networking and nova-network]

 
'''Friday:'''

* 09:00 - 12:20 and 13:20 - 16:40: [https://etherpad.openstack.org/p/YVR-nova-contributor-meetup Nova Contributor Meetup]

==Oslo==

'''Wednesday:'''

* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-oslo-versioned-objects-intro (F) Get to know your objects and learn how to version them (an introduction to oslo.versionedobjects)!]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-oslo-taskflow-plans (F) Give me liberty, or give me taskflow (come learn about taskflow liberty plans)!]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-oslo-rootwrap-plans (W) Give me liberty, or give me wraps (the future of oslo.rootwrap) ]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-oslo-functional-testing (W) The cost of liberty is less than the price of functional testing]
* 13:50 - 14:30: [https://etherpad.openstack.org/p/YVR-oslo-graduation-schedule (W) Give me more oslo in liberty or else! Schedule & new libraries]
* 14:40 - 15:20: [https://etherpad.openstack.org/p/YVR-oslo-messaging-zmq-status (F) State of zmq in oslo.messaging]
* 15:30 - 16:10: [https://etherpad.openstack.org/p/YVR-oslo-optional-dependencies (W) Emancipate/liberate your optional dependencies - Optional Dependencies]
* 16:30 - 17:10: [https://etherpad.openstack.org/p/YVR-oslo-release-process-review (W) Reviewing our release processes]
* 17:20 - 18:00: [https://etherpad.openstack.org/p/YVR-oslo-config-filter (W) Configuration Filters in oslo.config]

 
'''Thursday:'''

* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-oslo-strategy-discussion (F) The oslo liberty proclamation (and associated strategy discussion)]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-oslo-config-plans (F) Enfranchise oslo.config, let's discuss alternative data sources in oslo.config]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-oslo-log-plans (W) Life, Liberty, and the pursuit of oslo.log changes]
* 13:30 - 14:10: [https://etherpad.openstack.org/p/YVR-oslo-versioned-objects-deep-dive (W) Deep dive on oslo.versionedobjects, bring your wet suits.]
* 14:20 - 15:00: [https://etherpad.openstack.org/p/YVR-oslo-db-plans (F) For a people who are free, and who mean to remain so, a well organized and armed 'oslo.db' is their best security]
* 15:10 - 15:50: [https://etherpad.openstack.org/p/YVR-oslo-asyncio (F) Event loops, coroutines, yield from, futures, a discussion on asyncio (and triollus?)]
* 16:10 - 16:50: [https://etherpad.openstack.org/p/YVR-oslo-messaging-plans (F) Ping pong, oslo.messaging plans for liberty.]
* 17:00 - 17:40: [https://etherpad.openstack.org/p/YVR-oslo-tech-debt-deprecation (F) How to clean up your tech-debt; let's discuss best practices on how to deprecate things in oslo libraries]

 

==QA==

'''Wed'''

* 1150-1230 - [https://etherpad.openstack.org/p/YVR-QA-testing-beyond-the-gate Testing beyond the gate]
* 1350-1430 - [https://etherpad.openstack.org/p/YVR-QA-Devstack-Roadmap Devstack Roadmap]
* 1440-1520 - [https://etherpad.openstack.org/p/YVR-QA-Tempest-service-clients Work Session: Tempest service clients]
* 1630-1710 - [https://etherpad.openstack.org/p/YVR-QA-in-the-big-tent QA in the Big Tent]

'''Thurs.'''
* 0900-0940 - [https://etherpad.openstack.org/p/YVR-QA-idempotent-id Work Session: Idempotent ID]
* 1330-1410 - [https://etherpad.openstack.org/p/YVR-QA-Tempest-as-a-system-program Work Session: Tempest as System Program]
* 1420-1500 - [https://etherpad.openstack.org/p/YVR-QA-Tempest-CLI Work Session: Tempest CLI and Tempest Cleanup]
* 1610-1650 - [https://etherpad.openstack.org/p/YVR-QA-Tempest-Scope-Revisited Tempest Scope Revisited]
* 1700-1740 - [https://etherpad.openstack.org/p/YVR-QA-Liberty-Priorities Liberty Priorities]

==Release Management==
* Wed 17:20-18:00: [https://etherpad.openstack.org/p/YVR-relmgt-stable-branch Stable branch]
* Thu 11:00-11:40 [https://etherpad.openstack.org/p/YVR-relmgt-library-releases Work session: Library releases]

==Sahara==

'''Wed'''

* 11:00am (W) HA http://etherpad.openstack.org/p/sahara-liberty-ha
* 11:50am (W) API v2 https://etherpad.openstack.org/p/sahara-liberty-api-v2

''Thu'''

* 1:30pm (F) UI https://etherpad.openstack.org/p/sahara-liberty-ui
* 2:20pm (F) Security https://etherpad.openstack.org/p/sahara-liberty-security
* 3:10pm (W) EDP https://etherpad.openstack.org/p/sahara-liberty-edp
* 4:10pm (W) Resources Visibility/ACL https://etherpad.openstack.org/p/sahara-liberty-resources
* 5:00pm (W) Plugins https://etherpad.openstack.org/p/sahara-liberty-plugins

''Fri'''

* 9am - 4:40pm Meetup https://etherpad.openstack.org/p/sahara-liberty-meetup

==Security==
* Wed 13:50-14:30: [https://etherpad.openstack.org/p/YVR-security-vmt Vulnerability management]

==Swift==
==TripleO==

==Trove==
==Zaqar==
==Murano==
'''Thu'''

* 2:20pm (F) [https://etherpad.openstack.org/p/murano-liberty-now-n-future Murano: Now, Liberty and Future]
* 4:10pm (W) [https://etherpad.openstack.org/p/murano-liberty-work-session-1 Images & Murano]
* 5:00pm (W) [https://etherpad.openstack.org/p/murano-liberty-work-session-2 Authentication & Agent]

'''Fri'''

* 9am - 4:40pm [https://etherpad.openstack.org/p/murano-liberty-contributors-meetup Contributors Meetup]

==Event intro/closure==
* Tue 11:15-11:55: [https://etherpad.openstack.org/p/YVR-design-summit-101 Design Summit 101]
* Fri 16:50-17:30: [https://etherpad.openstack.org/p/YVR-design-summit-feedback Design Summit feedback session]

==Ops==
Tuesday:
* https://etherpad.openstack.org/p/YVR-ops-101
* https://etherpad.openstack.org/p/YVR-ops-federation
* https://etherpad.openstack.org/p/YVR-ops-rabbitmq
* https://etherpad.openstack.org/p/YVR-ops-logging
* https://etherpad.openstack.org/p/YVR-ops-arch-show-tell
* https://etherpad.openstack.org/p/YVR-ops-ceilometer
* https://etherpad.openstack.org/p/YVR-ops-billing
* https://etherpad.openstack.org/p/YVR-ops-cinder
* https://etherpad.openstack.org/p/YVR-ops-legacy-apps
* https://etherpad.openstack.org/p/YVR-ops-user-committee
* https://etherpad.openstack.org/p/YVR-ops-hypervisor-tuning
* https://etherpad.openstack.org/p/YVR-ops-security
* https://etherpad.openstack.org/p/YVR-ops-deployment
* https://etherpad.openstack.org/p/YVR-ops-database
* https://etherpad.openstack.org/p/YVR-ops-evangelism
* https://etherpad.openstack.org/p/YVR-ops-multi-site
* https://etherpad.openstack.org/p/YVR-ops-nova
* https://etherpad.openstack.org/p/YVR-ops-customer-onboarding
* https://etherpad.openstack.org/p/YVR-ops-containers
* https://etherpad.openstack.org/p/YVR-ops-neutron

Wednesday:
* https://etherpad.openstack.org/p/YVR-ops-telco
* https://etherpad.openstack.org/p/YVR-ops-puppet
* https://etherpad.openstack.org/p/YVR-ops-chef
* https://etherpad.openstack.org/p/YVR-ops-hpc
* https://etherpad.openstack.org/p/YVR-ops-tools
* https://etherpad.openstack.org/p/YVR-ops-ansible
* https://etherpad.openstack.org/p/YVR-ops-ceph
* https://etherpad.openstack.org/p/YVR-ops-tags
* https://etherpad.openstack.org/p/YVR-ops-large-deployments
* https://etherpad.openstack.org/p/YVR-ops-burning-issues
* https://etherpad.openstack.org/p/YVR-ops-docs
* https://etherpad.openstack.org/p/YVR-ops-tech-choices
* https://etherpad.openstack.org/p/YVR-ops-cmdb
* https://etherpad.openstack.org/p/YVR-ops-data-plane-transitions
* https://etherpad.openstack.org/p/YVR-ops-upgrades
* https://etherpad.openstack.org/p/YVR-ops-packaging
* https://etherpad.openstack.org/p/YVR-ops-nova-network

Manila/MountAutomation

2015-05-12T20:05:53Z

Clinton Knight:

= Intro =
We considered multiple approaches for addressing mount automation. The thinking has come a long way since we originally considered the problem 2+ years ago, but we still don't have a "silver bullet" approach that works in a wide variety of application and has no drawbacks.

= Questions =
We decided to evaluate the various approaches using the follow measurements:
* Is the approach limited to a subset of scenarios?
* Does the approach allow for instantaneous/synchronous mounting?
* Does the approach allow for persistent mounts that survive across reboots?
* Can the approach handle the case when the client is down at the time of attach/detach?
* What security considerations exist?
* What network considerations exist?
* Are there any external dependencies?
* Does the approach require Manila to store any state?
* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?
* How much effort is required to implement the approach?
* How resistant are users likely to be to the approach?
* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

The hardest thing about this feature, aside from the fact that there are no silver bullet solutions, is that the feature doesn't allow users to do anything that wasn't already possible. For the most part, this is about simplifying something which can be done already (with the exception of the DR use case, mentioned above).

= Approaches Summary =
Approaches considered:

* SSH/PsExec (very well understood approach we've considered from the beginning)
* Agent - write a custom agent to solve the problem (also well understood and old idea)
* Zeroconf-based advertisements - a new idea that came from the Paris summit
* AutoFS - leverage autofs to solve the problem on UNIX
* LDAP/AD - insert mount information into the directory service and let clients pull it from there
* Cloud-Init - piggyback on cloudinit (a shippable demo is available for this that will be shown at the Liberty summit)
* VirtFS - virtfs gives us the closest model to cinder, and attaches/detaches make significantly more sense in this context
* Do nothing, and prescribe recipes how to automate mounts with existing tools, such as Heat.

= Approaches =

== SSH/PsExec ==
The simplest and most widely usable approach for automating mounts is to use management software that already exists in the OS. For UNIX, this is SSH. For Windows, it would be the various forms of Windows remote-management such as PsExec (PsExec is an SSH-like tool that works out of the box on nearly all Windows machines) or PowerShell.

The idea here is to give Manila the root password, root key, or Administrator creds for all of your clients, and to let Manila reach out and poke them to trigger them to mount or unmount shares at the appropriate times.

The value of this approach is that it has no dependencies on anything other than standard OS facilities and is therefore likely to be widely compatible.

* Is the approach limited to a subset of scenarios?

good No.

* Does the approach allow for instantaneous/synchronous mounting?

good Yes.

* Does the approach allow for persistent mounts that survive across reboots?

good Yes.

* Can the approach handle the case when the client is down at the time of attach/detach?

bad No.

* What security considerations exist?

bad Tenants have to hand over to Manila administrator credentials for all of the guests for which they want Manila to automate mounts. This can create a huge trust barrier.

* What network considerations exist?

bad Manila will need network access directly to every client for which its automating mounts. This can be easy in some cases and near impossible in other cases.

* Are there any external dependencies?

caution Manila needs an SSH client (easy) and a PsExec client (hard). PsExec is a Windows binary, which could be ported to UNIX, but most likely at considerable cost.

* Does the approach require Manila to store any state?

caution Yes. Manila needs to store credentials for the clients, and also information about IP addresses and networks to find the clients.

* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

good Yes. If Manila can push mount operations at will, then automatically updating mounts in the DR failover case becomes possible.

* How much effort is required to implement the approach?

caution For UNIX-only, medium. For Windows, significantly more if we want to undertake something with PsExec.

* How resistant are users likely to be to the approach?

good It's optional, but some users may choose not to use this feature based on the requirement to hand overs root creds to Manila.

* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

bad Yes. Manila would needs to become aware of clients and their networks. This is currently out of scope for Manila.

== Agent ==
Writing an agent gives us ultimate flexibility. Agents can listen for requests from Manila, they can pull updates from Manila periodically or at boot, and they can perform ANY change we like on the clients, at any time. Many problems becomes simple when you can assume the existence of an agent on the client.

There are obvious downsides to this approach, chiefly the trust issue that most users will have about installing a new piece of software on their guests, and the significant effort involved with creating a new piece of software and maintaining it. There are also security concerns, because we will be responsible if a bug in the agent software exposes a user to mischief and mayhem.

* Is the approach limited to a subset of scenarios?

good No.

* Does the approach allow for instantaneous/synchronous mounting?

good Yes.

* Does the approach allow for persistent mounts that survive across reboots?

good Yes.

* Can the approach handle the case when the client is down at the time of attach/detach?

good Yes.

* What security considerations exist?

bad We get to design our own security. This gives us excellent flexibility, but we have to design something that strikes a good balance between protecting users and being usable, which is always hard. Regardless, we will face trust problems until we have a proven track record.

* What network considerations exist?

caution The network considerations are similar to the SSH approach, because Manila will need to reach out and poke the agent. Also the agent will need a reverse path back to the Manila server to do pull requests, which might be desirable at boot time.

* Are there any external dependencies?

caution The external dependency is the agent itself. We cannot force anyone to install it, so in cases where it doesn't exist, we will be unable to do anything.

* Does the approach require Manila to store any state?

caution Yes. Manila will need to track the agents and some security information in order to contact them and authenticate with them.

* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

good Yes.

* How much effort is required to implement the approach?

bad Very high.

* How resistant are users likely to be to the approach?

bad Very resistent. It's another thing to install and another thing to security audit. Another thing to worry about.

* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

bad Yes. Not only does Manila need to become aware of clients and client networks, but now Manila needs to create a whole new project and maintain that software as well.

== Zeroconf-based advertisements ==
Zeroconf is an interesting approach for advertising services on networks, which layers on top of DNS (or mDNS in peer to peer environments). Zeroconf has existing support for advertisement of CIFS share and NFS exports, but there is no evidence that modern operating systems react to these advertisements automatically by (for example) mounting them.

Zeroconf also only solves half the problem of mount automation. It can notify clients of the existence of something they can mount, but it doesn't tell them precisely WHERE to mount it. For some scenarios, a random mount point might be acceptable (VDI) but for others, it would be a dealbreaker (database as a service).

It's unlikely that we could address unmounting with this approach.

* Is the approach limited to a subset of scenarios?

bad Yes.

* Does the approach allow for instantaneous/synchronous mounting?

caution Maybe.

* Does the approach allow for persistent mounts that survive across reboots?

caution Maybe.

* Can the approach handle the case when the client is down at the time of attach/detach?

good Yes.

* What security considerations exist?

good Possibly none.

* What network considerations exist?

bad Zeroconf relies heavily on multicast, which rarely crosses router boundaries (the admin has to configure multicast routing). In order to get the advertisements onto the right networks, Manila would need to perform some weird gymnastics, most likely involving service VMs with lots and lots of network ports.

* Are there any external dependencies?

bad Yes. Zeroconf is not universally available in guest images or enabled.

* Does the approach require Manila to store any state?

caution Yes. Manila would need to know which clients to advertise to and on which networks.

* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

bad No. It's unlikely that an advertisement-based approach would allow for the fast unmount-move-mount semantics needed to cleanly failover.

* How much effort is required to implement the approach?

bad Unknown, but not small. There are probably existing libraries we can use to do zeroconf-based-advertising, but all of the service VM and networking stuff required to make it work will still be considerable effort.

* How resistant are users likely to be to the approach?

good Very little. The approach is not invasive at all, aside from the possibly scary service VM that attaches to thousands of networks to send advertisements.

* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

bad Yes. Manila needs to know about clients and networks to be able to advertise to them. Also, we need to maintain the software image that goes on the advertising service VM. This would require creating a new GitHub repo and maintaining a new project.

== AutoFS ==
Many UNIX users already use autofs to control large numbers of NFS mounts. The advantage to autofs is that you can make a single change to a file on a server, and suddenly thousands of clients can mount something new.

One downside is that autofs doesn't actually mount stuff immediately. It's an event-based system that relies on just-in-time mounting on the client side, so that if the client tries to access the mount point before it's mounted, it asks the server what is the correct thing to mount and mounts it. This creates real problems with changes to mounts or unmounts, because they basically become impossible without a reboot or some other approach described elsewhere in this document.

The other downside is that autofs is extremely flexible. Different users probably use it in very different ways and a single approach probably won't meet all users' needs without some customization by the user.

* Is the approach limited to a subset of scenarios?

bad Yes. It would be UNIX only, and only work if autofs was in use, and only if autofs was configured in a way that it was compatible with Manila.

* Does the approach allow for instantaneous/synchronous mounting?

bad Not really. The just-in-time approach is instantaneous, except that mount doesn't actually happen until its needed.

* Does the approach allow for persistent mounts that survive across reboots?

good Yes.

* Can the approach handle the case when the client is down at the time of attach/detach?

good Yes.

* What security considerations exist?

caution Manila needs write access to the place where the autofs tables live. This is likely to vary from case to case, but it would be sensitive information and access to it would always be protected.

* What network considerations exist?

good None.

* Are there any external dependencies?

caution Yes, autofs would be needed.

* Does the approach require Manila to store any state?

good Possibly not. Manila could simply push updates to autofs and then forget about them, and the right thing would happen.

* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

bad Probably not. Autofs is not friendly to changes, because once the client has its mount it doesn't go back and pull changes from the server anymore.

* How much effort is required to implement the approach?

good Fairly low. It's just some code that tweaks and autofs table. The effort gets higher if we try to get clever and support lots of ways of using autofs.

* How resistant are users likely to be to the approach?

good Very low. Users like this idea, but they probably haven't thought through all the downsides.

* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

good No. This feature can be implemented on the side without significantly changing Manila itself.

== LDAP/AD ==
Directory services create a centralized place for clients to get information. A list of mounts could be one kind of information stored in a directory service. For Windows, the AD server tells each user where his home directory is, for example. On UNIX, LDAP can be a similar repository, but coupled with autofs for mounting.

We haven't determined 100% whether these tools are flexible enough out of the box, but they provide a reasonable building block that could be combined with some scripts to facilitate mount automation. That being said, this approach has enough drawbacks that we didn't consider it worthwhile to dig deeper.

* Is the approach limited to a subset of scenarios?

bad Yes. Not all environments have LDAP/AD.

* Does the approach allow for instantaneous/synchronous mounting?

bad No.

* Does the approach allow for persistent mounts that survive across reboots?

good Yes.

* Can the approach handle the case when the client is down at the time of attach/detach?

good Yes.

* What security considerations exist?

caution Manila needs write access to the LDAP/AD server. In theory we already have this with the security service that's part of the share network, but not all clients actually use that feature, so it would make setting up security services a prerequisute.

* What network considerations exist?

good None.

* Are there any external dependencies?

caution Yes, the LDAP/AD server. Also software to communicate with those servers from Manila.

* Does the approach require Manila to store any state?

good No.

* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

bad No.

* How much effort is required to implement the approach?

caution Medium. Storing some records in a server is not super hard. The challenge would be getting client software to talk to those services.

* How resistant are users likely to be to the approach?

good Pretty low. It's not invasive and it piggybacks on stuff they already do.

* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

good No, not unless we need to write some new software to communicate with AD or LDAP.

== Cloud-Init ==
Clout-init is widely available in OpenStack clouds (but nowhere near universally available). It runs at startup and pulls identity information from a directory service. We could store our mount information in that directory and make Cloud-Init perform the mounts for us.

This is similar to an agent based approach, except the agent is owned by someone else and only runs at startup time, so listening for requests wouldn't be possible.

As a slight variation, a script inserted by Cloud-Init could get mount information from the Nova metadata service, which would not store Manila data but would query Manila for the latest share mounts on demand.

* Is the approach limited to a subset of scenarios?

caution Yes. Cloud-Init is common, but by no means universal.

* Does the approach allow for instantaneous/synchronous mounting?

bad No.

* Does the approach allow for persistent mounts that survive across reboots?

good Yes.

* Can the approach handle the case when the client is down at the time of attach/detach?

good Yes.

* What security considerations exist?

caution Manila needs access to modify the directory that Cloud-Init pulls from, or the service invoked by Cloud-Init (i.e. Nova metadata service) needs Manila credentials). However, this is a single trust relationship which should be easy to configure.

* What network considerations exist?

good None.

* Are there any external dependencies?

caution Dependency on Cloud-Init.

* Does the approach require Manila to store any state?

good No.

* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

good No (unless clients were rebooted as part of the same DR failover).

* How much effort is required to implement the approach?

good Fairly low. We prototyped this (Nova metadata service + Cloud-Init) in a week.

* How resistant are users likely to be to the approach?

good Fairly low. Users that like Cloud-Init will find this to be a natural expansion.

* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

good No.

== VirtFS ==
VirtFS is hypervisor controlled and extremely similar to the Cinder/block-based approach insofar as the guest simply sees a PCI PnP event and reacts to it. Manila would only need to communicate with Nova and we could safely ignore the guest.

The downsides to VirtFS are well known -- chiefly the lack of support in Windows guests, lack of support in certain hypervisors, and the lack of support for bare metal.

* Is the approach limited to a subset of scenarios?

bad Yes, only situations where VirtFS is used.

* Does the approach allow for instantaneous/synchronous mounting?

good Yes.

* Does the approach allow for persistent mounts that survive across reboots?

good Yes.

* Can the approach handle the case when the client is down at the time of attach/detach?

good Undefined. Like Cinder, there is no concept of a volume attached to an instance that's not currently running.

* What security considerations exist?

good None.

* What network considerations exist?

good None.

* Are there any external dependencies?

bad VirtFS. And something to manage the virtfs/hypervisor setup -- Nova would be the obvious thing to do this.

* Does the approach require Manila to store any state?

good No.

* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

good Yes. In fact this approach is by far the best approach for making DR seamless.

* How much effort is required to implement the approach?

good Low. In this scenario we really don't do much of anything in Manila because all of the VirtFS work would be in Nova anyways.

* How resistant are users likely to be to the approach?

caution Mixed. Some love virtfs, others hate it. For cases where it's possible, it would be a very attractive. For cases where it's not possible, who cares?

* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

good No.

== Do Nothing ==
The idea here is to recognize that mount automation is a non-feature. Nothing we add in Manila will enable anything you can't already do manually, and there is little point to making Manila try to simplify a task which can already be simplified more effectively using existing tools.

Consider a workflow that uses Heat to create a Manila share and some Nova instances to provide a service of some kind. Heat has all the of information it needs to perform the mount itself, and Heat is more than capable of automating these kinds of things. Customers may prefer to do this themselves, because they'll get extra flexibility and won't be locked into a narrow approach implemented inside Manila.

If we were to pursue this approach, we would focus on providing concrete examples to users for how to leverage tools like Heat to automate mounts, and we would add these examples to the documentation.

There are no obvious downsides to this approach, other than that it feels like we're sidestepping the problem.

* Is the approach limited to a subset of scenarios?

good No.

* Does the approach allow for instantaneous/synchronous mounting?

good Yes.

* Does the approach allow for persistent mounts that survive across reboots?

good Yes.

* Can the approach handle the case when the client is down at the time of attach/detach?

good Yes.

* What security considerations exist?

good None in Manila. Obviously the tool (like Heat) doing the automation would need authentication creds, but such tools need those anyways, so they are in many ways best suited to solve the security problems.

* What network considerations exist?

good None in Manila. Obviously the tool (like Heat) doing the automation would need network access to the client, but such tools need that anyways, so they are in many ways best suited to deal with network complexity.

* Are there any external dependencies?

good No real dependencies. You need to bring your own automation and the dependencies are whatever you bring.

* Does the approach require Manila to store any state?

good No.

* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

bad No, not unless the DR failover is also automated by the same process automating the mounts.

* How much effort is required to implement the approach?

good Very little. This is largely a documentation exercise.

* How resistant are users likely to be to the approach?

good Not resistant at all, since this choice gives them maximum flexibility. Some users may be disappointed that there is no magic button, but that was fantasy anyways.

* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

good No.

= Plugin Approach =

Based on the mixed advantages and disadvantages of the above approaches, and based on the likelihood that other people will have other ideas for how to tackle this problem, we recommend tackling this problem by building a "plugin" framework in Manila that enables mount automation with a variety of approaches.

We will assume that all of the above approaches will eventually be implemented by various people in the community, and the goal of the core feature is to provide the basic primitives to allow all of the approaches to work. Which approaches get used in practice will depend on choices made by individual deployers or tenants.

The plugin approach will need to provide a mechanism for "attachments" between clients and shares to be tracked and for the software responsible for performing mounts and unmounts to query that mechanism and to be notified of changes.

Some questions to be answered by the design:
* What are the types of things that shares can be attached to? IPs? Networks? Users? Nova Instances?
* When does a share become "attached"? When access is granted? When a separate "attach" API is called?
* Who is the consumer of the attachment information/notifications? The administrator? The tenants? Both?

The plugin design should include enough detail that we can be confident that ALL of the proposed approaches could be accommodated, however we won't actually design all of the above approaches. We will start with designs for 1 or 2 highly popular approaches (to prove the approach and make it actually usable) and encourage the community to expand on the others.

Assumed highly popular approaches:
* Cloud-Init w/ Nova metadata
* Nova agent

Manila/MountAutomation

2015-05-12T19:54:53Z

Clinton Knight:

= Intro =
We considered multiple approaches for addressing mount automation. The thinking has come a long way since we originally considered the problem 2+ years ago, but we still don't have a "silver bullet" approach that works in a wide variety of application and has no drawbacks.

== Questions ==
We decided to evaluate the various approaches using the follow measurements:
* Is the approach limited to a subset of scenarios?
* Does the approach allow for instantaneous/synchronous mounting?
* Does the approach allow for persistent mounts that survive across reboots?
* Can the approach handle the case when the client is down at the time of attach/detach?
* What security considerations exist?
* What network considerations exist?
* Are there any external dependencies?
* Does the approach require Manila to store any state?
* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?
* How much effort is required to implement the approach?
* How resistant are users likely to be to the approach?
* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

The hardest thing about this feature, aside from the fact that there are no silver bullet solutions, is that the feature doesn't allow users to do anything that wasn't already possible. For the most part, this is about simplifying something which can be done already (with the exception of the DR use case, mentioned above).

== Approaches Summary ==
Approaches considered:

* SSH/PsExec (very well understood approach we've considered from the beginning)
* Agent - write a custom agent to solve the problem (also well understood and old idea)
* Zeroconf-based advertisements - a new idea that came from the Paris summit
* AutoFS - leverage autofs to solve the problem on UNIX
* LDAP/AD - insert mount information into the directory service and let clients pull it from there
* Cloud-Init - piggyback on cloudinit (a shippable demo is available for this that will be shown at the Liberty summit)
* VirtFS - virtfs gives us the closest model to cinder, and attaches/detaches make significantly more sense in this context
* Do nothing, and prescribe recipes how to automate mounts with existing tools, such as Heat.

== Approaches ==

=== SSH/PsExec ===
The simplest and most widely usable approach for automating mounts is to use management software that already exists in the OS. For UNIX, this is SSH. For Windows, it would be the various forms of Windows remote-management such as PsExec (PsExec is an SSH-like tool that works out of the box on nearly all Windows machines) or PowerShell.

The idea here is to give Manila the root password, root key, or Administrator creds for all of your clients, and to let Manila reach out and poke them to trigger them to mount or unmount shares at the appropriate times.

The value of this approach is that it has no dependencies on anything other than standard OS facilities and is therefore likely to be widely compatible.

* Is the approach limited to a subset of scenarios?

good No.

* Does the approach allow for instantaneous/synchronous mounting?

good Yes.

* Does the approach allow for persistent mounts that survive across reboots?

good Yes.

* Can the approach handle the case when the client is down at the time of attach/detach?

bad No.

* What security considerations exist?

bad Tenants have to hand over to Manila administrator credentials for all of the guests for which they want Manila to automate mounts. This can create a huge trust barrier.

* What network considerations exist?

bad Manila will need network access directly to every client for which its automating mounts. This can be easy in some cases and near impossible in other cases.

* Are there any external dependencies?

caution Manila needs an SSH client (easy) and a PsExec client (hard). PsExec is a Windows binary, which could be ported to UNIX, but most likely at considerable cost.

* Does the approach require Manila to store any state?

caution Yes. Manila needs to store credentials for the clients, and also information about IP addresses and networks to find the clients.

* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

good Yes. If Manila can push mount operations at will, then automatically updating mounts in the DR failover case becomes possible.

* How much effort is required to implement the approach?

caution For UNIX-only, medium. For Windows, significantly more if we want to undertake something with PsExec.

* How resistant are users likely to be to the approach?

good It's optional, but some users may choose not to use this feature based on the requirement to hand overs root creds to Manila.

* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

bad Yes. Manila would needs to become aware of clients and their networks. This is currently out of scope for Manila.

=== Agent ===
Writing an agent gives us ultimate flexibility. Agents can listen for requests from Manila, they can pull updates from Manila periodically or at boot, and they can perform ANY change we like on the clients, at any time. Many problems becomes simple when you can assume the existence of an agent on the client.

There are obvious downsides to this approach, chiefly the trust issue that most users will have about installing a new piece of software on their guests, and the significant effort involved with creating a new piece of software and maintaining it. There are also security concerns, because we will be responsible if a bug in the agent software exposes a user to mischief and mayhem.

* Is the approach limited to a subset of scenarios?

good No.

* Does the approach allow for instantaneous/synchronous mounting?

good Yes.

* Does the approach allow for persistent mounts that survive across reboots?

good Yes.

* Can the approach handle the case when the client is down at the time of attach/detach?

good Yes.

* What security considerations exist?

bad We get to design our own security. This gives us excellent flexibility, but we have to design something that strikes a good balance between protecting users and being usable, which is always hard. Regardless, we will face trust problems until we have a proven track record.

* What network considerations exist?

caution The network considerations are similar to the SSH approach, because Manila will need to reach out and poke the agent. Also the agent will need a reverse path back to the Manila server to do pull requests, which might be desirable at boot time.

* Are there any external dependencies?

caution The external dependency is the agent itself. We cannot force anyone to install it, so in cases where it doesn't exist, we will be unable to do anything.

* Does the approach require Manila to store any state?

caution Yes. Manila will need to track the agents and some security information in order to contact them and authenticate with them.

* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

good Yes.

* How much effort is required to implement the approach?

bad Very high.

* How resistant are users likely to be to the approach?

bad Very resistent. It's another thing to install and another thing to security audit. Another thing to worry about.

* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

bad Yes. Not only does Manila need to become aware of clients and client networks, but now Manila needs to create a whole new project and maintain that software as well.

=== Zeroconf-based advertisements ===
Zeroconf is an interesting approach for advertising services on networks, which layers on top of DNS (or mDNS in peer to peer environments). Zeroconf has existing support for advertisement of CIFS share and NFS exports, but there is no evidence that modern operating systems react to these advertisements automatically by (for example) mounting them.

Zeroconf also only solves half the problem of mount automation. It can notify clients of the existence of something they can mount, but it doesn't tell them precisely WHERE to mount it. For some scenarios, a random mount point might be acceptable (VDI) but for others, it would be a dealbreaker (database as a service).

It's unlikely that we could address unmounting with this approach.

* Is the approach limited to a subset of scenarios?

bad Yes.

* Does the approach allow for instantaneous/synchronous mounting?

caution Maybe.

* Does the approach allow for persistent mounts that survive across reboots?

caution Maybe.

* Can the approach handle the case when the client is down at the time of attach/detach?

good Yes.

* What security considerations exist?

good Possibly none.

* What network considerations exist?

bad Zeroconf relies heavily on multicast, which rarely crosses router boundaries (the admin has to configure multicast routing). In order to get the advertisements onto the right networks, Manila would need to perform some weird gymnastics, most likely involving service VMs with lots and lots of network ports.

* Are there any external dependencies?

bad Yes. Zeroconf is not universally available in guest images or enabled.

* Does the approach require Manila to store any state?

caution Yes. Manila would need to know which clients to advertise to and on which networks.

* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

bad No. It's unlikely that an advertisement-based approach would allow for the fast unmount-move-mount semantics needed to cleanly failover.

* How much effort is required to implement the approach?

bad Unknown, but not small. There are probably existing libraries we can use to do zeroconf-based-advertising, but all of the service VM and networking stuff required to make it work will still be considerable effort.

* How resistant are users likely to be to the approach?

good Very little. The approach is not invasive at all, aside from the possibly scary service VM that attaches to thousands of networks to send advertisements.

* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

bad Yes. Manila needs to know about clients and networks to be able to advertise to them. Also, we need to maintain the software image that goes on the advertising service VM. This would require creating a new GitHub repo and maintaining a new project.

=== AutoFS ===
Many UNIX users already use autofs to control large numbers of NFS mounts. The advantage to autofs is that you can make a single change to a file on a server, and suddenly thousands of clients can mount something new.

One downside is that autofs doesn't actually mount stuff immediately. It's an event-based system that relies on just-in-time mounting on the client side, so that if the client tries to access the mount point before it's mounted, it asks the server what is the correct thing to mount and mounts it. This creates real problems with changes to mounts or unmounts, because they basically become impossible without a reboot or some other approach described elsewhere in this document.

The other downside is that autofs is extremely flexible. Different users probably use it in very different ways and a single approach probably won't meet all users' needs without some customization by the user.

* Is the approach limited to a subset of scenarios?

bad Yes. It would be UNIX only, and only work if autofs was in use, and only if autofs was configured in a way that it was compatible with Manila.

* Does the approach allow for instantaneous/synchronous mounting?

bad Not really. The just-in-time approach is instantaneous, except that mount doesn't actually happen until its needed.

* Does the approach allow for persistent mounts that survive across reboots?

good Yes.

* Can the approach handle the case when the client is down at the time of attach/detach?

good Yes.

* What security considerations exist?

caution Manila needs write access to the place where the autofs tables live. This is likely to vary from case to case, but it would be sensitive information and access to it would always be protected.

* What network considerations exist?

good None.

* Are there any external dependencies?

caution Yes, autofs would be needed.

* Does the approach require Manila to store any state?

good Possibly not. Manila could simply push updates to autofs and then forget about them, and the right thing would happen.

* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

bad Probably not. Autofs is not friendly to changes, because once the client has its mount it doesn't go back and pull changes from the server anymore.

* How much effort is required to implement the approach?

good Fairly low. It's just some code that tweaks and autofs table. The effort gets higher if we try to get clever and support lots of ways of using autofs.

* How resistant are users likely to be to the approach?

good Very low. Users like this idea, but they probably haven't thought through all the downsides.

* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

good No. This feature can be implemented on the side without significantly changing Manila itself.

=== LDAP/AD ===
Directory services create a centralized place for clients to get information. A list of mounts could be one kind of information stored in a directory service. For Windows, the AD server tells each user where his home directory is, for example. On UNIX, LDAP can be a similar repository, but coupled with autofs for mounting.

We haven't determined 100% whether these tools are flexible enough out of the box, but they provide a reasonable building block that could be combined with some scripts to facilitate mount automation. That being said, this approach has enough drawbacks that we didn't consider it worthwhile to dig deeper.

* Is the approach limited to a subset of scenarios?

bad Yes. Not all environments have LDAP/AD.

* Does the approach allow for instantaneous/synchronous mounting?

bad No.

* Does the approach allow for persistent mounts that survive across reboots?

good Yes.

* Can the approach handle the case when the client is down at the time of attach/detach?

good Yes.

* What security considerations exist?

caution Manila needs write access to the LDAP/AD server. In theory we already have this with the security service that's part of the share network, but not all clients actually use that feature, so it would make setting up security services a prerequisute.

* What network considerations exist?

good None.

* Are there any external dependencies?

caution Yes, the LDAP/AD server. Also software to communicate with those servers from Manila.

* Does the approach require Manila to store any state?

good No.

* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

bad No.

* How much effort is required to implement the approach?

caution Medium. Storing some records in a server is not super hard. The challenge would be getting client software to talk to those services.

* How resistant are users likely to be to the approach?

good Pretty low. It's not invasive and it piggybacks on stuff they already do.

* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

good No, not unless we need to write some new software to communicate with AD or LDAP.

=== Cloud-Init ===
Clout-init is widely available in OpenStack clouds (but nowhere near universally available). It runs at startup and pulls identity information from a directory service. We could store our mount information in that directory and make Cloud-Init perform the mounts for us.

This is similar to an agent based approach, except the agent is owned by someone else and only runs at startup time, so listening for requests wouldn't be possible.

As a slight variation, a script inserted by Cloud-Init could get mount information from the Nova metadata service, which would not store Manila data but would query Manila for the latest share mounts on demand.

* Is the approach limited to a subset of scenarios?

caution Yes. Cloud-Init is common, but by no means universal.

* Does the approach allow for instantaneous/synchronous mounting?

bad No.

* Does the approach allow for persistent mounts that survive across reboots?

good Yes.

* Can the approach handle the case when the client is down at the time of attach/detach?

good Yes.

* What security considerations exist?

caution Manila needs access to modify the directory that Cloud-Init pulls from, or the service invoked by Cloud-Init (i.e. Nova metadata service) needs Manila credentials). However, this is a single trust relationship which should be easy to configure.

* What network considerations exist?

good None.

* Are there any external dependencies?

caution Dependency on Cloud-Init.

* Does the approach require Manila to store any state?

good No.

* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

good No (unless clients were rebooted as part of the same DR failover).

* How much effort is required to implement the approach?

good Fairly low. We prototyped this (Nova metadata service + Cloud-Init) in a week.

* How resistant are users likely to be to the approach?

good Fairly low. Users that like Cloud-Init will find this to be a natural expansion.

* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

good No.

=== VirtFS ===
VirtFS is hypervisor controlled and extremely similar to the Cinder/block-based approach insofar as the guest simply sees a PCI PnP event and reacts to it. Manila would only need to communicate with Nova and we could safely ignore the guest.

The downsides to VirtFS are well known -- chiefly the lack of support in Windows guests, lack of support in certain hypervisors, and the lack of support for bare metal.

* Is the approach limited to a subset of scenarios?

bad Yes, only situations where VirtFS is used.

* Does the approach allow for instantaneous/synchronous mounting?

good Yes.

* Does the approach allow for persistent mounts that survive across reboots?

good Yes.

* Can the approach handle the case when the client is down at the time of attach/detach?

good Undefined. Like Cinder, there is no concept of a volume attached to an instance that's not currently running.

* What security considerations exist?

good None.

* What network considerations exist?

good None.

* Are there any external dependencies?

bad VirtFS. And something to manage the virtfs/hypervisor setup -- Nova would be the obvious thing to do this.

* Does the approach require Manila to store any state?

good No.

* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

good Yes. In fact this approach is by far the best approach for making DR seamless.

* How much effort is required to implement the approach?

good Low. In this scenario we really don't do much of anything in Manila because all of the VirtFS work would be in Nova anyways.

* How resistant are users likely to be to the approach?

caution Mixed. Some love virtfs, others hate it. For cases where it's possible, it would be a very attractive. For cases where it's not possible, who cares?

* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

good No.

=== Do Nothing ===
The idea here is to recognize that mount automation is a non-feature. Nothing we add in Manila will enable anything you can't already do manually, and there is little point to making Manila try to simplify a task which can already be simplified more effectively using existing tools.

Consider a workflow that uses Heat to create a Manila share and some Nova instances to provide a service of some kind. Heat has all the of information it needs to perform the mount itself, and Heat is more than capable of automating these kinds of things. Customers may prefer to do this themselves, because they'll get extra flexibility and won't be locked into a narrow approach implemented inside Manila.

If we were to pursue this approach, we would focus on providing concrete examples to users for how to leverage tools like Heat to automate mounts, and we would add these examples to the documentation.

There are no obvious downsides to this approach, other than that it feels like we're sidestepping the problem.

* Is the approach limited to a subset of scenarios?

good No.

* Does the approach allow for instantaneous/synchronous mounting?

good Yes.

* Does the approach allow for persistent mounts that survive across reboots?

good Yes.

* Can the approach handle the case when the client is down at the time of attach/detach?

good Yes.

* What security considerations exist?

good None in Manila. Obviously the tool (like Heat) doing the automation would need authentication creds, but such tools need those anyways, so they are in many ways best suited to solve the security problems.

* What network considerations exist?

good None in Manila. Obviously the tool (like Heat) doing the automation would need network access to the client, but such tools need that anyways, so they are in many ways best suited to deal with network complexity.

* Are there any external dependencies?

good No real dependencies. You need to bring your own automation and the dependencies are whatever you bring.

* Does the approach require Manila to store any state?

good No.

* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

bad No, not unless the DR failover is also automated by the same process automating the mounts.

* How much effort is required to implement the approach?

good Very little. This is largely a documentation exercise.

* How resistant are users likely to be to the approach?

good Not resistant at all, since this choice gives them maximum flexibility. Some users may be disappointed that there is no magic button, but that was fantasy anyways.

* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

good No.

== Plugin Approach ==

Based on the mixed advantages and disadvantages of the above approaches, and based on the likelihood that other people will have other ideas for how to tackle this problem, we recommend tackling this problem by building a "plugin" framework in Manila that enables mount automation with a variety of approaches.

We will assume that all of the above approaches will eventually be implemented by various people in the community, and the goal of the core feature is to provide the basic primitives to allow all of the approaches to work. Which approaches get used in practice will depend on choices made by individual deployers or tenants.

The plugin approach will need to provide a mechanism for "attachments" between clients and shares to be tracked and for the software responsible for performing mounts and unmounts to query that mechanism and to be notified of changes.

Some questions to be answered by the design:
* What are the types of things that shares can be attached to? IPs? Networks? Users? Nova Instances?
* When does a share become "attached"? When access is granted? When a separate "attach" API is called?
* Who is the consumer of the attachment information/notifications? The administrator? The tenants? Both?

The plugin design should include enough detail that we can be confident that ALL of the proposed approaches could be accommodated, however we won't actually design all of the above approaches. We will start with designs for 1 or 2 highly popular approaches (to prove the approach and make it actually usable) and encourage the community to expand on the others.

Assumed highly popular approaches:
* Cloud-Init w/ Nova metadata
* Nova agent

Manila/MountAutomation

2015-05-12T19:50:33Z

Clinton Knight: Created page with "= Intro = We considered multiple approaches for addressing mount automation. The thinking has come a long way since we originally considered the problem 2+ years ago, but we s..."

= Intro =
We considered multiple approaches for addressing mount automation. The thinking has come a long way since we originally considered the problem 2+ years ago, but we still don't have a "silver bullet" approach that works in a wide variety of application and has no drawbacks.

=== Questions ===
We decided to evaluate the various approaches using the follow measurements:
* Is the approach limited to a subset of scenarios?
* Does the approach allow for instantaneous/synchronous mounting?
* Does the approach allow for persistent mounts that survive across reboots?
* Can the approach handle the case when the client is down at the time of attach/detach?
* What security considerations exist?
* What network considerations exist?
* Are there any external dependencies?
* Does the approach require Manila to store any state?
* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?
* How much effort is required to implement the approach?
* How resistant are users likely to be to the approach?
* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

The hardest thing about this feature, aside from the fact that there are no silver bullet solutions, is that the feature doesn't allow users to do anything that wasn't already possible. For the most part, this is about simplifying something which can be done already (with the exception of the DR use case, mentioned above).

=== Approaches Summary ===
Approaches considered:

* SSH/PsExec (very well understood approach we've considered from the beginning)
* Agent - write a custom agent to solve the problem (also well understood and old idea)
* Zeroconf-based advertisements - a new idea that came from the Paris summit
* AutoFS - leverage autofs to solve the problem on UNIX
* LDAP/AD - insert mount information into the directory service and let clients pull it from there
* Cloud-Init - piggyback on cloudinit (a shippable demo is available for this that will be shown at the Liberty summit)
* VirtFS - virtfs gives us the closest model to cinder, and attaches/detaches make significantly more sense in this context
* Do nothing, and prescribe recipes how to automate mounts with existing tools, such as Heat.

= Approaches =

== SSH/PsExec ==
The simplest and most widely usable approach for automating mounts is to use management software that already exists in the OS. For UNIX, this is SSH. For Windows, it would be the various forms of Windows remote-management such as PsExec (PsExec is an SSH-like tool that works out of the box on nearly all Windows machines) or PowerShell.

The idea here is to give Manila the root password, root key, or Administrator creds for all of your clients, and to let Manila reach out and poke them to trigger them to mount or unmount shares at the appropriate times.

The value of this approach is that it has no dependencies on anything other than standard OS facilities and is therefore likely to be widely compatible.

* Is the approach limited to a subset of scenarios?

good No.

* Does the approach allow for instantaneous/synchronous mounting?

good Yes.

* Does the approach allow for persistent mounts that survive across reboots?

good Yes.

* Can the approach handle the case when the client is down at the time of attach/detach?

bad No.

* What security considerations exist?

bad Tenants have to hand over to Manila administrator credentials for all of the guests for which they want Manila to automate mounts. This can create a huge trust barrier.

* What network considerations exist?

bad Manila will need network access directly to every client for which its automating mounts. This can be easy in some cases and near impossible in other cases.

* Are there any external dependencies?

caution Manila needs an SSH client (easy) and a PsExec client (hard). PsExec is a Windows binary, which could be ported to UNIX, but most likely at considerable cost.

* Does the approach require Manila to store any state?

caution Yes. Manila needs to store credentials for the clients, and also information about IP addresses and networks to find the clients.

* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

good Yes. If Manila can push mount operations at will, then automatically updating mounts in the DR failover case becomes possible.

* How much effort is required to implement the approach?

caution For UNIX-only, medium. For Windows, significantly more if we want to undertake something with PsExec.

* How resistant are users likely to be to the approach?

good It's optional, but some users may choose not to use this feature based on the requirement to hand overs root creds to Manila.

* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

bad Yes. Manila would needs to become aware of clients and their networks. This is currently out of scope for Manila.

== Agent ==
Writing an agent gives us ultimate flexibility. Agents can listen for requests from Manila, they can pull updates from Manila periodically or at boot, and they can perform ANY change we like on the clients, at any time. Many problems becomes simple when you can assume the existence of an agent on the client.

There are obvious downsides to this approach, chiefly the trust issue that most users will have about installing a new piece of software on their guests, and the significant effort involved with creating a new piece of software and maintaining it. There are also security concerns, because we will be responsible if a bug in the agent software exposes a user to mischief and mayhem.

* Is the approach limited to a subset of scenarios?

good No.

* Does the approach allow for instantaneous/synchronous mounting?

good Yes.

* Does the approach allow for persistent mounts that survive across reboots?

good Yes.

* Can the approach handle the case when the client is down at the time of attach/detach?

good Yes.

* What security considerations exist?

bad We get to design our own security. This gives us excellent flexibility, but we have to design something that strikes a good balance between protecting users and being usable, which is always hard. Regardless, we will face trust problems until we have a proven track record.

* What network considerations exist?

caution The network considerations are similar to the SSH approach, because Manila will need to reach out and poke the agent. Also the agent will need a reverse path back to the Manila server to do pull requests, which might be desirable at boot time.

* Are there any external dependencies?

caution The external dependency is the agent itself. We cannot force anyone to install it, so in cases where it doesn't exist, we will be unable to do anything.

* Does the approach require Manila to store any state?

caution Yes. Manila will need to track the agents and some security information in order to contact them and authenticate with them.

* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

good Yes.

* How much effort is required to implement the approach?

bad Very high.

* How resistant are users likely to be to the approach?

bad Very resistent. It's another thing to install and another thing to security audit. Another thing to worry about.

* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

bad Yes. Not only does Manila need to become aware of clients and client networks, but now Manila needs to create a whole new project and maintain that software as well.

== Zeroconf-based advertisements ==
Zeroconf is an interesting approach for advertising services on networks, which layers on top of DNS (or mDNS in peer to peer environments). Zeroconf has existing support for advertisement of CIFS share and NFS exports, but there is no evidence that modern operating systems react to these advertisements automatically by (for example) mounting them.

Zeroconf also only solves half the problem of mount automation. It can notify clients of the existence of something they can mount, but it doesn't tell them precisely WHERE to mount it. For some scenarios, a random mount point might be acceptable (VDI) but for others, it would be a dealbreaker (database as a service).

It's unlikely that we could address unmounting with this approach.

* Is the approach limited to a subset of scenarios?

bad Yes.

* Does the approach allow for instantaneous/synchronous mounting?

caution Maybe.

* Does the approach allow for persistent mounts that survive across reboots?

caution Maybe.

* Can the approach handle the case when the client is down at the time of attach/detach?

good Yes.

* What security considerations exist?

good Possibly none.

* What network considerations exist?

bad Zeroconf relies heavily on multicast, which rarely crosses router boundaries (the admin has to configure multicast routing). In order to get the advertisements onto the right networks, Manila would need to perform some weird gymnastics, most likely involving service VMs with lots and lots of network ports.

* Are there any external dependencies?

bad Yes. Zeroconf is not universally available in guest images or enabled.

* Does the approach require Manila to store any state?

caution Yes. Manila would need to know which clients to advertise to and on which networks.

* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

bad No. It's unlikely that an advertisement-based approach would allow for the fast unmount-move-mount semantics needed to cleanly failover.

* How much effort is required to implement the approach?

bad Unknown, but not small. There are probably existing libraries we can use to do zeroconf-based-advertising, but all of the service VM and networking stuff required to make it work will still be considerable effort.

* How resistant are users likely to be to the approach?

good Very little. The approach is not invasive at all, aside from the possibly scary service VM that attaches to thousands of networks to send advertisements.

* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

bad Yes. Manila needs to know about clients and networks to be able to advertise to them. Also, we need to maintain the software image that goes on the advertising service VM. This would require creating a new GitHub repo and maintaining a new project.

== AutoFS ==
Many UNIX users already use autofs to control large numbers of NFS mounts. The advantage to autofs is that you can make a single change to a file on a server, and suddenly thousands of clients can mount something new.

One downside is that autofs doesn't actually mount stuff immediately. It's an event-based system that relies on just-in-time mounting on the client side, so that if the client tries to access the mount point before it's mounted, it asks the server what is the correct thing to mount and mounts it. This creates real problems with changes to mounts or unmounts, because they basically become impossible without a reboot or some other approach described elsewhere in this document.

The other downside is that autofs is extremely flexible. Different users probably use it in very different ways and a single approach probably won't meet all users' needs without some customization by the user.

* Is the approach limited to a subset of scenarios?

bad Yes. It would be UNIX only, and only work if autofs was in use, and only if autofs was configured in a way that it was compatible with Manila.

* Does the approach allow for instantaneous/synchronous mounting?

bad Not really. The just-in-time approach is instantaneous, except that mount doesn't actually happen until its needed.

* Does the approach allow for persistent mounts that survive across reboots?

good Yes.

* Can the approach handle the case when the client is down at the time of attach/detach?

good Yes.

* What security considerations exist?

caution Manila needs write access to the place where the autofs tables live. This is likely to vary from case to case, but it would be sensitive information and access to it would always be protected.

* What network considerations exist?

good None.

* Are there any external dependencies?

caution Yes, autofs would be needed.

* Does the approach require Manila to store any state?

good Possibly not. Manila could simply push updates to autofs and then forget about them, and the right thing would happen.

* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

bad Probably not. Autofs is not friendly to changes, because once the client has its mount it doesn't go back and pull changes from the server anymore.

* How much effort is required to implement the approach?

good Fairly low. It's just some code that tweaks and autofs table. The effort gets higher if we try to get clever and support lots of ways of using autofs.

* How resistant are users likely to be to the approach?

good Very low. Users like this idea, but they probably haven't thought through all the downsides.

* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

good No. This feature can be implemented on the side without significantly changing Manila itself.

== LDAP/AD ==
Directory services create a centralized place for clients to get information. A list of mounts could be one kind of information stored in a directory service. For Windows, the AD server tells each user where his home directory is, for example. On UNIX, LDAP can be a similar repository, but coupled with autofs for mounting.

We haven't determined 100% whether these tools are flexible enough out of the box, but they provide a reasonable building block that could be combined with some scripts to facilitate mount automation. That being said, this approach has enough drawbacks that we didn't consider it worthwhile to dig deeper.

* Is the approach limited to a subset of scenarios?

bad Yes. Not all environments have LDAP/AD.

* Does the approach allow for instantaneous/synchronous mounting?

bad No.

* Does the approach allow for persistent mounts that survive across reboots?

good Yes.

* Can the approach handle the case when the client is down at the time of attach/detach?

good Yes.

* What security considerations exist?

caution Manila needs write access to the LDAP/AD server. In theory we already have this with the security service that's part of the share network, but not all clients actually use that feature, so it would make setting up security services a prerequisute.

* What network considerations exist?

good None.

* Are there any external dependencies?

caution Yes, the LDAP/AD server. Also software to communicate with those servers from Manila.

* Does the approach require Manila to store any state?

good No.

* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

bad No.

* How much effort is required to implement the approach?

caution Medium. Storing some records in a server is not super hard. The challenge would be getting client software to talk to those services.

* How resistant are users likely to be to the approach?

good Pretty low. It's not invasive and it piggybacks on stuff they already do.

* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

good No, not unless we need to write some new software to communicate with AD or LDAP.

== Cloud-Init ==
Clout-init is widely available in OpenStack clouds (but nowhere near universally available). It runs at startup and pulls identity information from a directory service. We could store our mount information in that directory and make Cloud-Init perform the mounts for us.

This is similar to an agent based approach, except the agent is owned by someone else and only runs at startup time, so listening for requests wouldn't be possible.

As a slight variation, a script inserted by Cloud-Init could get mount information from the Nova metadata service, which would not store Manila data but would query Manila for the latest share mounts on demand.

* Is the approach limited to a subset of scenarios?

caution Yes. Cloud-Init is common, but by no means universal.

* Does the approach allow for instantaneous/synchronous mounting?

bad No.

* Does the approach allow for persistent mounts that survive across reboots?

good Yes.

* Can the approach handle the case when the client is down at the time of attach/detach?

good Yes.

* What security considerations exist?

caution Manila needs access to modify the directory that Cloud-Init pulls from, or the service invoked by Cloud-Init (i.e. Nova metadata service) needs Manila credentials). However, this is a single trust relationship which should be easy to configure.

* What network considerations exist?

good None.

* Are there any external dependencies?

caution Dependency on Cloud-Init.

* Does the approach require Manila to store any state?

good No.

* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

good No (unless clients were rebooted as part of the same DR failover).

* How much effort is required to implement the approach?

good Fairly low. We prototyped this (Nova metadata service + Cloud-Init) in a week.

* How resistant are users likely to be to the approach?

good Fairly low. Users that like Cloud-Init will find this to be a natural expansion.

* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

good No.

== VirtFS ==
VirtFS is hypervisor controlled and extremely similar to the Cinder/block-based approach insofar as the guest simply sees a PCI PnP event and reacts to it. Manila would only need to communicate with Nova and we could safely ignore the guest.

The downsides to VirtFS are well known -- chiefly the lack of support in Windows guests, lack of support in certain hypervisors, and the lack of support for bare metal.

* Is the approach limited to a subset of scenarios?

bad Yes, only situations where VirtFS is used.

* Does the approach allow for instantaneous/synchronous mounting?

good Yes.

* Does the approach allow for persistent mounts that survive across reboots?

good Yes.

* Can the approach handle the case when the client is down at the time of attach/detach?

good Undefined. Like Cinder, there is no concept of a volume attached to an instance that's not currently running.

* What security considerations exist?

good None.

* What network considerations exist?

good None.

* Are there any external dependencies?

bad VirtFS. And something to manage the virtfs/hypervisor setup -- Nova would be the obvious thing to do this.

* Does the approach require Manila to store any state?

good No.

* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

good Yes. In fact this approach is by far the best approach for making DR seamless.

* How much effort is required to implement the approach?

good Low. In this scenario we really don't do much of anything in Manila because all of the VirtFS work would be in Nova anyways.

* How resistant are users likely to be to the approach?

caution Mixed. Some love virtfs, others hate it. For cases where it's possible, it would be a very attractive. For cases where it's not possible, who cares?

* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

good No.

== Do Nothing ==
The idea here is to recognize that mount automation is a non-feature. Nothing we add in Manila will enable anything you can't already do manually, and there is little point to making Manila try to simplify a task which can already be simplified more effectively using existing tools.

Consider a workflow that uses Heat to create a Manila share and some Nova instances to provide a service of some kind. Heat has all the of information it needs to perform the mount itself, and Heat is more than capable of automating these kinds of things. Customers may prefer to do this themselves, because they'll get extra flexibility and won't be locked into a narrow approach implemented inside Manila.

If we were to pursue this approach, we would focus on providing concrete examples to users for how to leverage tools like Heat to automate mounts, and we would add these examples to the documentation.

There are no obvious downsides to this approach, other than that it feels like we're sidestepping the problem.

* Is the approach limited to a subset of scenarios?

good No.

* Does the approach allow for instantaneous/synchronous mounting?

good Yes.

* Does the approach allow for persistent mounts that survive across reboots?

good Yes.

* Can the approach handle the case when the client is down at the time of attach/detach?

good Yes.

* What security considerations exist?

good None in Manila. Obviously the tool (like Heat) doing the automation would need authentication creds, but such tools need those anyways, so they are in many ways best suited to solve the security problems.

* What network considerations exist?

good None in Manila. Obviously the tool (like Heat) doing the automation would need network access to the client, but such tools need that anyways, so they are in many ways best suited to deal with network complexity.

* Are there any external dependencies?

good No real dependencies. You need to bring your own automation and the dependencies are whatever you bring.

* Does the approach require Manila to store any state?

good No.

* Does the approach allow Manila to enable use cases that would be impossible without mount automation, such as automatically updating mounts in the event of a DR failover?

bad No, not unless the DR failover is also automated by the same process automating the mounts.

* How much effort is required to implement the approach?

good Very little. This is largely a documentation exercise.

* How resistant are users likely to be to the approach?

good Not resistant at all, since this choice gives them maximum flexibility. Some users may be disappointed that there is no magic button, but that was fantasy anyways.

* Does the approach require Manila to expand it's project scope (to manage things that it currently has no knowledge of)?

good No.

= Plugin Approach =

Based on the mixed advantages and disadvantages of the above approaches, and based on the likelihood that other people will have other ideas for how to tackle this problem, we recommend tackling this problem by building a "plugin" framework in Manila that enables mount automation with a variety of approaches.

We will assume that all of the above approaches will eventually be implemented by various people in the community, and the goal of the core feature is to provide the basic primitives to allow all of the approaches to work. Which approaches get used in practice will depend on choices made by individual deployers or tenants.

The plugin approach will need to provide a mechanism for "attachments" between clients and shares to be tracked and for the software responsible for performing mounts and unmounts to query that mechanism and to be notified of changes.

Some questions to be answered by the design:
* What are the types of things that shares can be attached to? IPs? Networks? Users? Nova Instances?
* When does a share become "attached"? When access is granted? When a separate "attach" API is called?
* Who is the consumer of the attachment information/notifications? The administrator? The tenants? Both?

The plugin design should include enough detail that we can be confident that ALL of the proposed approaches could be accommodated, however we won't actually design all of the above approaches. We will start with designs for 1 or 2 highly popular approaches (to prove the approach and make it actually usable) and encourage the community to expand on the others.

Assumed highly popular approaches:
* Cloud-Init w/ Nova metadata
* Nova agent

Design Summit/Liberty/Etherpads

2015-05-12T15:06:23Z

Clinton Knight: /* Manila */

[[Category:Summit]]
[[Category:Liberty]]
[[Category:Etherpad]]

<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
__TOC__
</div>
==Cross-Project workshops==
Tuesday:

* 11:15 - 11:55
** [https://etherpad.openstack.org/p/liberty-cross-project-python3 Moving apps to Python 3]
** [https://etherpad.openstack.org/p/liberty-cross-project-user-notifications Async status updates]
* 12:05 - 12:45
** Improving UX across all projects
** Functional Testing Show & Tell
* 2:00 - 2:40
** OpenStack SDK
** Modern JavaScript
* 2:50 - 3:30
** [https://etherpad.openstack.org/p/liberty-cross-project-in-team-scaling In-team scaling]
** Service Catalog Standardization
* 3:40 - 4:20
** API Working Group
** Towards one Network Stack: Part 1 - Neutron Gaps and Concerns
* 4:40 - 5:20
** [https://etherpad.openstack.org/p/liberty-cross-project-release-models OpenStack release model(s)]
** Unified Policy File
* 5:30 - 6:10
** OpenStack Documentation
** [https://etherpad.openstack.org/p/liberty-cross-project-managing-concurrency Managing concurrency]

==Barbican==
==Ceilometer==
wednesday:
* 0900 - 0940: [https://etherpad.openstack.org/p/ceilo-multi-identity componentisation / multi identity]
* 0950 - 1030: [https://etherpad.openstack.org/p/event_alarm event alarms]
* 1100 - 1140: [https://etherpad.openstack.org/p/liberty-ceilometer-pipeline-config pipeline configuration]
* 1150 - 1230: [https://etherpad.openstack.org/p/ceilo-declarative-notifications declarative notification meters]
* 1440 - 1520: [https://etherpad.openstack.org/p/YVR-ops-ceilometer ops followup]
* 1530 - 1610: [https://etherpad.openstack.org/p/ceilo-multi-identity componentization carry over]

 thursday:
* 0900 - 0940: [https://etherpad.openstack.org/p/liberty-ceilometer-meter-deprecation meter deprecation]
* 0950 - 1030: [https://etherpad.openstack.org/p/liberty-ceilometer-meter-event samples/events integration]
* 1100 - 1140: [https://etherpad.openstack.org/p/liberty-ceilometer-versioned-objects versioned objects]
* 1150 - 1230: ops followup

 friday:
* 0900 - 1200: [https://etherpad.openstack.org/p/liberty-ceilometer-contributors-meetup contributor meetup]

==Cinder==
==Designate==
==Documentation==
==Glance==
==Heat==
=== Wed ===
* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-heat-liberty-convergence-1 (W) Finishing off Convergence phase 1]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-heat-liberty-convergence-2 (W) Convergence phase 2 planning]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-heat-liberty-testing (W) Functional and integration testing, identify gaps and suggest improvements]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-heat-liberty-lifecycle-ops (W) Better support for lifecycle operations on (tripleo) heat stacks (upgrades etc)]
* 13:50 - 14:30: [https://etherpad.openstack.org/p/YVR-heat-liberty-client-usability (F) Heat client usability]
* 14:40 - 15:20: [https://etherpad.openstack.org/p/YVR-heat-liberty-template-format (F) Heat template format improvements]
* 15:30 - 16:10: [https://etherpad.openstack.org/p/YVR-heat-liberty-rest-api (F) REST API design and roadmap]
* 16:30 - 17:10: [https://etherpad.openstack.org/p/YVR-heat-liberty-senlin-autoscaling (F) Senlin Autoscaling Project - Deep Dive]
* 17:20 - 18:00: [https://etherpad.openstack.org/p/YVR-heat-liberty-containers (F) Orchestrating containers with Heat]

=== Thu ===
* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-heat-liberty-alembic (W) Migrating from sqlalchemy-migrations to alembic]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-heat-liberty-versioned-objects (W) Future of Versioned objects in heat]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-heat-liberty-autoscaling (W) Autoscaling improvements]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-heat-liberty-deprecation (W) Our deprecation process]
* 16:10 - 16:50: [https://etherpad.openstack.org/p/YVR-heat-liberty-contrib-plugins (W) Moving most contrib plugins in-tree]
* 17:00 - 17:40: [https://etherpad.openstack.org/p/YVR-heat-liberty-resource-registering (W) Conditional registering of resources]

=== Fri ===
* 09:00-12:20: [https://etherpad.openstack.org/p/YVR-heat-contributor-meetup Contributor Meetup]

==Horizon==
==Infrastructure==
==Ironic==
==Keystone==
==Manila==
=== Wed ===
* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-manila-liberty-mount-automation (F) Mount automation]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-manila-liberty-share-migration (F) Share migration]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-manila-liberty-share-replication (F) Share replication]
* 16:30 - 17:10: [https://etherpad.openstack.org/p/YVR-manila-liberty-versioned-objects-and-apis (W) Versioned objects and APIs]
* 17:20 - 18:00: [https://etherpad.openstack.org/p/YVR-manila-liberty-fault-indication (W) Fault indication]

=== Thu ===
* 16:10 - 16:50: [https://etherpad.openstack.org/p/YVR-manila-liberty-thin-provisioning-consistency-groups (W) Thin provisioning and consistency groups]
* 17:00 - 17:40: [https://etherpad.openstack.org/p/YVR-manila-liberty-snapshot-share-relationship-manila-service-image (W) Snapshot-share relationship and Manila service image]

==Neutron==
=== Tue ===
=== Wed ===
* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-neutron-liberty-development Neutron Liberty Development]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-neutron-use-case-discussion Neutron Use Case Discussion]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-neutron-vlan-trunk VLAN Aware VMs]
* 11:50 - 12:30: Neutron Lightning Talks (no etherpad)
* 13:50 - 14:30: [https://etherpad.openstack.org/p/YVR-neutron-octavia Octavia]
* 14:40 - 15:20: [https://etherpad.openstack.org/p/YVR-neutron-opendaylight OpenDaylight]
* 15:30 - 16:10: [https://etherpad.openstack.org/p/YVR-neutron-ovn OVN]
* 16:30 - 17:10: [https://etherpad.openstack.org/p/YVR-neutron-ironic Ironic and Neutron integration]
* 17:20 - 18:00: [https://etherpad.openstack.org/p/YVR-neutron-get-me-a-network Get Me a Network!]

=== Thu ===
* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-neutron-lbaas-use-cases Neutron LBaaS Use Cases]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-neutron-l3 Neutron L3]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-neutron-qos QoS]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-neutron-third-party-ci-liberty Third Party CI in Liberty and Beyond]
* 13:30 - 14:10: [https://etherpad.openstack.org/p/YVR-neutron-testing-in-liberty Testing In Liberty]
* 14:20 - 15:00: [https://etherpad.openstack.org/p/YVR-neutron-RBAC Neutron RBAC]
* 15:10 - 15:50: [https://etherpad.openstack.org/p/YVR-neutron-sg-fwaas-future-direction SG and FWaaS Future Directions]
* 16:10 - 16:50: [https://etherpad.openstack.org/p/YVR-neutron-nfv-enhancements Neutron NFV Enhancements]
=== Fri ===
* 09:00-12:20: [https://etherpad.openstack.org/p/YVR-neutron-contributor-meetup Contributor Meetup]

==Nova==

'''Wednesday:'''

* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-nova-scheduler-in-liberty Scheduler in Liberty]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-nova-scalling-out-scheduler-for-cells Scaling out scheduler for cells ]

* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-nova-cells-v2 Cells v2]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-nova-resource-tracker Resource Tracker, Clustered Hypervisors and NFV]

* 13:50 - 14:30: [https://etherpad.openstack.org/p/YVR-nova-spec-blueprint-unconference Nova Spec/Blueprint Unconference]
* 14:40 - 15:20: [https://etherpad.openstack.org/p/YVR-nova-database-internals Database (part 1)]
* 15:30 - 16:10: [https://etherpad.openstack.org/p/YVR-nova-functional-testing-feature-classification Functional Testing and Feature Classification]

* 16:30 - 17:10: [https://etherpad.openstack.org/p/YVR-nova-instance-ha-evacuate-resize Dealing with compute host failure: Instance HA, Evacuate, Resize]
* 17:20 - 18:00: [https://etherpad.openstack.org/p/YVR-nova-api-2.1-in-liberty Nova API v2.1 in Liberty]

 
'''Thursday:'''

* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-nova-api-2.0-3rd-party Future of Nova API v2.0 and 3rd Party APIs]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-nova-quotas-and-database Quotas and Database (part 2)]

* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-nova-flavors-and-image-properties Flavors and Image Properties]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-nova-error-handling Error Handling]

* 13:30 - 14:10: [https://etherpad.openstack.org/p/YVR-nova-spec-blueprint-unconference Nova Spec/Blueprint Unconference]
* 14:20 - 15:00: [https://etherpad.openstack.org/p/YVR-nova-liberty-priorities Liberty Priorities (part 1)]
* 15:10 - 15:50: [https://etherpad.openstack.org/p/YVR-nova-liberty-priorities Liberty Priorities (part 2)]

* 16:10 - 16:50: [https://etherpad.openstack.org/p/YVR-nova-liberty-process Liberty Process and Scaling out Reviews]
* 17:00 - 17:40: [https://etherpad.openstack.org/p/YVR-nova-network Future of Nova's networking and nova-network]

 
'''Friday:'''

* 09:00 - 12:20 and 13:20 - 16:40: [https://etherpad.openstack.org/p/YVR-nova-contributor-meetup Nova Contributor Meetup]

==Oslo==

'''Wednesday:'''

* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-oslo-versioned-objects-intro (F) Get to know your objects and learn how to version them (an introduction to oslo.versionedobjects)!]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-oslo-taskflow-plans (F) Give me liberty, or give me taskflow (come learn about taskflow liberty plans)!]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-oslo-rootwrap-plans (W) Give me liberty, or give me wraps (the future of oslo.rootwrap) ]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-oslo-functional-testing (W) The cost of liberty is less than the price of functional testing]
* 13:50 - 14:30: [https://etherpad.openstack.org/p/YVR-oslo-graduation-schedule (W) Give me more oslo in liberty or else! Schedule & new libraries]
* 14:40 - 15:20: [https://etherpad.openstack.org/p/YVR-oslo-messaging-zmq-status (F) State of zmq in oslo.messaging]
* 15:30 - 16:10: [https://etherpad.openstack.org/p/YVR-oslo-optional-dependencies (W) Emancipate/liberate your optional dependencies - Optional Dependencies]
* 16:30 - 17:10: [https://etherpad.openstack.org/p/YVR-oslo-release-process-review (W) Reviewing our release processes]
* 17:20 - 18:00: [https://etherpad.openstack.org/p/YVR-oslo-config-filter (W) Configuration Filters in oslo.config]

 
'''Thursday:'''

* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-oslo-strategy-discussion (F) The oslo liberty proclamation (and associated strategy discussion)]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-oslo-config-plans (F) Enfranchise oslo.config, let's discuss alternative data sources in oslo.config]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-oslo-log-plans (W) Life, Liberty, and the pursuit of oslo.log changes]
* 13:30 - 14:10: [https://etherpad.openstack.org/p/YVR-oslo-versioned-objects-deep-dive (W) Deep dive on oslo.versionedobjects, bring your wet suits.]
* 14:20 - 15:00: [https://etherpad.openstack.org/p/YVR-oslo-db-plans (F) For a people who are free, and who mean to remain so, a well organized and armed 'oslo.db' is their best security]
* 15:10 - 15:50: [https://etherpad.openstack.org/p/YVR-oslo-asyncio (F) Event loops, coroutines, yield from, futures, a discussion on asyncio (and triollus?)]
* 16:10 - 16:50: [https://etherpad.openstack.org/p/YVR-oslo-messaging-plans (F) Ping pong, oslo.messaging plans for liberty.]
* 17:00 - 17:40: [https://etherpad.openstack.org/p/YVR-oslo-tech-debt-deprecation (F) How to clean up your tech-debt; let's discuss best practices on how to deprecate things in oslo libraries]

 

==QA==

=== Wed. ===

* 1150-1230 - [https://etherpad.openstack.org/p/YVR-QA-testing-beyond-the-gate Testing beyond the gate]
* 1350-1430 - [https://etherpad.openstack.org/p/YVR-QA-Devstack-Roadmap Devstack Roadmap]
* 1440-1520 - [https://etherpad.openstack.org/p/YVR-QA-Tempest-service-clients Work Session: Tempest service clients]
* 1630-1710 - [https://etherpad.openstack.org/p/YVR-QA-in-the-big-tent QA in the Big Tent]

=== Thurs. ===
* 0900-0940 - [https://etherpad.openstack.org/p/YVR-QA-idempotent-id Work Session: Idempotent ID]
* 1330-1410 - [https://etherpad.openstack.org/p/YVR-QA-Tempest-as-a-system-program Work Session: Tempest as System Program]
* 1420-1500 - [https://etherpad.openstack.org/p/YVR-QA-Tempest-CLI Work Session: Tempest CLI and Tempest Cleanup]
* 1610-1650 - [https://etherpad.openstack.org/p/YVR-QA-Tempest-Scope-Revisited Tempest Scope Revisited]
* 1700-1740 - [https://etherpad.openstack.org/p/YVR-QA-Liberty-Priorities Liberty Priorities]

==Release Management==
* Wed 17:20-18:00: [https://etherpad.openstack.org/p/YVR-relmgt-stable-branch Stable branch]
* Thu 11:00-11:40 [https://etherpad.openstack.org/p/YVR-relmgt-library-releases Work session: Library releases]

==Sahara==

=== Wed ===

* 11:00am (W) HA http://etherpad.openstack.org/p/sahara-liberty-ha
* 11:50am (W) API v2 https://etherpad.openstack.org/p/sahara-liberty-api-v2

=== Thu ===

* 1:30pm (F) UI https://etherpad.openstack.org/p/sahara-liberty-ui
* 2:20pm (F) Security https://etherpad.openstack.org/p/sahara-liberty-security
* 3:10pm (W) EDP https://etherpad.openstack.org/p/sahara-liberty-edp
* 4:10pm (W) Resources Visibility/ACL https://etherpad.openstack.org/p/sahara-liberty-resources
* 5:00pm (W) Plugins https://etherpad.openstack.org/p/sahara-liberty-plugins

=== Fri ===

* 9am - 4:40pm Meetup https://etherpad.openstack.org/p/sahara-liberty-meetup

==Swift==
==TripleO==

==Trove==
==Zaqar==
==Event intro/closure==
* Tue 11:15-11:55: [https://etherpad.openstack.org/p/YVR-design-summit-101 Design Summit 101]
* Fri 16:50-17:30: [https://etherpad.openstack.org/p/YVR-design-summit-feedback Design Summit feedback session]

==Ops==
Tuesday:
* https://etherpad.openstack.org/p/YVR-ops-101
* https://etherpad.openstack.org/p/YVR-ops-federation
* https://etherpad.openstack.org/p/YVR-ops-rabbitmq
* https://etherpad.openstack.org/p/YVR-ops-logging
* https://etherpad.openstack.org/p/YVR-ops-arch-show-tell
* https://etherpad.openstack.org/p/YVR-ops-ceilometer
* https://etherpad.openstack.org/p/YVR-ops-billing
* https://etherpad.openstack.org/p/YVR-ops-cinder
* https://etherpad.openstack.org/p/YVR-ops-legacy-apps
* https://etherpad.openstack.org/p/YVR-ops-user-committee
* https://etherpad.openstack.org/p/YVR-ops-hypervisor-tuning
* https://etherpad.openstack.org/p/YVR-ops-security
* https://etherpad.openstack.org/p/YVR-ops-deployment
* https://etherpad.openstack.org/p/YVR-ops-database
* https://etherpad.openstack.org/p/YVR-ops-evangelism
* https://etherpad.openstack.org/p/YVR-ops-multi-site
* https://etherpad.openstack.org/p/YVR-ops-nova
* https://etherpad.openstack.org/p/YVR-ops-customer-onboarding
* https://etherpad.openstack.org/p/YVR-ops-containers
* https://etherpad.openstack.org/p/YVR-ops-neutron

Wednesday:
* https://etherpad.openstack.org/p/YVR-ops-telco
* https://etherpad.openstack.org/p/YVR-ops-puppet
* https://etherpad.openstack.org/p/YVR-ops-chef
* https://etherpad.openstack.org/p/YVR-ops-hpc
* https://etherpad.openstack.org/p/YVR-ops-tools
* https://etherpad.openstack.org/p/YVR-ops-ansible
* https://etherpad.openstack.org/p/YVR-ops-ceph
* https://etherpad.openstack.org/p/YVR-ops-tags
* https://etherpad.openstack.org/p/YVR-ops-large-deployments
* https://etherpad.openstack.org/p/YVR-ops-burning-issues
* https://etherpad.openstack.org/p/YVR-ops-docs
* https://etherpad.openstack.org/p/YVR-ops-tech-choices
* https://etherpad.openstack.org/p/YVR-ops-cmdb
* https://etherpad.openstack.org/p/YVR-ops-data-plane-transitions
* https://etherpad.openstack.org/p/YVR-ops-upgrades
* https://etherpad.openstack.org/p/YVR-ops-packaging
* https://etherpad.openstack.org/p/YVR-ops-nova-network

Design Summit/Liberty/Etherpads

2015-05-12T14:56:50Z

Clinton Knight: /* Manila */

[[Category:Summit]]
[[Category:Liberty]]
[[Category:Etherpad]]

<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
__TOC__
</div>
==Cross-Project workshops==
Tuesday:

* 11:15 - 11:55
** [https://etherpad.openstack.org/p/liberty-cross-project-python3 Moving apps to Python 3]
** [https://etherpad.openstack.org/p/liberty-cross-project-user-notifications Async status updates]
* 12:05 - 12:45
** Improving UX across all projects
** Functional Testing Show & Tell
* 2:00 - 2:40
** OpenStack SDK
** Modern JavaScript
* 2:50 - 3:30
** [https://etherpad.openstack.org/p/liberty-cross-project-in-team-scaling In-team scaling]
** Service Catalog Standardization
* 3:40 - 4:20
** API Working Group
** Towards one Network Stack: Part 1 - Neutron Gaps and Concerns
* 4:40 - 5:20
** [https://etherpad.openstack.org/p/liberty-cross-project-release-models OpenStack release model(s)]
** Unified Policy File
* 5:30 - 6:10
** OpenStack Documentation
** [https://etherpad.openstack.org/p/liberty-cross-project-managing-concurrency Managing concurrency]

==Barbican==
==Ceilometer==
wednesday:
* 0900 - 0940: [https://etherpad.openstack.org/p/ceilo-multi-identity componentisation / multi identity]
* 0950 - 1030: [https://etherpad.openstack.org/p/event_alarm event alarms]
* 1100 - 1140: [https://etherpad.openstack.org/p/liberty-ceilometer-pipeline-config pipeline configuration]
* 1150 - 1230: [https://etherpad.openstack.org/p/ceilo-declarative-notifications declarative notification meters]
* 1440 - 1520: [https://etherpad.openstack.org/p/YVR-ops-ceilometer ops followup]
* 1530 - 1610: [https://etherpad.openstack.org/p/ceilo-multi-identity componentization carry over]

 thursday:
* 0900 - 0940: [https://etherpad.openstack.org/p/liberty-ceilometer-meter-deprecation meter deprecation]
* 0950 - 1030: [https://etherpad.openstack.org/p/liberty-ceilometer-meter-event samples/events integration]
* 1100 - 1140: [https://etherpad.openstack.org/p/liberty-ceilometer-versioned-objects versioned objects]
* 1150 - 1230: ops followup

 friday:
* 0900 - 1200: [https://etherpad.openstack.org/p/liberty-ceilometer-contributors-meetup contributor meetup]

==Cinder==
==Designate==
==Documentation==
==Glance==
==Heat==
=== Wed ===
* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-heat-liberty-convergence-1 (W) Finishing off Convergence phase 1]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-heat-liberty-convergence-2 (W) Convergence phase 2 planning]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-heat-liberty-testing (W) Functional and integration testing, identify gaps and suggest improvements]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-heat-liberty-lifecycle-ops (W) Better support for lifecycle operations on (tripleo) heat stacks (upgrades etc)]
* 13:50 - 14:30: [https://etherpad.openstack.org/p/YVR-heat-liberty-client-usability (F) Heat client usability]
* 14:40 - 15:20: [https://etherpad.openstack.org/p/YVR-heat-liberty-template-format (F) Heat template format improvements]
* 15:30 - 16:10: [https://etherpad.openstack.org/p/YVR-heat-liberty-rest-api (F) REST API design and roadmap]
* 16:30 - 17:10: [https://etherpad.openstack.org/p/YVR-heat-liberty-senlin-autoscaling (F) Senlin Autoscaling Project - Deep Dive]
* 17:20 - 18:00: [https://etherpad.openstack.org/p/YVR-heat-liberty-containers (F) Orchestrating containers with Heat]

=== Thu ===
* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-heat-liberty-alembic (W) Migrating from sqlalchemy-migrations to alembic]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-heat-liberty-versioned-objects (W) Future of Versioned objects in heat]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-heat-liberty-autoscaling (W) Autoscaling improvements]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-heat-liberty-deprecation (W) Our deprecation process]
* 16:10 - 16:50: [https://etherpad.openstack.org/p/YVR-heat-liberty-contrib-plugins (W) Moving most contrib plugins in-tree]
* 17:00 - 17:40: [https://etherpad.openstack.org/p/YVR-heat-liberty-resource-registering (W) Conditional registering of resources]

=== Fri ===
* 09:00-12:20: [https://etherpad.openstack.org/p/YVR-heat-contributor-meetup Contributor Meetup]

==Horizon==
==Infrastructure==
==Ironic==
==Keystone==
==Manila==
=== Wed ===
* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-manila-liberty-mount-automation (F) Mount automation]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-manila-liberty-share-migration (F) Share migration]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-manila-liberty-share-replication (F) Share replication]
* 16:30 - 17:10: [https://etherpad.openstack.org/p/YVR-manila-liberty-working-session-1 (W) Working session 1]
* 17:20 - 18:00: [https://etherpad.openstack.org/p/YVR-manila-liberty-working-session-2 (W) Working session 2]

==Neutron==
=== Tue ===
=== Wed ===
* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-neutron-liberty-development Neutron Liberty Development]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-neutron-use-case-discussion Neutron Use Case Discussion]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-neutron-vlan-trunk VLAN Aware VMs]
* 11:50 - 12:30: Neutron Lightning Talks (no etherpad)
* 13:50 - 14:30: [https://etherpad.openstack.org/p/YVR-neutron-octavia Octavia]
* 14:40 - 15:20: [https://etherpad.openstack.org/p/YVR-neutron-opendaylight OpenDaylight]
* 15:30 - 16:10: [https://etherpad.openstack.org/p/YVR-neutron-ovn OVN]
* 16:30 - 17:10: [https://etherpad.openstack.org/p/YVR-neutron-ironic Ironic and Neutron integration]
* 17:20 - 18:00: [https://etherpad.openstack.org/p/YVR-neutron-get-me-a-network Get Me a Network!]

=== Thu ===
* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-neutron-lbaas-use-cases Neutron LBaaS Use Cases]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-neutron-l3 Neutron L3]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-neutron-qos QoS]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-neutron-third-party-ci-liberty Third Party CI in Liberty and Beyond]
* 13:30 - 14:10: [https://etherpad.openstack.org/p/YVR-neutron-testing-in-liberty Testing In Liberty]
* 14:20 - 15:00: [https://etherpad.openstack.org/p/YVR-neutron-RBAC Neutron RBAC]
* 15:10 - 15:50: [https://etherpad.openstack.org/p/YVR-neutron-sg-fwaas-future-direction SG and FWaaS Future Directions]
* 16:10 - 16:50: [https://etherpad.openstack.org/p/YVR-neutron-nfv-enhancements Neutron NFV Enhancements]
=== Fri ===
* 09:00-12:20: [https://etherpad.openstack.org/p/YVR-neutron-contributor-meetup Contributor Meetup]

==Nova==

'''Wednesday:'''

* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-nova-scheduler-in-liberty Scheduler in Liberty]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-nova-scalling-out-scheduler-for-cells Scaling out scheduler for cells ]

* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-nova-cells-v2 Cells v2]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-nova-resource-tracker Resource Tracker, Clustered Hypervisors and NFV]

* 13:50 - 14:30: [https://etherpad.openstack.org/p/YVR-nova-spec-blueprint-unconference Nova Spec/Blueprint Unconference]
* 14:40 - 15:20: [https://etherpad.openstack.org/p/YVR-nova-database-internals Database (part 1)]
* 15:30 - 16:10: [https://etherpad.openstack.org/p/YVR-nova-functional-testing-feature-classification Functional Testing and Feature Classification]

* 16:30 - 17:10: [https://etherpad.openstack.org/p/YVR-nova-instance-ha-evacuate-resize Dealing with compute host failure: Instance HA, Evacuate, Resize]
* 17:20 - 18:00: [https://etherpad.openstack.org/p/YVR-nova-api-2.1-in-liberty Nova API v2.1 in Liberty]

 
'''Thursday:'''

* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-nova-api-2.0-3rd-party Future of Nova API v2.0 and 3rd Party APIs]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-nova-quotas-and-database Quotas and Database (part 2)]

* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-nova-flavors-and-image-properties Flavors and Image Properties]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-nova-error-handling Error Handling]

* 13:30 - 14:10: [https://etherpad.openstack.org/p/YVR-nova-spec-blueprint-unconference Nova Spec/Blueprint Unconference]
* 14:20 - 15:00: [https://etherpad.openstack.org/p/YVR-nova-liberty-priorities Liberty Priorities (part 1)]
* 15:10 - 15:50: [https://etherpad.openstack.org/p/YVR-nova-liberty-priorities Liberty Priorities (part 2)]

* 16:10 - 16:50: [https://etherpad.openstack.org/p/YVR-nova-liberty-process Liberty Process and Scaling out Reviews]
* 17:00 - 17:40: [https://etherpad.openstack.org/p/YVR-nova-network Future of Nova's networking and nova-network]

 
'''Friday:'''

* 09:00 - 12:20 and 13:20 - 16:40: [https://etherpad.openstack.org/p/YVR-nova-contributor-meetup Nova Contributor Meetup]

==Oslo==

'''Wednesday:'''

* 09:00 - 09:40: [https://etherpad.openstack.org/p/YVR-oslo-versioned-objects-intro (F) Get to know your objects and learn how to version them (an introduction to oslo.versionedobjects)!]
* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-oslo-taskflow-plans (F) Give me liberty, or give me taskflow (come learn about taskflow liberty plans)!]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-oslo-rootwrap-plans (W) Give me liberty, or give me wraps (the future of oslo.rootwrap) ]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-oslo-functional-testing (W) The cost of liberty is less than the price of functional testing]
* 13:50 - 14:30: [https://etherpad.openstack.org/p/YVR-oslo-graduation-schedule (W) Give me more oslo in liberty or else! Schedule & new libraries]
* 14:40 - 15:20: [https://etherpad.openstack.org/p/YVR-oslo-messaging-zmq-status (F) State of zmq in oslo.messaging]
* 15:30 - 16:10: [https://etherpad.openstack.org/p/YVR-oslo-optional-dependencies (W) Emancipate/liberate your optional dependencies - Optional Dependencies]
* 16:30 - 17:10: [https://etherpad.openstack.org/p/YVR-oslo-release-process-review (W) Reviewing our release processes]
* 17:20 - 18:00: [https://etherpad.openstack.org/p/YVR-oslo-config-filter (W) Configuration Filters in oslo.config]

 
'''Thursday:'''

* 09:50 - 10:30: [https://etherpad.openstack.org/p/YVR-oslo-strategy-discussion (F) The oslo liberty proclamation (and associated strategy discussion)]
* 11:00 - 11:40: [https://etherpad.openstack.org/p/YVR-oslo-config-plans (F) Enfranchise oslo.config, let's discuss alternative data sources in oslo.config]
* 11:50 - 12:30: [https://etherpad.openstack.org/p/YVR-oslo-log-plans (W) Life, Liberty, and the pursuit of oslo.log changes]
* 13:30 - 14:10: [https://etherpad.openstack.org/p/YVR-oslo-versioned-objects-deep-dive (W) Deep dive on oslo.versionedobjects, bring your wet suits.]
* 14:20 - 15:00: [https://etherpad.openstack.org/p/YVR-oslo-db-plans (F) For a people who are free, and who mean to remain so, a well organized and armed 'oslo.db' is their best security]
* 15:10 - 15:50: [https://etherpad.openstack.org/p/YVR-oslo-asyncio (F) Event loops, coroutines, yield from, futures, a discussion on asyncio (and triollus?)]
* 16:10 - 16:50: [https://etherpad.openstack.org/p/YVR-oslo-messaging-plans (F) Ping pong, oslo.messaging plans for liberty.]
* 17:00 - 17:40: [https://etherpad.openstack.org/p/YVR-oslo-tech-debt-deprecation (F) How to clean up your tech-debt; let's discuss best practices on how to deprecate things in oslo libraries]

 

==QA==

=== Wed. ===

* 1150-1230 - [https://etherpad.openstack.org/p/YVR-QA-testing-beyond-the-gate Testing beyond the gate]
* 1350-1430 - [https://etherpad.openstack.org/p/YVR-QA-Devstack-Roadmap Devstack Roadmap]
* 1440-1520 - [https://etherpad.openstack.org/p/YVR-QA-Tempest-service-clients Work Session: Tempest service clients]
* 1630-1710 - [https://etherpad.openstack.org/p/YVR-QA-in-the-big-tent QA in the Big Tent]

=== Thurs. ===
* 0900-0940 - [https://etherpad.openstack.org/p/YVR-QA-idempotent-id Work Session: Idempotent ID]
* 1330-1410 - [https://etherpad.openstack.org/p/YVR-QA-Tempest-as-a-system-program Work Session: Tempest as System Program]
* 1420-1500 - [https://etherpad.openstack.org/p/YVR-QA-Tempest-CLI Work Session: Tempest CLI and Tempest Cleanup]
* 1610-1650 - [https://etherpad.openstack.org/p/YVR-QA-Tempest-Scope-Revisited Tempest Scope Revisited]
* 1700-1740 - [https://etherpad.openstack.org/p/YVR-QA-Liberty-Priorities Liberty Priorities]

==Release Management==
* Wed 17:20-18:00: [https://etherpad.openstack.org/p/YVR-relmgt-stable-branch Stable branch]
* Thu 11:00-11:40 [https://etherpad.openstack.org/p/YVR-relmgt-library-releases Work session: Library releases]

==Sahara==

=== Wed ===

* 11:00am (W) HA http://etherpad.openstack.org/p/sahara-liberty-ha
* 11:50am (W) API v2 https://etherpad.openstack.org/p/sahara-liberty-api-v2

=== Thu ===

* 1:30pm (F) UI https://etherpad.openstack.org/p/sahara-liberty-ui
* 2:20pm (F) Security https://etherpad.openstack.org/p/sahara-liberty-security
* 3:10pm (W) EDP https://etherpad.openstack.org/p/sahara-liberty-edp
* 4:10pm (W) Resources Visibility/ACL https://etherpad.openstack.org/p/sahara-liberty-resources
* 5:00pm (W) Plugins https://etherpad.openstack.org/p/sahara-liberty-plugins

=== Fri ===

* 9am - 4:40pm Meetup https://etherpad.openstack.org/p/sahara-liberty-meetup

==Swift==
==TripleO==

==Trove==
==Zaqar==
==Other Projects==
==Event intro/closure==
==Ops==
Tuesday:
* https://etherpad.openstack.org/p/YVR-ops-101
* https://etherpad.openstack.org/p/YVR-ops-federation
* https://etherpad.openstack.org/p/YVR-ops-rabbitmq
* https://etherpad.openstack.org/p/YVR-ops-logging
* https://etherpad.openstack.org/p/YVR-ops-arch-show-tell
* https://etherpad.openstack.org/p/YVR-ops-ceilometer
* https://etherpad.openstack.org/p/YVR-ops-billing
* https://etherpad.openstack.org/p/YVR-ops-cinder
* https://etherpad.openstack.org/p/YVR-ops-legacy-apps
* https://etherpad.openstack.org/p/YVR-ops-user-committee
* https://etherpad.openstack.org/p/YVR-ops-hypervisor-tuning
* https://etherpad.openstack.org/p/YVR-ops-security
* https://etherpad.openstack.org/p/YVR-ops-deployment
* https://etherpad.openstack.org/p/YVR-ops-database
* https://etherpad.openstack.org/p/YVR-ops-evangelism
* https://etherpad.openstack.org/p/YVR-ops-multi-site
* https://etherpad.openstack.org/p/YVR-ops-nova
* https://etherpad.openstack.org/p/YVR-ops-customer-onboarding
* https://etherpad.openstack.org/p/YVR-ops-containers
* https://etherpad.openstack.org/p/YVR-ops-neutron

Wednesday:
* https://etherpad.openstack.org/p/YVR-ops-telco
* https://etherpad.openstack.org/p/YVR-ops-puppet
* https://etherpad.openstack.org/p/YVR-ops-chef
* https://etherpad.openstack.org/p/YVR-ops-hpc
* https://etherpad.openstack.org/p/YVR-ops-tools
* https://etherpad.openstack.org/p/YVR-ops-ansible
* https://etherpad.openstack.org/p/YVR-ops-ceph
* https://etherpad.openstack.org/p/YVR-ops-tags
* https://etherpad.openstack.org/p/YVR-ops-large-deployments
* https://etherpad.openstack.org/p/YVR-ops-burning-issues
* https://etherpad.openstack.org/p/YVR-ops-docs
* https://etherpad.openstack.org/p/YVR-ops-tech-choices
* https://etherpad.openstack.org/p/YVR-ops-cmdb
* https://etherpad.openstack.org/p/YVR-ops-data-plane-transitions
* https://etherpad.openstack.org/p/YVR-ops-upgrades
* https://etherpad.openstack.org/p/YVR-ops-packaging
* https://etherpad.openstack.org/p/YVR-ops-nova-network