OpenStack - User contributions [en]

ThirdPartySystems/NetApp CI

2016-02-16T21:54:23Z

Alex Meade:

{{ThirdPartySystemInfo|name=NetApp CI
|account=netapp-ci
|contact= xdl-openstack-jenkins@netapp.com, andrew.kerr@netapp.com (akerr), alex.meade@netapp.com (ameade)
|intent=Test Openstack with NetApp Cinder and Manila drivers using storage arrays
|structure=Nodepool, Zuul, Jenkins, modified upstream components with local configuration
|method=Using Zuul to connect to Gerrit
|programs=Cinder, Manila
|status= Currently reporting on both Cinder and Manila
|recheck=run-netapp-ci}}
'''Recheck trigger:''' "run-netapp-ci"

ThirdPartySystems/NetApp CI

2016-02-11T18:09:54Z

Alex Meade:

{{ThirdPartySystemInfo|name=NetApp CI
|account=netapp-ci
|contact= xdl-openstack-jenkins@netapp.com, andrew.kerr@netapp.com (akerr), alex.meade@netapp.com (ameade)
|intent=Test Openstack with NetApp Cinder and Manila drivers using storage arrays
|structure=Nodepool, Zuul, Jenkins, modified upstream components with local configuration
|method=Using Zuul to connect to Gerrit
|programs=Cinder, Manila
|status=Not Reporting, Experiencing technical difficulties with a provider.
|recheck=run-netapp-ci}}
'''Recheck trigger:''' "run-netapp-ci"

Manila/design/manila-mitaka-data-replication

2016-01-27T18:31:53Z

Alex Meade: /* Example Conf */

=Manila Disaster Recovery / Data Replication Design=

==Introduction==

Replicated shares will be implemented in Manila without adding any new services or any new drivers. The code for creating replicated shares and for adding/removing replicas and other replication-related operations will all go into the existing share drivers. The most significant change required to allow this will be that share drivers will be responsible for communicating with ALL storage controllers necessary to achieve any replication tasks, even if that involves sending commands to other storage controllers in other AZs.

While we can't know how every storage controller works and how each vendor will implement replication, this approach should give driver writers all the flexibility needed to implement what they need with minimal added complexity in the Manila core. There are already examples of drivers reading config sections for other drivers when a operation requires communicating with 2 backends. The Manila Share Manager itself is expected to communicate all necessary backend details for share replicas that exist across AZs to a backend which it requests an operation upon.

Intra-AZ replication is supported, but in an ideal solution, AZs should be perceived as failure domains. So this feature tends to shine in an inter-AZ replication use case.

===Supported Replication Types===
There are 3 styles of DR that we would like to support in the long run:

# writable - Amazon EFS-style synchronously replicated shares where all replicas are writable. Promotion it not supported and not needed.
# readable - Mirror-style replication with a primary (writable) copy and one or more secondary (read-only) copies which can become writable after a promotion of the secondary.
# dr - Generalized replication with secondary copies that are inaccessible until after a promotion of one of the secondary copies.

===Replica States===
Each replica has a ''replica_state'' which has 4 possible values:

# active - All writable replicas are active.
# in_sync - Passive replica which is up to date with the active replica, and can be promoted to active.
# out_of_sync - Passive replica which has gone out of date, or new replica that is not yet up to date.
# error - Either the scheduler failed to schedule this replica or some irrecoverable damage occurred during promotion of a non-active replica. Drivers may set the ''replica_state'' to ''error'' if some irrecoverable damage to the replica is discovered at any point during its existence. (through the periodic ''replica_state'' update)
[[File:Manila instance replication states.png|center]]

===New share state===
replication_change - New transient state triggered by a change of the active replica. Access to the share is cut off while in this state.

===Promotion===
For '''readable''' and '''dr''' styles, we refer to the task of switching a non-active replica with the '''active''' replica as '''promotion'''. For the '''writable''' style of replication, '''promotion''' does not make sense since all replicas are '''active''' (writable) at all given points of time. Promotion of replicas with '''replica_state''' set to '''out_of_sync''' or '''error''' may be unsupported by the backend. However, we would like to allow the action as an administrator feature and such an attempt may be honored by backends if possible.

When multiple replicas exist, multiple replication relationships between shares may need to be redefined at the backend. If the driver fails at this stage, the replicas may be left at an inconsistent state. The manager will set all replica instances to have '''replica_state''' and '''status''' to '''error'''. Recovery from this stage would require administrator intervention.

==User Workflows==
====Creating a share supporting replication====
* Administrators can create a share type with extra-spec ''replication_type'', specifying the style of replication the backend supports.
* Users can use the share type to create a new share that allows/supports replication.
* A replicated share always starts out with one replica instance, the share itself. This should not be confused as the share having a replica already. The user can verify if the share has any replicas by requesting the details of the share.

====Creating a replica====
* POST to /share-replicas
User has to specify the share name/id of the share that is supposed to be replicated, an availability zone for the replica to exist in and optionally a share_network_id.
According to existing design, replicas of a share cannot exist in the same availability zone as the share itself.
* A newly created share_replica starts out in ''out_of_sync'' state and might transition to ''in_sync'' state when the driver reports this state.

====Listing and showing replicas====
* GET from /share-replicas
User can list all replicas and verify their ''status'' and ''replica_state''.
* GET from /share-replicas?share_id=<share_id>
User can list replicas for a particular share.
* GET from /share-replicas/<id>
User can view details of a replica.

====Promoting a non-active replica====
* POST to /share-replicas/action with body {'promote': None}
For replication types that permit promotion, the user can promote a replica with ''in_sync'' replica_state to ''active'' replica_state by initiating the ''promote_replica'' call.
* Only administrators can attempt promoting replicas with replica_state ''error'' or ''out_of_sync''.

====Deleting Replicas====
* DELETE to /share-replicas/<id>
The user can delete a replica. The last ''active'' replica cannot be deleted using this ''delete_replica'' call.

==System Workflows==
* Creating a share that supports replication
** Same process as creating a new share.
** The ''replication_type'' extra-spec from the share-type is copied over to the share''s data on the DB.
** Scheduler uses the ''replication_type'' extra-spec to filter available hosts and schedule the share on an appropriate backend that supports that specific style of ''replication_type''.

* Creating a replica:
** Create a share instance on the database.
** Update the share instance ''replica_state'' to ''out_of_sync''.
** Cast a call to the scheduler to find an appropriate host to schedule the replica on.
** In the scheduler, find weighted host for the replica.
*** If host cannot be chosen, update the replica's ''status'' and ''replica_state'' on the Database to ''error''. Throw an exception.
** Cast the create_share_replica call to the driver based on the weighted host selection.
** Prior to invoking the driver's call, collate information regarding existing ''active'' replica instance, existing share access rules and the replica's share_server, pass these to the driver while invoking ''create_replica''.
** The driver may return ''export_locations'' and ''replica_state''. If they are returned, the database is updated with these values.
** If the ''replication_type'' is ''writable'', the driver MUST return the ''replica_state'' set to ''active''.
** All the access rules for the new replica are set to ''active'' state on the database.
** The driver may throw an exception, update the ''status'' and ''replica_state'' of the replica to ''error''.

* Listing/showing shares:
** For shares that support replication, the result of listing/showing the share will have new fields, ''replication_type'': denoting the replication style supported and ''has_replicas'' to denote if the share has replicas.
** For replicated shares, the primary instance for the share must be preferred among those replicas that have a status ''replication_change'' or among those that have a replica_state set to ''active''.

* Listing replicas
** For listing, Grab share instances from the database that have a replica_state among {''in_sync'', ''active'', ''out_of_sync'', ''error''}.
** If share_id is provided, grab only share instances that belong to that share and have a replica_state among {''in_sync'', ''active'', ''out_of_sync'', ''error''}.
** Limits and offsets must be respected on the list calls.

* Promoting a non-active replica:
** If replica is already active, do nothing.
** Update the ''status'' of the replica being promoted to ''replication_change''.
** Grab all available replicas and invoke the appropriate driver''s promote_replica call by passing the available replicas and the new replica.
** If the driver throws an exception at this stage, there is a good chance that the replicas are somehow altered on the backend. Loop through the replicas and set their ''replica_state''s to ''error'' and leave the ''status'' unchanged. Also set the ''status'' of the replica that failed to promote to ''available'' as before this operation. The backend may choose to update the actual replica_state during the replica monitoring call.
** The driver may return an updated list of replicas. Update the ''export_locations'' and ''replica_state''s to the database.
** The ''status'' of the replica that was promoted should return to ''available'' from ''replication_change''.

* Periodic replica update:
** The share manager implements a looping call with default interval of 5 minutes to query from each driver and each backend the ''replica_state'' of all non-active replicas that are associated with them.
** The driver is allowed to set the ''replica_state'' to ''in_sync'', ''out_of_sync'', and in exceptional cases, ''error''.
** If the driver sets the ''replica_state'' of a replica to ''error'', it is assumed that some irrecoverable damage has occured to the replica instance. The ''status'' of the replica instance must be set to ''error'' as well.

* Deleting a share replica
** If the replica had no host, it is simply removed from the database.
** The ''status'' of the replica is set to ''deleting'' on the database and the appropriate driver method is called to delete the replica.
** If the driver fails to delete the replica, the ''status'' of the replica is updated to ''error_deleting'' on the database.
** If the driver returns without errors, the replica instance is removed from the database.

==Scheduler Impact==
* The host_manager must update the ''replication_type'' capability from the backend. The backends must report this capability at the host/pool level.
* The filter_scheduler must then match the ''replication_type'' capability with the capability that the replication share_type demands.
* The ''replication_type'' must be one of the styles mentioned [[#Supported Replication Styles|above]].

==DB Impact==
* Share Export Locations:
Preferred export locations will only be from the instances with ''replica_state'' set to ''active''.
def export_locations(self):
# TODO(gouthamr): Return AZ specific export locations for replicated
# shares
# NOTE(gouthamr): For a replicated share, export locations of the
# 'active' instances are taken, if 'available'.
all_export_locations = []
select_instances = list(filter(
lambda x: x['replica_state'] == constants.REPLICA_STATE_ACTIVE,
self.instances)) or self.instances
for instance in select_instances:
if instance['status'] == constants.STATUS_AVAILABLE:
for export_location in instance.export_locations:
all_export_locations.append(export_location['path'])
return all_export_locations

* Share Instance:
Preferred instance will be an instance with ''status'' set to ''replication_change'' or any instance with ''status'' set to ''available'' and ''replica_state'' set to ''active''.
def instance(self):
# NOTE(gouthamr): The order of preference: status 'replication_change',
# followed by 'available' and 'creating'. If replicated share and
# not undergoing a 'replication_change', only 'active' instances are
# preferred.
result = None
if len(self.instances) > 0:
order = [constants.STATUS_REPLICATION_CHANGE,
constants.STATUS_AVAILABLE, constants.STATUS_CREATING]
other_statuses = ([x['status'] for x in self.instances
if x['status'] not in order])
order.extend(other_statuses)
sorted_instances = sorted(
self.instances, key=lambda x: order.index(x['status']))
select_instances = sorted_instances
if (select_instances[0]['status'] !=
constants.STATUS_REPLICATION_CHANGE):
select_instances = (
list(filter(lambda x: x['replica_state'] ==
constants.REPLICA_STATE_ACTIVE,
sorted_instances)) or sorted_instances
)
result = select_instances[0]
return result

* New field on Share:
replication_type = Column(String(255), nullable=True)

* New field on ShareInstance
replica_state = Column(String(255), nullable=True)

==API Design==
GET /share-replicas/
GET /share-replicas?share-id=<share_id>
GET /share-replicas/<replica_id>

POST /share-replicas/<share-id>
Body:
{
'share_replica':
{
'availability_zone':<availability_zone_id>,
'share_id':<share_id>,
'share_network_id':<share_network_id>
}
}
POST /share-replicas/<replica-id>/action
Body:
{'os-promote_replica': null}

DELETE /share-replicas/<replica-id>

==Policies==

policy.json - All replication group policies should default to the default policy
"share_replica:get_all": "rule:default",
"share_replica:show": "rule:default",
"share_replica:create" : "rule:default",
"share_replica:delete": "rule:default",
"share_replica:promote": "rule:default"

==Driver API==
Only a single replication driver method will be called at a time for a given share as they are locked by share id. Drivers should avoid raising errors to the manager for any of these methods. If the driver hits an issue it should attempt to return the correct model updates with the correct error states set to represent the actual state of the resources. A driver raising an exception will result in all replicas in the error status with the replica_state unchanged.

def create_replica(self, context, active_replica, new_replica,
access_rules, share_server=None):
"""Replicate the active replica to a new replica on this backend.
:param context: Current context
:param active_replica: A current active replica instance dictionary.
EXAMPLE:
.. code::
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS1',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'active',
'availability_zone_id': 'e2c2db5c-cb2f-4697-9966-c06fb200cb80',
'export_locations': [
<models.ShareInstanceExportLocations>._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param new_replica: The share replica dictionary.
EXAMPLE:
.. code::
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS2',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'out_of_sync',
'availability_zone_id': 'f6e146d0-65f0-11e5-9d70-feff819cdc9f',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': 'e6155221-ea00-49ef-abf9-9f89b7dd900a',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param access_rules: A list of access rules that other instances of
the share already obey.
EXAMPLE:
.. code::
[ {
'id': 'f0875f6f-766b-4865-8b41-cccb4cdf1676',
'deleted' = False,
'share_id' = 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'access_type' = 'ip',
'access_to' = '172.16.20.1',
'access_level' = 'rw',
}]
:param share_server: <models.ShareServer>._as_dict() or None,
Share server of the replica being created.
:return: (export_locations, replica_state)
export_locations is a list of paths and replica_state is one of
active, in-sync, out-of-sync or error.
A backend supporting 'writable' type replication should return
'active' as the replica_state.
Export locations should be in the same format as returned by a
share_create. This list may be empty or None.
EXAMPLE:
.. code::
[{'id': 'uuid', 'export_locations': ['export_path']}]
"""
raise NotImplementedError()

def delete_replica(self, context, active_replica, replica,
share_server=None):
"""Delete a replica.
:param context:Current context
:param active_replica: A current active replica instance dictionary.
EXAMPLE:
.. code::
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS1',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'active',
'availability_zone_id': 'e2c2db5c-cb2f-4697-9966-c06fb200cb80',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param replica: Dictionary of the share replica being deleted.
EXAMPLE:
.. code::
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS2',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'in_sync',
'availability_zone_id': 'f6e146d0-65f0-11e5-9d70-feff819cdc9f',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '53099868-65f1-11e5-9d70-feff819cdc9f',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param share_server: <models.ShareServer>._as_dict() or None,
Share server of the replica to be deleted.
:return: None.
"""
raise NotImplementedError()

def promote_replica(self, context, replica_list, replica, access_rules, share_server=None):
"""Promote a replica to 'active' replica state.
:param context:Current context
:param replica_list: List of all replicas for a particular share.
This list also contains the replica to be promoted. The 'active'
replica will have its 'replica_state' attr set to 'active'.
EXAMPLE:
.. code::
[
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'replica_state': 'in-sync',
...
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
'share_server': <models.ShareServer>._as_dict() or None,
},
{
'id': '10e49c3e-aca9-483b-8c2d-1c337b38d6af',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'replica_state': 'active',
...
'share_server_id': 'f63629b3-e126-4448-bec2-03f788f76094',
'share_server': <models.ShareServer>._as_dict() or None,
},
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'replica_state': 'in-sync',
...
'share_server_id': '07574742-67ea-4dfd-9844-9fbd8ada3d87',
'share_server': <models.ShareServer>._as_dict() or None,
},
...
]
:param replica: Dictionary of the replica to be promoted.
EXAMPLE:
.. code::
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS2',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'in_sync',
'availability_zone_id': 'f6e146d0-65f0-11e5-9d70-feff819cdc9f',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '07574742-67ea-4dfd-9844-9fbd8ada3d87',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param access_rules: A list of access rules that other instances of
the share already obey.
EXAMPLE:
.. code::
[ {
'id': 'f0875f6f-766b-4865-8b41-cccb4cdf1676',
'deleted' = False,
'share_id' = 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'access_type' = 'ip',
'access_to' = '172.16.20.1',
'access_level' = 'rw',
}]
:param share_server: <models.ShareServer>._as_dict() or None,
Share server of the replica to be promoted.
:return: updated_replica_list or None
The driver can return the updated list as in the request
parameter. Changes that will be updated to the Database are:
'export_locations' and 'replica_state'.
:raises Exception
This can be any exception derived from BaseException. This is
re-raised by the manager after some necessary cleanup. If the
driver raises an exception during promotion, it is assumed
that all of the replicas of the share are in an inconsistent
state. Recovery is only possible through the periodic update
call and/or administrator intervention to correct the 'status'
of the affected replicas if they become healthy again.
"""
raise NotImplementedError()

def update_replica_state(self, context, replica, share_server=None):
"""Update the replica_state of a replica.
Drivers should fix replication relationships that were broken if
possible inside this method.
:param context:Current context
:param replica: Dictionary of the replica being updated.
EXAMPLE:
.. code::
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS1',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'active',
'availability_zone_id': 'e2c2db5c-cb2f-4697-9966-c06fb200cb80',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
}
:param share_server: <models.ShareServer>._as_dict() or None
:return: replica_state
replica_state - a str value denoting the replica_state that the
replica can have. Valid values are 'in_sync' and 'out_of_sync'
or None (to leave the current replica_state unchanged).
"""
raise NotImplementedError()

==Manila Client==
share-replica-create Creates a replica
share-replica-delete Remove one or more share replicas.
share-replica-list List share replicas.
share-replica-promote Promote replica to active replica.
share-replica-show Show details about a replica.

==Configuration Options==
replica_state_update_interval
This value, specified in seconds, determines how often the share manager will poll for the health (replica_state) of each replica instance.

==First-Party Driver==
DR will require first-party driver support before it can be promoted from an experimental feature for two main reasons.
# A feature is not truly open if only a single vendor can develop it due to it only being supported by a proprietary driver.
# In order to ensure the quality of a feature, it must be testable in upstream gate.

=FAQs=
# '''How do we deal with Network issues in multi-SVM replication?''' If we choose to make replication a single-svm-only feature, the share-network API doesn't need to change. In order to support replication with share-networks, we also need to modify the share-network create API which allows creation of share networks with a table of AZ-to-subnet mappings. This approach allows us to keep a single share-network per share (with associated security service) while allowing the tenant to specify enough information that each share instance can be attached to the appropriate network in each AZ. Multi-AZ share networks would also be useful for non-replicated use cases. 
# '''Do we support Pool-level Replication?''' Some vendors have suggested that certain backends can mirror groups of shares or whole pools more efficiently than individual shares. This design only addresses the mirroring of individual shares. In the future, we may allow allow replication of groups of shares, but only if those groups are contained within a single tenant and defined by the tenant. 
# '''From where to where do we allow replication? Is it intra-cloud or inter-cloud? Do we allow replication to something that's not managed by Manila?''' Intra-cloud. Replicating to something outside of Manila allows a bit more freedom, but with significantly less value, because there's practically nothing we can do to automate the failover/failback portion of a disaster. For use cases involving replication outside of Manila, we would need to involve other tools with more breadth/scope to manage the process. 
# '''Who configures the replication? The admin? The end user? The manila scheduler?''' The end user. In the original design we presumed that the actual replication relationships should be hidden from the end user, but this doesn't match well with the concept of AZs that we are adding to Manila. If the users need to have control over which AZ the primary copy of their data lives in, then they also need to control where the other copies live. This means that the administrator's job is to ensure that for any share type that is replicated, it can be replicated from any AZ to any other AZ. 
# '''Is replication supported only intra-region or inter-region as well? How about intra-AZ vs inter-AZ?''' This design is for intra-region only. Inter-region replication is clearly a feature we want, but it would look pretty different from intra-region replication; this design does not cover that. Intra-AZ replication is supported, however inter-AZ replication is recommended as AZs should be perceived as failure domains.

=Unanswered Questions=
# '''Is there no way to achieve non-disruptive failover?''' (bswartz) I would love to find out that our initial intuition here is wrong, because it would change a lot of aspects of the design. It's worth spending time to brainstorm and research possibilities in this area. So far the most promising ideas involve:
#*Using VirtFS to mediate filesystem access and achieving non-disruptive failover that way
#*Using some kind of agent inside the guests to mediate file access 
# '''Can a replica have a different share_type? There is a valid use case where a user would want to create a share replica on backend with different capabilities than the one the original share resides on. For instance, replicas might need to be on a less expensive backend. In that case, can the replica have a different share_type altogether?''' "Currently", we inherit the share_type of the share and believe that replication has to be on symmetric terms, where both backends have similar capabilities. 
#'''Can we allow the driver to restrict replication support between available backends? Backends may support replication to other compatible backends only. Hence, they must report some sort of information to the scheduler so that when creating a replica for an existing share, the scheduler would use that information to schedule the creation of the replica. What information should this be?''' (gouthamr) We're investigating including 'driver_class_name' in a ReplicationFilter, including the possibility of backend reported configuration, 'replication_partners'. 
#'''How are access rules persisted across replicas/share instances?''' 
#*Do all replicas have the same access rules applied? (gouthamr): This is Currently being pursued
#*Should access rules be applied only to "active" replicas? 
#'''How does migration affect replicated shares?'''
#*(gouthamr)In the Tokyo summit, it was decided that we currently disallow migration of a share that has replicas. The replication relationships must be broken and reestablished after the migration is complete in order to migrate a share with replicas. 
# '''Do we need an API to initiate a sync? (resync or force-update)''' This is a great idea and one certain use case is during planned failovers. (gouthamr) 
# '''How to recover from promotion failures?''' (gouthamr) Currently, if some exception occurs at the driver level, the replicas are rendered useless. Should we provide escape hatches to avoid complete loss in case these are recoverable exceptions? The administrator has the ability to 'reset-state' and affect the 'status' of any replica. Currently, there exists no API to 'reset-replica-state' for a replica.

= Examples =

These examples assume that the replicas exist in different AZs.

== Writable replication example ==

# Administrator sets up backends in AZs b1 and b2 that have capability replication_type=writeable
# Administrator creates a new share_type called foo
# Administrator sets replication_type=writeable extra spec on share type foo
# User creates new share of type foo in AZ b1
# Share is created with replication_type=writeable, and 1 active replica in AZ b1
# User grants access on share to client1 in AZ b1, obtains the export location of the replica, mounts the share on a client, and starts to write data
# User add new replica of share in AZ b2
# A second replica is created in AZ b2 which initially has state out_of_sync
# Shortly afterwards, the replica state changes to active (after the replica finishes syncing with the original copy)
# The user grants access on the share to client2 in AZ b2, obtains the export location of the new replica, mounts the share, and sees the same data that client1 wrote
# Client2 writes some data to the share, which is immediately visible to client1

== Readable replication example ==

# Administrator sets up backends in AZs b1 and b2 that have capability replication_type=readable
# Administrator creates a new share_type called bar
# Administrator sets replication_type=readable extra spec on share type bar
# User creates new share of type bar in AZ b1
# Share is created with replication_type=readable, and 1 active replica in AZ b1
# User grants access on share to client1 in AZ b1, obtains the export location of the replica, mounts the share on a client, and starts to write data
# User add new replica of share in AZ b2
# A second replica is created in AZ b2 which initially has state out_of_sync
# Shortly afterwards, the replica state changes to in_sync (after the replica finishes syncing with the original copy)
# The user grants access on the share to client2 in AZ b2, obtains the export location of the new replica, mounts the share, and sees the same data that client1 wrote
# Client2 cannot write data to the share but continues to see updates

== Failover/failback example ==

(Continued from above)
# An outage occurs in AZ b1
# Administrator sends out a bulletin about the outage to his users "the power transformer in b1 turned to slag, it will be 12 hours before it can be replaced, please bear with us, yada yada"
# User notices that his application on client1 is no longer running, and investigates
# User finds out that client1 is gone, and reads a bulletin from the admin explaining why
# User notes that the b1 replica is his share is still active and the b2 replica is in_sync, while the state of the share is available
# User calls set active replica to AZ b2 on his share
# The share goes to state replication_change and access to the share is briefly lost on client2
# The state of replica b2 changes to active and the state of replica b1 changes to out_of_sync after Manila fails to contact the original primary. The state of the share changes back to available, and access is restored on client2
# User starts his application on client2, and the application recovers from an apparent crash, with a consistent copy of the application's data
# User application is back up and running, disaster is averted
# Eventually maintenance on the b1 AZ is completed, and all of the equipment is reactivated
# Administrator sends out a bulletin to his users about the outage ending "we replaced the transformer with a better one, everything is back online, thanks for your understanding, yada yada"
# Manila notices the out_of_sync replica is reachable and initiates a resync, bringing the b1 replica in_sync within a short time
# After some time, the users reads the bulletin and observes that his share is being replicated again. He decides to intentionally move back to b1.
# User gracefully shuts down his application, and flushes I/O
# User calls set active replica to AZ b1 on his share
# The share goes to state replication_change and no disruption is observed because the application is shut down
# The state of replica b1 changes to active and the state of replica b2 changes to in_sync after Manila reverses the replication again. The state of the share changes back to available
# User starts his application on client1, and the application starts cleanly, having been shut down gracefully

=Implementation Progress=
===Core Work===
[https://blueprints.launchpad.net/manila/+spec/manila-share-replication Blueprint for Core API/Scheduler implementation]
[https://blueprints.launchpad.net/manila/+spec/manilaclient-share-replication Blueprint for Client implementation]

===Ex Driver Implementation===
[https://blueprints.launchpad.net/manila/+spec/netapp-cdot-single-svm-replication Blueprint for cDOT Driver Implementation]

=Older Design Pages=
* Replication Use Cases: https://wiki.openstack.org/wiki/Manila/Replication_Use_Cases
* Design Notes: https://wiki.openstack.org/wiki/Manila/Replication_Design_Notes

Manila/design/manila-mitaka-data-replication

2016-01-13T14:03:55Z

Alex Meade: /* Example Conf */

=Manila Disaster Recovery / Data Replication Design=

==Introduction==

Replicated shares will be implemented in Manila without adding any new services or any new drivers. The code for creating replicated shares and for adding/removing replicas and other replication-related operations will all go into the existing share drivers. The most significant change required to allow this will be that share drivers will be responsible for communicating with ALL storage controllers necessary to achieve any replication tasks, even if that involves sending commands to other storage controllers in other AZs.

While we can't know how every storage controller works and how each vendor will implement replication, this approach should give driver writers all the flexibility needed to implement what they need with minimal added complexity in the Manila core. There are already examples of drivers reading config sections for other drivers when a operation requires communicating with 2 backends. The Manila Share Manager itself is expected to communicate all necessary backend details for share replicas that exist across AZs to a backend which it requests an operation upon.

Intra-AZ replication is supported, but in an ideal solution, AZs should be perceived as failure domains. So this feature tends to shine in an inter-AZ replication use case.

===Supported Replication Types===
There are 3 styles of DR that we would like to support in the long run:

# writable - Amazon EFS-style synchronously replicated shares where all replicas are writable. Promotion it not supported and not needed.
# readable - Mirror-style replication with a primary (writable) copy and one or more secondary (read-only) copies which can become writable after a promotion of the secondary.
# dr - Generalized replication with secondary copies that are inaccessible until after a promotion of one of the secondary copies.

===Replica States===
Each replica has a ''replica_state'' which has 4 possible values:

# active - All writable replicas are active.
# in_sync - Passive replica which is up to date with the active replica, and can be promoted to active.
# out_of_sync - Passive replica which has gone out of date, or new replica that is not yet up to date.
# error - Either the scheduler failed to schedule this replica or some irrecoverable damage occurred during promotion of a non-active replica. Drivers may set the ''replica_state'' to ''error'' if some irrecoverable damage to the replica is discovered at any point during its existence. (through the periodic ''replica_state'' update)
[[File:Manila instance replication states.png|center]]

===New share state===
replication_change - New transient state triggered by a change of the active replica. Access to the share is cut off while in this state.

===Promotion===
For '''readable''' and '''dr''' styles, we refer to the task of switching a non-active replica with the '''active''' replica as '''promotion'''. For the '''writable''' style of replication, '''promotion''' does not make sense since all replicas are '''active''' (writable) at all given points of time. Promotion of replicas with '''replica_state''' set to '''out_of_sync''' or '''error''' may be unsupported by the backend. However, we would like to allow the action as an administrator feature and such an attempt may be honored by backends if possible.

When multiple replicas exist, multiple replication relationships between shares may need to be redefined at the backend. If the driver fails at this stage, the replicas may be left at an inconsistent state. The manager will set all replica instances to have '''replica_state''' and '''status''' to '''error'''. Recovery from this stage would require administrator intervention.

==User Workflows==
====Creating a share supporting replication====
* Administrators can create a share type with extra-spec ''replication_type'', specifying the style of replication the backend supports.
* Users can use the share type to create a new share that allows/supports replication.
* A replicated share always starts out with one replica instance, the share itself. This should not be confused as the share having a replica already. The user can verify if the share has any replicas by requesting the details of the share.

====Creating a replica====
* POST to /share-replicas
User has to specify the share name/id of the share that is supposed to be replicated, an availability zone for the replica to exist in and optionally a share_network_id.
According to existing design, replicas of a share cannot exist in the same availability zone as the share itself.
* A newly created share_replica starts out in ''out_of_sync'' state and might transition to ''in_sync'' state when the driver reports this state.

====Listing and showing replicas====
* GET from /share-replicas
User can list all replicas and verify their ''status'' and ''replica_state''.
* GET from /share-replicas?share_id=<share_id>
User can list replicas for a particular share.
* GET from /share-replicas/<id>
User can view details of a replica.

====Promoting a non-active replica====
* POST to /share-replicas/action with body {'promote': None}
For replication types that permit promotion, the user can promote a replica with ''in_sync'' replica_state to ''active'' replica_state by initiating the ''promote_replica'' call.
* Only administrators can attempt promoting replicas with replica_state ''error'' or ''out_of_sync''.

====Deleting Replicas====
* DELETE to /share-replicas/<id>
The user can delete a replica. The last ''active'' replica cannot be deleted using this ''delete_replica'' call.

==System Workflows==
* Creating a share that supports replication
** Same process as creating a new share.
** The ''replication_type'' extra-spec from the share-type is copied over to the share''s data on the DB.
** Scheduler uses the ''replication_type'' extra-spec to filter available hosts and schedule the share on an appropriate backend that supports that specific style of ''replication_type''.

* Creating a replica:
** Create a share instance on the database.
** Update the share instance ''replica_state'' to ''out_of_sync''.
** Cast a call to the scheduler to find an appropriate host to schedule the replica on.
** In the scheduler, find weighted host for the replica.
*** If host cannot be chosen, update the replica's ''status'' and ''replica_state'' on the Database to ''error''. Throw an exception.
** Cast the create_share_replica call to the driver based on the weighted host selection.
** Prior to invoking the driver's call, collate information regarding existing ''active'' replica instance, existing share access rules and the replica's share_server, pass these to the driver while invoking ''create_replica''.
** The driver may return ''export_locations'' and ''replica_state''. If they are returned, the database is updated with these values.
** If the ''replication_type'' is ''writable'', the driver MUST return the ''replica_state'' set to ''active''.
** All the access rules for the new replica are set to ''active'' state on the database.
** The driver may throw an exception, update the ''status'' and ''replica_state'' of the replica to ''error''.

* Listing/showing shares:
** For shares that support replication, the result of listing/showing the share will have new fields, ''replication_type'': denoting the replication style supported and ''has_replicas'' to denote if the share has replicas.
** For replicated shares, the primary instance for the share must be preferred among those replicas that have a status ''replication_change'' or among those that have a replica_state set to ''active''.

* Listing replicas
** For listing, Grab share instances from the database that have a replica_state among {''in_sync'', ''active'', ''out_of_sync'', ''error''}.
** If share_id is provided, grab only share instances that belong to that share and have a replica_state among {''in_sync'', ''active'', ''out_of_sync'', ''error''}.
** Limits and offsets must be respected on the list calls.

* Promoting a non-active replica:
** If replica is already active, do nothing.
** Update the ''status'' of the replica being promoted to ''replication_change''.
** Grab all available replicas and invoke the appropriate driver''s promote_replica call by passing the available replicas and the new replica.
** If the driver throws an exception at this stage, there is a good chance that the replicas are somehow altered on the backend. Loop through the replicas and set their ''replica_state''s to ''error'' and leave the ''status'' unchanged. Also set the ''status'' of the replica that failed to promote to ''available'' as before this operation. The backend may choose to update the actual replica_state during the replica monitoring call.
** The driver may return an updated list of replicas. Update the ''export_locations'' and ''replica_state''s to the database.
** The ''status'' of the replica that was promoted should return to ''available'' from ''replication_change''.

* Periodic replica update:
** The share manager implements a looping call with default interval of 5 minutes to query from each driver and each backend the ''replica_state'' of all non-active replicas that are associated with them.
** The driver is allowed to set the ''replica_state'' to ''in_sync'', ''out_of_sync'', and in exceptional cases, ''error''.
** If the driver sets the ''replica_state'' of a replica to ''error'', it is assumed that some irrecoverable damage has occured to the replica instance. The ''status'' of the replica instance must be set to ''error'' as well.

* Deleting a share replica
** If the replica had no host, it is simply removed from the database.
** The ''status'' of the replica is set to ''deleting'' on the database and the appropriate driver method is called to delete the replica.
** If the driver fails to delete the replica, the ''status'' of the replica is updated to ''error_deleting'' on the database.
** If the driver returns without errors, the replica instance is removed from the database.

==Scheduler Impact==
* The host_manager must update the ''replication_type'' capability from the backend. The backends must report this capability at the host/pool level.
* The filter_scheduler must then match the ''replication_type'' capability with the capability that the replication share_type demands.
* The ''replication_type'' must be one of the styles mentioned [[#Supported Replication Styles|above]].

==DB Impact==
* Share Export Locations:
Preferred export locations will only be from the instances with ''replica_state'' set to ''active''.
def export_locations(self):
# TODO(gouthamr): Return AZ specific export locations for replicated
# shares
# NOTE(gouthamr): For a replicated share, export locations of the
# 'active' instances are taken, if 'available'.
all_export_locations = []
select_instances = list(filter(
lambda x: x['replica_state'] == constants.REPLICA_STATE_ACTIVE,
self.instances)) or self.instances
for instance in select_instances:
if instance['status'] == constants.STATUS_AVAILABLE:
for export_location in instance.export_locations:
all_export_locations.append(export_location['path'])
return all_export_locations

* Share Instance:
Preferred instance will be an instance with ''status'' set to ''replication_change'' or any instance with ''status'' set to ''available'' and ''replica_state'' set to ''active''.
def instance(self):
# NOTE(gouthamr): The order of preference: status 'replication_change',
# followed by 'available' and 'creating'. If replicated share and
# not undergoing a 'replication_change', only 'active' instances are
# preferred.
result = None
if len(self.instances) > 0:
order = [constants.STATUS_REPLICATION_CHANGE,
constants.STATUS_AVAILABLE, constants.STATUS_CREATING]
other_statuses = ([x['status'] for x in self.instances
if x['status'] not in order])
order.extend(other_statuses)
sorted_instances = sorted(
self.instances, key=lambda x: order.index(x['status']))
select_instances = sorted_instances
if (select_instances[0]['status'] !=
constants.STATUS_REPLICATION_CHANGE):
select_instances = (
list(filter(lambda x: x['replica_state'] ==
constants.REPLICA_STATE_ACTIVE,
sorted_instances)) or sorted_instances
)
result = select_instances[0]
return result

* New field on Share:
replication_type = Column(String(255), nullable=True)

* New field on ShareInstance
replica_state = Column(String(255), nullable=True)

==API Design==
GET /share-replicas/
GET /share-replicas?share-id=<share_id>
GET /share-replicas/<replica_id>

POST /share-replicas/<share-id>
Body:
{
'share_replica':
{
'availability_zone':<availability_zone_id>,
'share_id':<share_id>,
'share_network_id':<share_network_id>
}
}
POST /share-replicas/<replica-id>/action
Body:
{'os-promote_replica': null}

DELETE /share-replicas/<replica-id>

==Policies==

policy.json - All replication group policies should default to the default policy
"share_replica:get_all": "rule:default",
"share_replica:show": "rule:default",
"share_replica:create" : "rule:default",
"share_replica:delete": "rule:default",
"share_replica:promote": "rule:default"

==Driver API==
Only a single replication driver method will be called at a time for a given share as they are locked by share id. Drivers should avoid raising errors to the manager for any of these methods. If the driver hits an issue it should attempt to return the correct model updates with the correct error states set to represent the actual state of the resources. A driver raising an exception will result in all replicas in the error status with the replica_state unchanged.

def create_replica(self, context, active_replica, new_replica,
access_rules, share_server=None):
"""Replicate the active replica to a new replica on this backend.
:param context: Current context
:param active_replica: A current active replica instance dictionary.
EXAMPLE:
.. code::
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS1',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'active',
'availability_zone_id': 'e2c2db5c-cb2f-4697-9966-c06fb200cb80',
'export_locations': [
<models.ShareInstanceExportLocations>._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param new_replica: The share replica dictionary.
EXAMPLE:
.. code::
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS2',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'out_of_sync',
'availability_zone_id': 'f6e146d0-65f0-11e5-9d70-feff819cdc9f',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': 'e6155221-ea00-49ef-abf9-9f89b7dd900a',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param access_rules: A list of access rules that other instances of
the share already obey.
EXAMPLE:
.. code::
[ {
'id': 'f0875f6f-766b-4865-8b41-cccb4cdf1676',
'deleted' = False,
'share_id' = 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'access_type' = 'ip',
'access_to' = '172.16.20.1',
'access_level' = 'rw',
}]
:param share_server: <models.ShareServer>._as_dict() or None,
Share server of the replica being created.
:return: (export_locations, replica_state)
export_locations is a list of paths and replica_state is one of
active, in-sync, out-of-sync or error.
A backend supporting 'writable' type replication should return
'active' as the replica_state.
Export locations should be in the same format as returned by a
share_create. This list may be empty or None.
EXAMPLE:
.. code::
[{'id': 'uuid', 'export_locations': ['export_path']}]
"""
raise NotImplementedError()

def delete_replica(self, context, active_replica, replica,
share_server=None):
"""Delete a replica.
:param context:Current context
:param active_replica: A current active replica instance dictionary.
EXAMPLE:
.. code::
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS1',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'active',
'availability_zone_id': 'e2c2db5c-cb2f-4697-9966-c06fb200cb80',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param replica: Dictionary of the share replica being deleted.
EXAMPLE:
.. code::
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS2',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'in_sync',
'availability_zone_id': 'f6e146d0-65f0-11e5-9d70-feff819cdc9f',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '53099868-65f1-11e5-9d70-feff819cdc9f',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param share_server: <models.ShareServer>._as_dict() or None,
Share server of the replica to be deleted.
:return: None.
"""
raise NotImplementedError()

def promote_replica(self, context, replica_list, replica, access_rules, share_server=None):
"""Promote a replica to 'active' replica state.
:param context:Current context
:param replica_list: List of all replicas for a particular share.
This list also contains the replica to be promoted. The 'active'
replica will have its 'replica_state' attr set to 'active'.
EXAMPLE:
.. code::
[
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'replica_state': 'in-sync',
...
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
'share_server': <models.ShareServer>._as_dict() or None,
},
{
'id': '10e49c3e-aca9-483b-8c2d-1c337b38d6af',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'replica_state': 'active',
...
'share_server_id': 'f63629b3-e126-4448-bec2-03f788f76094',
'share_server': <models.ShareServer>._as_dict() or None,
},
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'replica_state': 'in-sync',
...
'share_server_id': '07574742-67ea-4dfd-9844-9fbd8ada3d87',
'share_server': <models.ShareServer>._as_dict() or None,
},
...
]
:param replica: Dictionary of the replica to be promoted.
EXAMPLE:
.. code::
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS2',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'in_sync',
'availability_zone_id': 'f6e146d0-65f0-11e5-9d70-feff819cdc9f',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '07574742-67ea-4dfd-9844-9fbd8ada3d87',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param access_rules: A list of access rules that other instances of
the share already obey.
EXAMPLE:
.. code::
[ {
'id': 'f0875f6f-766b-4865-8b41-cccb4cdf1676',
'deleted' = False,
'share_id' = 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'access_type' = 'ip',
'access_to' = '172.16.20.1',
'access_level' = 'rw',
}]
:param share_server: <models.ShareServer>._as_dict() or None,
Share server of the replica to be promoted.
:return: updated_replica_list or None
The driver can return the updated list as in the request
parameter. Changes that will be updated to the Database are:
'export_locations' and 'replica_state'.
:raises Exception
This can be any exception derived from BaseException. This is
re-raised by the manager after some necessary cleanup. If the
driver raises an exception during promotion, it is assumed
that all of the replicas of the share are in an inconsistent
state. Recovery is only possible through the periodic update
call and/or administrator intervention to correct the 'status'
of the affected replicas if they become healthy again.
"""
raise NotImplementedError()

def update_replica_state(self, context, replica, share_server=None):
"""Update the replica_state of a replica.
Drivers should fix replication relationships that were broken if
possible inside this method.
:param context:Current context
:param replica: Dictionary of the replica being updated.
EXAMPLE:
.. code::
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS1',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'active',
'availability_zone_id': 'e2c2db5c-cb2f-4697-9966-c06fb200cb80',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
}
:param share_server: <models.ShareServer>._as_dict() or None
:return: replica_state
replica_state - a str value denoting the replica_state that the
replica can have. Valid values are 'in_sync' and 'out_of_sync'
or None (to leave the current replica_state unchanged).
"""
raise NotImplementedError()

==Manila Client==
share-replica-create Creates a replica
share-replica-delete Remove one or more share replicas.
share-replica-list List share replicas.
share-replica-promote Promote replica to active replica.
share-replica-show Show details about a replica.

==Example Conf==
There must be stanzas containing information about each of the replication partners.
[default]
storage_availability_zone = az1
enabled_backends = backend1
[backend1]
replication_partners = backend2
[backend2]
replication_partners = backend1

==First-Party Driver==
DR will require first-party driver support before it can be promoted from an experimental feature for two main reasons.
# A feature is not truly open if only a single vendor can develop it due to it only being supported by a proprietary driver.
# In order to ensure the quality of a feature, it must be testable in upstream gate.

=FAQs=
# '''How do we deal with Network issues in multi-SVM replication?''' If we choose to make replication a single-svm-only feature, the share-network API doesn't need to change. In order to support replication with share-networks, we also need to modify the share-network create API which allows creation of share networks with a table of AZ-to-subnet mappings. This approach allows us to keep a single share-network per share (with associated security service) while allowing the tenant to specify enough information that each share instance can be attached to the appropriate network in each AZ. Multi-AZ share networks would also be useful for non-replicated use cases. 
# '''Do we support Pool-level Replication?''' Some vendors have suggested that certain backends can mirror groups of shares or whole pools more efficiently than individual shares. This design only addresses the mirroring of individual shares. In the future, we may allow allow replication of groups of shares, but only if those groups are contained within a single tenant and defined by the tenant. 
# '''From where to where do we allow replication? Is it intra-cloud or inter-cloud? Do we allow replication to something that's not managed by Manila?''' Intra-cloud. Replicating to something outside of Manila allows a bit more freedom, but with significantly less value, because there's practically nothing we can do to automate the failover/failback portion of a disaster. For use cases involving replication outside of Manila, we would need to involve other tools with more breadth/scope to manage the process. 
# '''Who configures the replication? The admin? The end user? The manila scheduler?''' The end user. In the original design we presumed that the actual replication relationships should be hidden from the end user, but this doesn't match well with the concept of AZs that we are adding to Manila. If the users need to have control over which AZ the primary copy of their data lives in, then they also need to control where the other copies live. This means that the administrator's job is to ensure that for any share type that is replicated, it can be replicated from any AZ to any other AZ. 
# '''Is replication supported only intra-region or inter-region as well? How about intra-AZ vs inter-AZ?''' This design is for intra-region only. Inter-region replication is clearly a feature we want, but it would look pretty different from intra-region replication; this design does not cover that. Intra-AZ replication is supported, however inter-AZ replication is recommended as AZs should be perceived as failure domains.

=Unanswered Questions=
# '''Is there no way to achieve non-disruptive failover?''' (bswartz) I would love to find out that our initial intuition here is wrong, because it would change a lot of aspects of the design. It's worth spending time to brainstorm and research possibilities in this area. So far the most promising ideas involve:
#*Using VirtFS to mediate filesystem access and achieving non-disruptive failover that way
#*Using some kind of agent inside the guests to mediate file access 
# '''Can a replica have a different share_type? There is a valid use case where a user would want to create a share replica on backend with different capabilities than the one the original share resides on. For instance, replicas might need to be on a less expensive backend. In that case, can the replica have a different share_type altogether?''' "Currently", we inherit the share_type of the share and believe that replication has to be on symmetric terms, where both backends have similar capabilities. 
#'''Can we allow the driver to restrict replication support between available backends? Backends may support replication to other compatible backends only. Hence, they must report some sort of information to the scheduler so that when creating a replica for an existing share, the scheduler would use that information to schedule the creation of the replica. What information should this be?''' (gouthamr) We're investigating including 'driver_class_name' in a ReplicationFilter, including the possibility of backend reported configuration, 'replication_partners'. 
#'''How are access rules persisted across replicas/share instances?''' 
#*Do all replicas have the same access rules applied? (gouthamr): This is Currently being pursued
#*Should access rules be applied only to "active" replicas? 
#'''How does migration affect replicated shares?'''
#*(gouthamr)In the Tokyo summit, it was decided that we currently disallow migration of a share that has replicas. The replication relationships must be broken and reestablished after the migration is complete in order to migrate a share with replicas. 
# '''Do we need an API to initiate a sync? (resync or force-update)''' This is a great idea and one certain use case is during planned failovers. (gouthamr) 
# '''How to recover from promotion failures?''' (gouthamr) Currently, if some exception occurs at the driver level, the replicas are rendered useless. Should we provide escape hatches to avoid complete loss in case these are recoverable exceptions? The administrator has the ability to 'reset-state' and affect the 'status' of any replica. Currently, there exists no API to 'reset-replica-state' for a replica.

= Examples =

These examples assume that the replicas exist in different AZs.

== Writable replication example ==

# Administrator sets up backends in AZs b1 and b2 that have capability replication_type=writeable
# Administrator creates a new share_type called foo
# Administrator sets replication_type=writeable extra spec on share type foo
# User creates new share of type foo in AZ b1
# Share is created with replication_type=writeable, and 1 active replica in AZ b1
# User grants access on share to client1 in AZ b1, obtains the export location of the replica, mounts the share on a client, and starts to write data
# User add new replica of share in AZ b2
# A second replica is created in AZ b2 which initially has state out_of_sync
# Shortly afterwards, the replica state changes to active (after the replica finishes syncing with the original copy)
# The user grants access on the share to client2 in AZ b2, obtains the export location of the new replica, mounts the share, and sees the same data that client1 wrote
# Client2 writes some data to the share, which is immediately visible to client1

== Readable replication example ==

# Administrator sets up backends in AZs b1 and b2 that have capability replication_type=readable
# Administrator creates a new share_type called bar
# Administrator sets replication_type=readable extra spec on share type bar
# User creates new share of type bar in AZ b1
# Share is created with replication_type=readable, and 1 active replica in AZ b1
# User grants access on share to client1 in AZ b1, obtains the export location of the replica, mounts the share on a client, and starts to write data
# User add new replica of share in AZ b2
# A second replica is created in AZ b2 which initially has state out_of_sync
# Shortly afterwards, the replica state changes to in_sync (after the replica finishes syncing with the original copy)
# The user grants access on the share to client2 in AZ b2, obtains the export location of the new replica, mounts the share, and sees the same data that client1 wrote
# Client2 cannot write data to the share but continues to see updates

== Failover/failback example ==

(Continued from above)
# An outage occurs in AZ b1
# Administrator sends out a bulletin about the outage to his users "the power transformer in b1 turned to slag, it will be 12 hours before it can be replaced, please bear with us, yada yada"
# User notices that his application on client1 is no longer running, and investigates
# User finds out that client1 is gone, and reads a bulletin from the admin explaining why
# User notes that the b1 replica is his share is still active and the b2 replica is in_sync, while the state of the share is available
# User calls set active replica to AZ b2 on his share
# The share goes to state replication_change and access to the share is briefly lost on client2
# The state of replica b2 changes to active and the state of replica b1 changes to out_of_sync after Manila fails to contact the original primary. The state of the share changes back to available, and access is restored on client2
# User starts his application on client2, and the application recovers from an apparent crash, with a consistent copy of the application's data
# User application is back up and running, disaster is averted
# Eventually maintenance on the b1 AZ is completed, and all of the equipment is reactivated
# Administrator sends out a bulletin to his users about the outage ending "we replaced the transformer with a better one, everything is back online, thanks for your understanding, yada yada"
# Manila notices the out_of_sync replica is reachable and initiates a resync, bringing the b1 replica in_sync within a short time
# After some time, the users reads the bulletin and observes that his share is being replicated again. He decides to intentionally move back to b1.
# User gracefully shuts down his application, and flushes I/O
# User calls set active replica to AZ b1 on his share
# The share goes to state replication_change and no disruption is observed because the application is shut down
# The state of replica b1 changes to active and the state of replica b2 changes to in_sync after Manila reverses the replication again. The state of the share changes back to available
# User starts his application on client1, and the application starts cleanly, having been shut down gracefully

=Implementation Progress=
===Core Work===
[https://blueprints.launchpad.net/manila/+spec/manila-share-replication Blueprint for Core API/Scheduler implementation]
[https://blueprints.launchpad.net/manila/+spec/manilaclient-share-replication Blueprint for Client implementation]

===Ex Driver Implementation===
[https://blueprints.launchpad.net/manila/+spec/netapp-cdot-single-svm-replication Blueprint for cDOT Driver Implementation]

=Older Design Pages=
* Replication Use Cases: https://wiki.openstack.org/wiki/Manila/Replication_Use_Cases
* Design Notes: https://wiki.openstack.org/wiki/Manila/Replication_Design_Notes

Manila/design/manila-mitaka-data-replication

2016-01-13T14:00:55Z

Alex Meade:

=Manila Disaster Recovery / Data Replication Design=

==Introduction==

Replicated shares will be implemented in Manila without adding any new services or any new drivers. The code for creating replicated shares and for adding/removing replicas and other replication-related operations will all go into the existing share drivers. The most significant change required to allow this will be that share drivers will be responsible for communicating with ALL storage controllers necessary to achieve any replication tasks, even if that involves sending commands to other storage controllers in other AZs.

While we can't know how every storage controller works and how each vendor will implement replication, this approach should give driver writers all the flexibility needed to implement what they need with minimal added complexity in the Manila core. There are already examples of drivers reading config sections for other drivers when a operation requires communicating with 2 backends. The Manila Share Manager itself is expected to communicate all necessary backend details for share replicas that exist across AZs to a backend which it requests an operation upon.

Intra-AZ replication is supported, but in an ideal solution, AZs should be perceived as failure domains. So this feature tends to shine in an inter-AZ replication use case.

===Supported Replication Types===
There are 3 styles of DR that we would like to support in the long run:

# writable - Amazon EFS-style synchronously replicated shares where all replicas are writable. Promotion it not supported and not needed.
# readable - Mirror-style replication with a primary (writable) copy and one or more secondary (read-only) copies which can become writable after a promotion of the secondary.
# dr - Generalized replication with secondary copies that are inaccessible until after a promotion of one of the secondary copies.

===Replica States===
Each replica has a ''replica_state'' which has 4 possible values:

# active - All writable replicas are active.
# in_sync - Passive replica which is up to date with the active replica, and can be promoted to active.
# out_of_sync - Passive replica which has gone out of date, or new replica that is not yet up to date.
# error - Either the scheduler failed to schedule this replica or some irrecoverable damage occurred during promotion of a non-active replica. Drivers may set the ''replica_state'' to ''error'' if some irrecoverable damage to the replica is discovered at any point during its existence. (through the periodic ''replica_state'' update)
[[File:Manila instance replication states.png|center]]

===New share state===
replication_change - New transient state triggered by a change of the active replica. Access to the share is cut off while in this state.

===Promotion===
For '''readable''' and '''dr''' styles, we refer to the task of switching a non-active replica with the '''active''' replica as '''promotion'''. For the '''writable''' style of replication, '''promotion''' does not make sense since all replicas are '''active''' (writable) at all given points of time. Promotion of replicas with '''replica_state''' set to '''out_of_sync''' or '''error''' may be unsupported by the backend. However, we would like to allow the action as an administrator feature and such an attempt may be honored by backends if possible.

When multiple replicas exist, multiple replication relationships between shares may need to be redefined at the backend. If the driver fails at this stage, the replicas may be left at an inconsistent state. The manager will set all replica instances to have '''replica_state''' and '''status''' to '''error'''. Recovery from this stage would require administrator intervention.

==User Workflows==
====Creating a share supporting replication====
* Administrators can create a share type with extra-spec ''replication_type'', specifying the style of replication the backend supports.
* Users can use the share type to create a new share that allows/supports replication.
* A replicated share always starts out with one replica instance, the share itself. This should not be confused as the share having a replica already. The user can verify if the share has any replicas by requesting the details of the share.

====Creating a replica====
* POST to /share-replicas
User has to specify the share name/id of the share that is supposed to be replicated, an availability zone for the replica to exist in and optionally a share_network_id.
According to existing design, replicas of a share cannot exist in the same availability zone as the share itself.
* A newly created share_replica starts out in ''out_of_sync'' state and might transition to ''in_sync'' state when the driver reports this state.

====Listing and showing replicas====
* GET from /share-replicas
User can list all replicas and verify their ''status'' and ''replica_state''.
* GET from /share-replicas?share_id=<share_id>
User can list replicas for a particular share.
* GET from /share-replicas/<id>
User can view details of a replica.

====Promoting a non-active replica====
* POST to /share-replicas/action with body {'promote': None}
For replication types that permit promotion, the user can promote a replica with ''in_sync'' replica_state to ''active'' replica_state by initiating the ''promote_replica'' call.
* Only administrators can attempt promoting replicas with replica_state ''error'' or ''out_of_sync''.

====Deleting Replicas====
* DELETE to /share-replicas/<id>
The user can delete a replica. The last ''active'' replica cannot be deleted using this ''delete_replica'' call.

==System Workflows==
* Creating a share that supports replication
** Same process as creating a new share.
** The ''replication_type'' extra-spec from the share-type is copied over to the share''s data on the DB.
** Scheduler uses the ''replication_type'' extra-spec to filter available hosts and schedule the share on an appropriate backend that supports that specific style of ''replication_type''.

* Creating a replica:
** Create a share instance on the database.
** Update the share instance ''replica_state'' to ''out_of_sync''.
** Cast a call to the scheduler to find an appropriate host to schedule the replica on.
** In the scheduler, find weighted host for the replica.
*** If host cannot be chosen, update the replica's ''status'' and ''replica_state'' on the Database to ''error''. Throw an exception.
** Cast the create_share_replica call to the driver based on the weighted host selection.
** Prior to invoking the driver's call, collate information regarding existing ''active'' replica instance, existing share access rules and the replica's share_server, pass these to the driver while invoking ''create_replica''.
** The driver may return ''export_locations'' and ''replica_state''. If they are returned, the database is updated with these values.
** If the ''replication_type'' is ''writable'', the driver MUST return the ''replica_state'' set to ''active''.
** All the access rules for the new replica are set to ''active'' state on the database.
** The driver may throw an exception, update the ''status'' and ''replica_state'' of the replica to ''error''.

* Listing/showing shares:
** For shares that support replication, the result of listing/showing the share will have new fields, ''replication_type'': denoting the replication style supported and ''has_replicas'' to denote if the share has replicas.
** For replicated shares, the primary instance for the share must be preferred among those replicas that have a status ''replication_change'' or among those that have a replica_state set to ''active''.

* Listing replicas
** For listing, Grab share instances from the database that have a replica_state among {''in_sync'', ''active'', ''out_of_sync'', ''error''}.
** If share_id is provided, grab only share instances that belong to that share and have a replica_state among {''in_sync'', ''active'', ''out_of_sync'', ''error''}.
** Limits and offsets must be respected on the list calls.

* Promoting a non-active replica:
** If replica is already active, do nothing.
** Update the ''status'' of the replica being promoted to ''replication_change''.
** Grab all available replicas and invoke the appropriate driver''s promote_replica call by passing the available replicas and the new replica.
** If the driver throws an exception at this stage, there is a good chance that the replicas are somehow altered on the backend. Loop through the replicas and set their ''replica_state''s to ''error'' and leave the ''status'' unchanged. Also set the ''status'' of the replica that failed to promote to ''available'' as before this operation. The backend may choose to update the actual replica_state during the replica monitoring call.
** The driver may return an updated list of replicas. Update the ''export_locations'' and ''replica_state''s to the database.
** The ''status'' of the replica that was promoted should return to ''available'' from ''replication_change''.

* Periodic replica update:
** The share manager implements a looping call with default interval of 5 minutes to query from each driver and each backend the ''replica_state'' of all non-active replicas that are associated with them.
** The driver is allowed to set the ''replica_state'' to ''in_sync'', ''out_of_sync'', and in exceptional cases, ''error''.
** If the driver sets the ''replica_state'' of a replica to ''error'', it is assumed that some irrecoverable damage has occured to the replica instance. The ''status'' of the replica instance must be set to ''error'' as well.

* Deleting a share replica
** If the replica had no host, it is simply removed from the database.
** The ''status'' of the replica is set to ''deleting'' on the database and the appropriate driver method is called to delete the replica.
** If the driver fails to delete the replica, the ''status'' of the replica is updated to ''error_deleting'' on the database.
** If the driver returns without errors, the replica instance is removed from the database.

==Scheduler Impact==
* The host_manager must update the ''replication_type'' capability from the backend. The backends must report this capability at the host/pool level.
* The filter_scheduler must then match the ''replication_type'' capability with the capability that the replication share_type demands.
* The ''replication_type'' must be one of the styles mentioned [[#Supported Replication Styles|above]].

==DB Impact==
* Share Export Locations:
Preferred export locations will only be from the instances with ''replica_state'' set to ''active''.
def export_locations(self):
# TODO(gouthamr): Return AZ specific export locations for replicated
# shares
# NOTE(gouthamr): For a replicated share, export locations of the
# 'active' instances are taken, if 'available'.
all_export_locations = []
select_instances = list(filter(
lambda x: x['replica_state'] == constants.REPLICA_STATE_ACTIVE,
self.instances)) or self.instances
for instance in select_instances:
if instance['status'] == constants.STATUS_AVAILABLE:
for export_location in instance.export_locations:
all_export_locations.append(export_location['path'])
return all_export_locations

* Share Instance:
Preferred instance will be an instance with ''status'' set to ''replication_change'' or any instance with ''status'' set to ''available'' and ''replica_state'' set to ''active''.
def instance(self):
# NOTE(gouthamr): The order of preference: status 'replication_change',
# followed by 'available' and 'creating'. If replicated share and
# not undergoing a 'replication_change', only 'active' instances are
# preferred.
result = None
if len(self.instances) > 0:
order = [constants.STATUS_REPLICATION_CHANGE,
constants.STATUS_AVAILABLE, constants.STATUS_CREATING]
other_statuses = ([x['status'] for x in self.instances
if x['status'] not in order])
order.extend(other_statuses)
sorted_instances = sorted(
self.instances, key=lambda x: order.index(x['status']))
select_instances = sorted_instances
if (select_instances[0]['status'] !=
constants.STATUS_REPLICATION_CHANGE):
select_instances = (
list(filter(lambda x: x['replica_state'] ==
constants.REPLICA_STATE_ACTIVE,
sorted_instances)) or sorted_instances
)
result = select_instances[0]
return result

* New field on Share:
replication_type = Column(String(255), nullable=True)

* New field on ShareInstance
replica_state = Column(String(255), nullable=True)

==API Design==
GET /share-replicas/
GET /share-replicas?share-id=<share_id>
GET /share-replicas/<replica_id>

POST /share-replicas/<share-id>
Body:
{
'share_replica':
{
'availability_zone':<availability_zone_id>,
'share_id':<share_id>,
'share_network_id':<share_network_id>
}
}
POST /share-replicas/<replica-id>/action
Body:
{'os-promote_replica': null}

DELETE /share-replicas/<replica-id>

==Policies==

policy.json - All replication group policies should default to the default policy
"share_replica:get_all": "rule:default",
"share_replica:show": "rule:default",
"share_replica:create" : "rule:default",
"share_replica:delete": "rule:default",
"share_replica:promote": "rule:default"

==Driver API==
Only a single replication driver method will be called at a time for a given share as they are locked by share id. Drivers should avoid raising errors to the manager for any of these methods. If the driver hits an issue it should attempt to return the correct model updates with the correct error states set to represent the actual state of the resources. A driver raising an exception will result in all replicas in the error status with the replica_state unchanged.

def create_replica(self, context, active_replica, new_replica,
access_rules, share_server=None):
"""Replicate the active replica to a new replica on this backend.
:param context: Current context
:param active_replica: A current active replica instance dictionary.
EXAMPLE:
.. code::
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS1',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'active',
'availability_zone_id': 'e2c2db5c-cb2f-4697-9966-c06fb200cb80',
'export_locations': [
<models.ShareInstanceExportLocations>._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param new_replica: The share replica dictionary.
EXAMPLE:
.. code::
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS2',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'out_of_sync',
'availability_zone_id': 'f6e146d0-65f0-11e5-9d70-feff819cdc9f',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': 'e6155221-ea00-49ef-abf9-9f89b7dd900a',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param access_rules: A list of access rules that other instances of
the share already obey.
EXAMPLE:
.. code::
[ {
'id': 'f0875f6f-766b-4865-8b41-cccb4cdf1676',
'deleted' = False,
'share_id' = 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'access_type' = 'ip',
'access_to' = '172.16.20.1',
'access_level' = 'rw',
}]
:param share_server: <models.ShareServer>._as_dict() or None,
Share server of the replica being created.
:return: (export_locations, replica_state)
export_locations is a list of paths and replica_state is one of
active, in-sync, out-of-sync or error.
A backend supporting 'writable' type replication should return
'active' as the replica_state.
Export locations should be in the same format as returned by a
share_create. This list may be empty or None.
EXAMPLE:
.. code::
[{'id': 'uuid', 'export_locations': ['export_path']}]
"""
raise NotImplementedError()

def delete_replica(self, context, active_replica, replica,
share_server=None):
"""Delete a replica.
:param context:Current context
:param active_replica: A current active replica instance dictionary.
EXAMPLE:
.. code::
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS1',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'active',
'availability_zone_id': 'e2c2db5c-cb2f-4697-9966-c06fb200cb80',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param replica: Dictionary of the share replica being deleted.
EXAMPLE:
.. code::
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS2',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'in_sync',
'availability_zone_id': 'f6e146d0-65f0-11e5-9d70-feff819cdc9f',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '53099868-65f1-11e5-9d70-feff819cdc9f',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param share_server: <models.ShareServer>._as_dict() or None,
Share server of the replica to be deleted.
:return: None.
"""
raise NotImplementedError()

def promote_replica(self, context, replica_list, replica, access_rules, share_server=None):
"""Promote a replica to 'active' replica state.
:param context:Current context
:param replica_list: List of all replicas for a particular share.
This list also contains the replica to be promoted. The 'active'
replica will have its 'replica_state' attr set to 'active'.
EXAMPLE:
.. code::
[
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'replica_state': 'in-sync',
...
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
'share_server': <models.ShareServer>._as_dict() or None,
},
{
'id': '10e49c3e-aca9-483b-8c2d-1c337b38d6af',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'replica_state': 'active',
...
'share_server_id': 'f63629b3-e126-4448-bec2-03f788f76094',
'share_server': <models.ShareServer>._as_dict() or None,
},
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'replica_state': 'in-sync',
...
'share_server_id': '07574742-67ea-4dfd-9844-9fbd8ada3d87',
'share_server': <models.ShareServer>._as_dict() or None,
},
...
]
:param replica: Dictionary of the replica to be promoted.
EXAMPLE:
.. code::
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS2',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'in_sync',
'availability_zone_id': 'f6e146d0-65f0-11e5-9d70-feff819cdc9f',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '07574742-67ea-4dfd-9844-9fbd8ada3d87',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param access_rules: A list of access rules that other instances of
the share already obey.
EXAMPLE:
.. code::
[ {
'id': 'f0875f6f-766b-4865-8b41-cccb4cdf1676',
'deleted' = False,
'share_id' = 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'access_type' = 'ip',
'access_to' = '172.16.20.1',
'access_level' = 'rw',
}]
:param share_server: <models.ShareServer>._as_dict() or None,
Share server of the replica to be promoted.
:return: updated_replica_list or None
The driver can return the updated list as in the request
parameter. Changes that will be updated to the Database are:
'export_locations' and 'replica_state'.
:raises Exception
This can be any exception derived from BaseException. This is
re-raised by the manager after some necessary cleanup. If the
driver raises an exception during promotion, it is assumed
that all of the replicas of the share are in an inconsistent
state. Recovery is only possible through the periodic update
call and/or administrator intervention to correct the 'status'
of the affected replicas if they become healthy again.
"""
raise NotImplementedError()

def update_replica_state(self, context, replica, share_server=None):
"""Update the replica_state of a replica.
Drivers should fix replication relationships that were broken if
possible inside this method.
:param context:Current context
:param replica: Dictionary of the replica being updated.
EXAMPLE:
.. code::
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS1',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'active',
'availability_zone_id': 'e2c2db5c-cb2f-4697-9966-c06fb200cb80',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
}
:param share_server: <models.ShareServer>._as_dict() or None
:return: replica_state
replica_state - a str value denoting the replica_state that the
replica can have. Valid values are 'in_sync' and 'out_of_sync'
or None (to leave the current replica_state unchanged).
"""
raise NotImplementedError()

==Manila Client==
share-replica-create Creates a replica
share-replica-delete Remove one or more share replicas.
share-replica-list List share replicas.
share-replica-promote Promote replica to active replica.
share-replica-show Show details about a replica.

==Example Conf==
[backend1]
replication_partners = backend2
[backend2]
replication_partners = backend1

==First-Party Driver==
DR will require first-party driver support before it can be promoted from an experimental feature for two main reasons.
# A feature is not truly open if only a single vendor can develop it due to it only being supported by a proprietary driver.
# In order to ensure the quality of a feature, it must be testable in upstream gate.

=FAQs=
# '''How do we deal with Network issues in multi-SVM replication?''' If we choose to make replication a single-svm-only feature, the share-network API doesn't need to change. In order to support replication with share-networks, we also need to modify the share-network create API which allows creation of share networks with a table of AZ-to-subnet mappings. This approach allows us to keep a single share-network per share (with associated security service) while allowing the tenant to specify enough information that each share instance can be attached to the appropriate network in each AZ. Multi-AZ share networks would also be useful for non-replicated use cases. 
# '''Do we support Pool-level Replication?''' Some vendors have suggested that certain backends can mirror groups of shares or whole pools more efficiently than individual shares. This design only addresses the mirroring of individual shares. In the future, we may allow allow replication of groups of shares, but only if those groups are contained within a single tenant and defined by the tenant. 
# '''From where to where do we allow replication? Is it intra-cloud or inter-cloud? Do we allow replication to something that's not managed by Manila?''' Intra-cloud. Replicating to something outside of Manila allows a bit more freedom, but with significantly less value, because there's practically nothing we can do to automate the failover/failback portion of a disaster. For use cases involving replication outside of Manila, we would need to involve other tools with more breadth/scope to manage the process. 
# '''Who configures the replication? The admin? The end user? The manila scheduler?''' The end user. In the original design we presumed that the actual replication relationships should be hidden from the end user, but this doesn't match well with the concept of AZs that we are adding to Manila. If the users need to have control over which AZ the primary copy of their data lives in, then they also need to control where the other copies live. This means that the administrator's job is to ensure that for any share type that is replicated, it can be replicated from any AZ to any other AZ. 
# '''Is replication supported only intra-region or inter-region as well? How about intra-AZ vs inter-AZ?''' This design is for intra-region only. Inter-region replication is clearly a feature we want, but it would look pretty different from intra-region replication; this design does not cover that. Intra-AZ replication is supported, however inter-AZ replication is recommended as AZs should be perceived as failure domains.

=Unanswered Questions=
# '''Is there no way to achieve non-disruptive failover?''' (bswartz) I would love to find out that our initial intuition here is wrong, because it would change a lot of aspects of the design. It's worth spending time to brainstorm and research possibilities in this area. So far the most promising ideas involve:
#*Using VirtFS to mediate filesystem access and achieving non-disruptive failover that way
#*Using some kind of agent inside the guests to mediate file access 
# '''Can a replica have a different share_type? There is a valid use case where a user would want to create a share replica on backend with different capabilities than the one the original share resides on. For instance, replicas might need to be on a less expensive backend. In that case, can the replica have a different share_type altogether?''' "Currently", we inherit the share_type of the share and believe that replication has to be on symmetric terms, where both backends have similar capabilities. 
#'''Can we allow the driver to restrict replication support between available backends? Backends may support replication to other compatible backends only. Hence, they must report some sort of information to the scheduler so that when creating a replica for an existing share, the scheduler would use that information to schedule the creation of the replica. What information should this be?''' (gouthamr) We're investigating including 'driver_class_name' in a ReplicationFilter, including the possibility of backend reported configuration, 'replication_partners'. 
#'''How are access rules persisted across replicas/share instances?''' 
#*Do all replicas have the same access rules applied? (gouthamr): This is Currently being pursued
#*Should access rules be applied only to "active" replicas? 
#'''How does migration affect replicated shares?'''
#*(gouthamr)In the Tokyo summit, it was decided that we currently disallow migration of a share that has replicas. The replication relationships must be broken and reestablished after the migration is complete in order to migrate a share with replicas. 
# '''Do we need an API to initiate a sync? (resync or force-update)''' This is a great idea and one certain use case is during planned failovers. (gouthamr) 
# '''How to recover from promotion failures?''' (gouthamr) Currently, if some exception occurs at the driver level, the replicas are rendered useless. Should we provide escape hatches to avoid complete loss in case these are recoverable exceptions? The administrator has the ability to 'reset-state' and affect the 'status' of any replica. Currently, there exists no API to 'reset-replica-state' for a replica.

= Examples =

These examples assume that the replicas exist in different AZs.

== Writable replication example ==

# Administrator sets up backends in AZs b1 and b2 that have capability replication_type=writeable
# Administrator creates a new share_type called foo
# Administrator sets replication_type=writeable extra spec on share type foo
# User creates new share of type foo in AZ b1
# Share is created with replication_type=writeable, and 1 active replica in AZ b1
# User grants access on share to client1 in AZ b1, obtains the export location of the replica, mounts the share on a client, and starts to write data
# User add new replica of share in AZ b2
# A second replica is created in AZ b2 which initially has state out_of_sync
# Shortly afterwards, the replica state changes to active (after the replica finishes syncing with the original copy)
# The user grants access on the share to client2 in AZ b2, obtains the export location of the new replica, mounts the share, and sees the same data that client1 wrote
# Client2 writes some data to the share, which is immediately visible to client1

== Readable replication example ==

# Administrator sets up backends in AZs b1 and b2 that have capability replication_type=readable
# Administrator creates a new share_type called bar
# Administrator sets replication_type=readable extra spec on share type bar
# User creates new share of type bar in AZ b1
# Share is created with replication_type=readable, and 1 active replica in AZ b1
# User grants access on share to client1 in AZ b1, obtains the export location of the replica, mounts the share on a client, and starts to write data
# User add new replica of share in AZ b2
# A second replica is created in AZ b2 which initially has state out_of_sync
# Shortly afterwards, the replica state changes to in_sync (after the replica finishes syncing with the original copy)
# The user grants access on the share to client2 in AZ b2, obtains the export location of the new replica, mounts the share, and sees the same data that client1 wrote
# Client2 cannot write data to the share but continues to see updates

== Failover/failback example ==

(Continued from above)
# An outage occurs in AZ b1
# Administrator sends out a bulletin about the outage to his users "the power transformer in b1 turned to slag, it will be 12 hours before it can be replaced, please bear with us, yada yada"
# User notices that his application on client1 is no longer running, and investigates
# User finds out that client1 is gone, and reads a bulletin from the admin explaining why
# User notes that the b1 replica is his share is still active and the b2 replica is in_sync, while the state of the share is available
# User calls set active replica to AZ b2 on his share
# The share goes to state replication_change and access to the share is briefly lost on client2
# The state of replica b2 changes to active and the state of replica b1 changes to out_of_sync after Manila fails to contact the original primary. The state of the share changes back to available, and access is restored on client2
# User starts his application on client2, and the application recovers from an apparent crash, with a consistent copy of the application's data
# User application is back up and running, disaster is averted
# Eventually maintenance on the b1 AZ is completed, and all of the equipment is reactivated
# Administrator sends out a bulletin to his users about the outage ending "we replaced the transformer with a better one, everything is back online, thanks for your understanding, yada yada"
# Manila notices the out_of_sync replica is reachable and initiates a resync, bringing the b1 replica in_sync within a short time
# After some time, the users reads the bulletin and observes that his share is being replicated again. He decides to intentionally move back to b1.
# User gracefully shuts down his application, and flushes I/O
# User calls set active replica to AZ b1 on his share
# The share goes to state replication_change and no disruption is observed because the application is shut down
# The state of replica b1 changes to active and the state of replica b2 changes to in_sync after Manila reverses the replication again. The state of the share changes back to available
# User starts his application on client1, and the application starts cleanly, having been shut down gracefully

=Implementation Progress=
===Core Work===
[https://blueprints.launchpad.net/manila/+spec/manila-share-replication Blueprint for Core API/Scheduler implementation]
[https://blueprints.launchpad.net/manila/+spec/manilaclient-share-replication Blueprint for Client implementation]

===Ex Driver Implementation===
[https://blueprints.launchpad.net/manila/+spec/netapp-cdot-single-svm-replication Blueprint for cDOT Driver Implementation]

=Older Design Pages=
* Replication Use Cases: https://wiki.openstack.org/wiki/Manila/Replication_Use_Cases
* Design Notes: https://wiki.openstack.org/wiki/Manila/Replication_Design_Notes

ThirdPartySystems/NetApp CI

2016-01-12T15:15:17Z

Alex Meade:

{{ThirdPartySystemInfo|name=NetApp CI
|account=netapp-ci
|contact= xdl-openstack-jenkins@netapp.com, andrew.kerr@netapp.com (akerr), alex.meade@netapp.com (ameade)
|intent=Test Openstack with NetApp Cinder and Manila drivers using storage arrays
|structure=Nodepool, Zuul, Jenkins, modified upstream components with local configuration
|method=Using Zuul to connect to Gerrit
|programs=Cinder, Manila
|status=Reporting on openstack/cinder and openstack/manila. System will be down January 22-29, 2016 due to Lab equipment moving.
|recheck=run-netapp-ci}}
'''Recheck trigger:''' "run-netapp-ci"

ThirdPartySystems/NetApp CI

2016-01-12T15:13:32Z

Alex Meade: Notice of downtime

{{ThirdPartySystemInfo|name=NetApp CI
|account=netapp-ci
|contact= xdl-openstack-jenkins@netapp.com, andrew.kerr@netapp.com (akerr), alex.meade@netapp.com (ameade)
|intent=Test Openstack with NetApp Cinder and Manila drivers using storage arrays
|structure=Nodepool, Zuul, Jenkins, modified upstream components with local configuration
|method=Using Zuul to connect to Gerrit
|programs=Cinder, Manila
|status=Reporting on openstack/cinder and openstack/manila. System will be down the week of January 18, 2016 due to Lab equipment moving.
|recheck=run-netapp-ci}}
'''Recheck trigger:''' "run-netapp-ci"

Manila/design/manila-mitaka-data-replication

2015-12-08T21:31:41Z

Alex Meade: /* Driver API */

=Manila Disaster Recovery / Data Replication Design=

==Introduction==

Replicated shares will be implemented in Manila without adding any new services or any new drivers. The code for creating replicated shares and for adding/removing replicas and other replication-related operations will all go into the existing share drivers. The most significant change required to allow this will be that share drivers will be responsible for communicating with ALL storage controllers necessary to achieve any replication tasks, even if that involves sending commands to other storage controllers in other AZs.

While we can't know how every storage controller works and how each vendor will implement replication, this approach should give driver writers all the flexibility needed to implement what they need with minimal added complexity in the Manila core. There are already examples of drivers reading config sections for other drivers when a operation requires communicating with 2 backends. The Manila Share Manager itself is expected to communicate all necessary backend details for share replicas that exist across AZs to a backend which it requests an operation upon.

Intra-AZ replication is supported, but in an ideal solution, AZs should be perceived as failure domains. So this feature tends to shine in an inter-AZ replication use case.

===Supported Replication Types===
There are 3 styles of DR that we would like to support in the long run:

# writable - Amazon EFS-style synchronously replicated shares where all replicas are writable. Promotion it not supported and not needed.
# readable - Mirror-style replication with a primary (writable) copy and one or more secondary (read-only) copies which can become writable after a promotion of the secondary.
# dr - Generalized replication with secondary copies that are inaccessible until after a promotion of one of the secondary copies.

===Replica States===
Each replica has a ''replica_state'' which has 4 possible values:

# active - All writable replicas are active.
# in_sync - Passive replica which is up to date with the active replica, and can be promoted to active.
# out_of_sync - Passive replica which has gone out of date, or new replica that is not yet up to date.
# error - Either the scheduler failed to schedule this replica or some irrecoverable damage occurred during promotion of a non-active replica. Drivers may set the ''replica_state'' to ''error'' if some irrecoverable damage to the replica is discovered at any point during its existence. (through the periodic ''replica_state'' update)
[[File:Manila instance replication states.png|center]]

===New share state===
replication_change - New transient state triggered by a change of the active replica. Access to the share is cut off while in this state.

===Promotion===
For '''readable''' and '''dr''' styles, we refer to the task of switching a non-active replica with the '''active''' replica as '''promotion'''. For the '''writable''' style of replication, '''promotion''' does not make sense since all replicas are '''active''' (writable) at all given points of time. Promotion of replicas with '''replica_state''' set to '''out_of_sync''' or '''error''' may be unsupported by the backend. However, we would like to allow the action as an administrator feature and such an attempt may be honored by backends if possible.

When multiple replicas exist, multiple replication relationships between shares may need to be redefined at the backend. If the driver fails at this stage, the replicas may be left at an inconsistent state. The manager will set all replica instances to have '''replica_state''' and '''status''' to '''error'''. Recovery from this stage would require administrator intervention.

==User Workflows==
====Creating a share supporting replication====
* Administrators can create a share type with extra-spec ''replication_type'', specifying the style of replication the backend supports.
* Users can use the share type to create a new share that allows/supports replication.
* A replicated share always starts out with one replica instance, the share itself. This should not be confused as the share having a replica already. The user can verify if the share has any replicas by requesting the details of the share.

====Creating a replica====
* POST to /share-replicas
User has to specify the share name/id of the share that is supposed to be replicated, an availability zone for the replica to exist in and optionally a share_network_id.
According to existing design, replicas of a share cannot exist in the same availability zone as the share itself.
* A newly created share_replica starts out in ''out_of_sync'' state and might transition to ''in_sync'' state when the driver reports this state.

====Listing and showing replicas====
* GET from /share-replicas
User can list all replicas and verify their ''status'' and ''replica_state''.
* GET from /share-replicas?share_id=<share_id>
User can list replicas for a particular share.
* GET from /share-replicas/<id>
User can view details of a replica.

====Promoting a non-active replica====
* POST to /share-replicas/action with body {'promote': None}
For replication types that permit promotion, the user can promote a replica with ''in_sync'' replica_state to ''active'' replica_state by initiating the ''promote_replica'' call.
* Only administrators can attempt promoting replicas with replica_state ''error'' or ''out_of_sync''.

====Deleting Replicas====
* DELETE to /share-replicas/<id>
The user can delete a replica. The last ''active'' replica cannot be deleted using this ''delete_replica'' call.

==System Workflows==
* Creating a share that supports replication
** Same process as creating a new share.
** The ''replication_type'' extra-spec from the share-type is copied over to the share''s data on the DB.
** Scheduler uses the ''replication_type'' extra-spec to filter available hosts and schedule the share on an appropriate backend that supports that specific style of ''replication_type''.

* Creating a replica:
** Create a share instance on the database.
** Update the share instance ''replica_state'' to ''out_of_sync''.
** Cast a call to the scheduler to find an appropriate host to schedule the replica on.
** In the scheduler, find weighted host for the replica.
*** If host cannot be chosen, update the replica's ''status'' and ''replica_state'' on the Database to ''error''. Throw an exception.
** Cast the create_share_replica call to the driver based on the weighted host selection.
** Prior to invoking the driver's call, collate information regarding existing ''active'' replica instance, existing share access rules and the replica's share_server, pass these to the driver while invoking ''create_replica''.
** The driver may return ''export_locations'' and ''replica_state''. If they are returned, the database is updated with these values.
** If the ''replication_type'' is ''writable'', the driver MUST return the ''replica_state'' set to ''active''.
** All the access rules for the new replica are set to ''active'' state on the database.
** The driver may throw an exception, update the ''status'' and ''replica_state'' of the replica to ''error''.

* Listing/showing shares:
** For shares that support replication, the result of listing/showing the share will have new fields, ''replication_type'': denoting the replication style supported and ''has_replicas'' to denote if the share has replicas.
** For replicated shares, the primary instance for the share must be preferred among those replicas that have a status ''replication_change'' or among those that have a replica_state set to ''active''.

* Listing replicas
** For listing, Grab share instances from the database that have a replica_state among {''in_sync'', ''active'', ''out_of_sync'', ''error''}.
** If share_id is provided, grab only share instances that belong to that share and have a replica_state among {''in_sync'', ''active'', ''out_of_sync'', ''error''}.
** Limits and offsets must be respected on the list calls.

* Promoting a non-active replica:
** If replica is already active, do nothing.
** Update the ''status'' of the replica being promoted to ''replication_change''.
** Grab all available replicas and invoke the appropriate driver''s promote_replica call by passing the available replicas and the new replica.
** If the driver throws an exception at this stage, there is a good chance that the replicas are somehow altered on the backend. Loop through the replicas and set their ''replica_state''s to ''error'' and leave the ''status'' unchanged. Also set the ''status'' of the replica that failed to promote to ''available'' as before this operation. The backend may choose to update the actual replica_state during the replica monitoring call.
** The driver may return an updated list of replicas. Update the ''export_locations'' and ''replica_state''s to the database.
** The ''status'' of the replica that was promoted should return to ''available'' from ''replication_change''.

* Periodic replica update:
** The share manager implements a looping call with default interval of 5 minutes to query from each driver and each backend the ''replica_state'' of all non-active replicas that are associated with them.
** The driver is allowed to set the ''replica_state'' to ''in_sync'', ''out_of_sync'', and in exceptional cases, ''error''.
** If the driver sets the ''replica_state'' of a replica to ''error'', it is assumed that some irrecoverable damage has occured to the replica instance. The ''status'' of the replica instance must be set to ''error'' as well.

* Deleting a share replica
** If the replica had no host, it is simply removed from the database.
** The ''status'' of the replica is set to ''deleting'' on the database and the appropriate driver method is called to delete the replica.
** If the driver fails to delete the replica, the ''status'' of the replica is updated to ''error_deleting'' on the database.
** If the driver returns without errors, the replica instance is removed from the database.

==Scheduler Impact==
* The host_manager must update the ''replication_type'' capability from the backend. The backends must report this capability at the host/pool level.
* The filter_scheduler must then match the ''replication_type'' capability with the capability that the replication share_type demands.
* The ''replication_type'' must be one of the styles mentioned [[#Supported Replication Styles|above]].

==DB Impact==
* Share Export Locations:
Preferred export locations will only be from the instances with ''replica_state'' set to ''active''.
def export_locations(self):
# TODO(gouthamr): Return AZ specific export locations for replicated
# shares
# NOTE(gouthamr): For a replicated share, export locations of the
# 'active' instances are taken, if 'available'.
all_export_locations = []
select_instances = list(filter(
lambda x: x['replica_state'] == constants.REPLICA_STATE_ACTIVE,
self.instances)) or self.instances
for instance in select_instances:
if instance['status'] == constants.STATUS_AVAILABLE:
for export_location in instance.export_locations:
all_export_locations.append(export_location['path'])
return all_export_locations

* Share Instance:
Preferred instance will be an instance with ''status'' set to ''replication_change'' or any instance with ''status'' set to ''available'' and ''replica_state'' set to ''active''.
def instance(self):
# NOTE(gouthamr): The order of preference: status 'replication_change',
# followed by 'available' and 'creating'. If replicated share and
# not undergoing a 'replication_change', only 'active' instances are
# preferred.
result = None
if len(self.instances) > 0:
order = [constants.STATUS_REPLICATION_CHANGE,
constants.STATUS_AVAILABLE, constants.STATUS_CREATING]
other_statuses = ([x['status'] for x in self.instances
if x['status'] not in order])
order.extend(other_statuses)
sorted_instances = sorted(
self.instances, key=lambda x: order.index(x['status']))
select_instances = sorted_instances
if (select_instances[0]['status'] !=
constants.STATUS_REPLICATION_CHANGE):
select_instances = (
list(filter(lambda x: x['replica_state'] ==
constants.REPLICA_STATE_ACTIVE,
sorted_instances)) or sorted_instances
)
result = select_instances[0]
return result

* New field on Share:
replication_type = Column(String(255), nullable=True)

* New field on ShareInstance
replica_state = Column(String(255), nullable=True)

==API Design==
GET /share-replicas/
GET /share-replicas?share-id=<share_id>
GET /share-replicas/<replica_id>

POST /share-replicas/<share-id>
Body:
{
'share_replica':
{
'availability_zone':<availability_zone_id>,
'share_id':<share_id>,
'share_network_id':<share_network_id>
}
}
POST /share-replicas/<replica-id>/action
Body:
{'os-promote_replica': null}

DELETE /share-replicas/<replica-id>

==Policies==

policy.json - All replication group policies should default to the default policy
"share_replica:get_all": "rule:default",
"share_replica:show": "rule:default",
"share_replica:create" : "rule:default",
"share_replica:delete": "rule:default",
"share_replica:promote": "rule:default"

==Driver API==
Only a single replication driver method will be called at a time for a given share as they are locked by share id. Drivers should avoid raising errors to the manager for any of these methods. If the driver hits an issue it should attempt to return the correct model updates with the correct error states set to represent the actual state of the resources. A driver raising an exception will result in all replicas in the error status with the replica_state unchanged.

def create_replica(self, context, active_replica, new_replica,
access_rules, share_server=None):
"""Replicate the active replica to a new replica on this backend.
:param context: Current context
:param active_replica: A current active replica instance dictionary.
EXAMPLE:
.. code::
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS1',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'active',
'availability_zone_id': 'e2c2db5c-cb2f-4697-9966-c06fb200cb80',
'export_locations': [
<models.ShareInstanceExportLocations>._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param new_replica: The share replica dictionary.
EXAMPLE:
.. code::
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS2',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'out_of_sync',
'availability_zone_id': 'f6e146d0-65f0-11e5-9d70-feff819cdc9f',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': 'e6155221-ea00-49ef-abf9-9f89b7dd900a',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param access_rules: A list of access rules that other instances of
the share already obey.
EXAMPLE:
.. code::
[ {
'id': 'f0875f6f-766b-4865-8b41-cccb4cdf1676',
'deleted' = False,
'share_id' = 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'access_type' = 'ip',
'access_to' = '172.16.20.1',
'access_level' = 'rw',
}]
:param share_server: <models.ShareServer>._as_dict() or None,
Share server of the replica being created.
:return: (export_locations, replica_state)
export_locations is a list of paths and replica_state is one of
active, in-sync, out-of-sync or error.
A backend supporting 'writable' type replication should return
'active' as the replica_state.
Export locations should be in the same format as returned by a
share_create. This list may be empty or None.
EXAMPLE:
.. code::
[{'id': 'uuid', 'export_locations': ['export_path']}]
"""
raise NotImplementedError()

def delete_replica(self, context, active_replica, replica,
share_server=None):
"""Delete a replica.
:param context:Current context
:param active_replica: A current active replica instance dictionary.
EXAMPLE:
.. code::
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS1',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'active',
'availability_zone_id': 'e2c2db5c-cb2f-4697-9966-c06fb200cb80',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param replica: Dictionary of the share replica being deleted.
EXAMPLE:
.. code::
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS2',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'in_sync',
'availability_zone_id': 'f6e146d0-65f0-11e5-9d70-feff819cdc9f',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '53099868-65f1-11e5-9d70-feff819cdc9f',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param share_server: <models.ShareServer>._as_dict() or None,
Share server of the replica to be deleted.
:return: None.
"""
raise NotImplementedError()

def promote_replica(self, context, replica_list, replica, access_rules, share_server=None):
"""Promote a replica to 'active' replica state.
:param context:Current context
:param replica_list: List of all replicas for a particular share.
This list also contains the replica to be promoted. The 'active'
replica will have its 'replica_state' attr set to 'active'.
EXAMPLE:
.. code::
[
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'replica_state': 'in-sync',
...
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
'share_server': <models.ShareServer>._as_dict() or None,
},
{
'id': '10e49c3e-aca9-483b-8c2d-1c337b38d6af',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'replica_state': 'active',
...
'share_server_id': 'f63629b3-e126-4448-bec2-03f788f76094',
'share_server': <models.ShareServer>._as_dict() or None,
},
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'replica_state': 'in-sync',
...
'share_server_id': '07574742-67ea-4dfd-9844-9fbd8ada3d87',
'share_server': <models.ShareServer>._as_dict() or None,
},
...
]
:param replica: Dictionary of the replica to be promoted.
EXAMPLE:
.. code::
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS2',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'in_sync',
'availability_zone_id': 'f6e146d0-65f0-11e5-9d70-feff819cdc9f',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '07574742-67ea-4dfd-9844-9fbd8ada3d87',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param access_rules: A list of access rules that other instances of
the share already obey.
EXAMPLE:
.. code::
[ {
'id': 'f0875f6f-766b-4865-8b41-cccb4cdf1676',
'deleted' = False,
'share_id' = 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'access_type' = 'ip',
'access_to' = '172.16.20.1',
'access_level' = 'rw',
}]
:param share_server: <models.ShareServer>._as_dict() or None,
Share server of the replica to be promoted.
:return: updated_replica_list or None
The driver can return the updated list as in the request
parameter. Changes that will be updated to the Database are:
'export_locations' and 'replica_state'.
:raises Exception
This can be any exception derived from BaseException. This is
re-raised by the manager after some necessary cleanup. If the
driver raises an exception during promotion, it is assumed
that all of the replicas of the share are in an inconsistent
state. Recovery is only possible through the periodic update
call and/or administrator intervention to correct the 'status'
of the affected replicas if they become healthy again.
"""
raise NotImplementedError()

def update_replica_state(self, context, replica, share_server=None):
"""Update the replica_state of a replica.
Drivers should fix replication relationships that were broken if
possible inside this method.
:param context:Current context
:param replica: Dictionary of the replica being updated.
EXAMPLE:
.. code::
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS1',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'active',
'availability_zone_id': 'e2c2db5c-cb2f-4697-9966-c06fb200cb80',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
}
:param share_server: <models.ShareServer>._as_dict() or None
:return: replica_state
replica_state - a str value denoting the replica_state that the
replica can have. Valid values are 'in_sync' and 'out_of_sync'
or None (to leave the current replica_state unchanged).
"""
raise NotImplementedError()

==Manila Client==
share-replica-create Creates a replica
share-replica-delete Remove one or more share replicas.
share-replica-list List share replicas.
share-replica-promote Promote replica to active replica.
share-replica-show Show details about a replica.

==First-Party Driver==
DR will require first-party driver support before it can be promoted from an experimental feature for two main reasons. 1. A feature is not truly open if only a single vendor can develop it due to it only being supported by a proprietary driver. 2. In order to ensure the quality of a feature, it must be testable in upstream gate.

=FAQs=
# '''How do we deal with Network issues in multi-SVM replication?''' If we choose to make replication a single-svm-only feature, the share-network API doesn't need to change. In order to support replication with share-networks, we also need to modify the share-network create API which allows creation of share networks with a table of AZ-to-subnet mappings. This approach allows us to keep a single share-network per share (with associated security service) while allowing the tenant to specify enough information that each share instance can be attached to the appropriate network in each AZ. Multi-AZ share networks would also be useful for non-replicated use cases. 
# '''Do we support Pool-level Replication?''' Some vendors have suggested that certain backends can mirror groups of shares or whole pools more efficiently than individual shares. This design only addresses the mirroring of individual shares. In the future, we may allow allow replication of groups of shares, but only if those groups are contained within a single tenant and defined by the tenant. 
# '''From where to where do we allow replication? Is it intra-cloud or inter-cloud? Do we allow replication to something that's not managed by Manila?''' Intra-cloud. Replicating to something outside of Manila allows a bit more freedom, but with significantly less value, because there's practically nothing we can do to automate the failover/failback portion of a disaster. For use cases involving replication outside of Manila, we would need to involve other tools with more breadth/scope to manage the process. 
# '''Who configures the replication? The admin? The end user? The manila scheduler?''' The end user. In the original design we presumed that the actual replication relationships should be hidden from the end user, but this doesn't match well with the concept of AZs that we are adding to Manila. If the users need to have control over which AZ the primary copy of their data lives in, then they also need to control where the other copies live. This means that the administrator's job is to ensure that for any share type that is replicated, it can be replicated from any AZ to any other AZ. 
# '''Is replication supported only intra-region or inter-region as well? How about intra-AZ vs inter-AZ?''' This design is for intra-region only. Inter-region replication is clearly a feature we want, but it would look pretty different from intra-region replication; this design does not cover that. Intra-AZ replication is supported, however inter-AZ replication is recommended as AZs should be perceived as failure domains.

=Unanswered Questions=
# '''Is there no way to achieve non-disruptive failover?''' (bswartz) I would love to find out that our initial intuition here is wrong, because it would change a lot of aspects of the design. It's worth spending time to brainstorm and research possibilities in this area. So far the most promising ideas involve:
#*Using VirtFS to mediate filesystem access and achieving non-disruptive failover that way
#*Using some kind of agent inside the guests to mediate file access 
# '''Can a replica have a different share_type? There is a valid use case where a user would want to create a share replica on backend with different capabilities than the one the original share resides on. For instance, replicas might need to be on a less expensive backend. In that case, can the replica have a different share_type altogether?''' "Currently", we inherit the share_type of the share and believe that replication has to be on symmetric terms, where both backends have similar capabilities. 
#'''Can we allow the driver to restrict replication support between available backends? Backends may support replication to other compatible backends only. Hence, they must report some sort of information to the scheduler so that when creating a replica for an existing share, the scheduler would use that information to schedule the creation of the replica. What information should this be?''' (gouthamr) We're investigating including 'driver_class_name' in a ReplicationFilter, including the possibility of backend reported configuration, 'replication_partners'. 
#'''How are access rules persisted across replicas/share instances?''' 
#*Do all replicas have the same access rules applied? (gouthamr): This is Currently being pursued
#*Should access rules be applied only to "active" replicas? 
#'''How does migration affect replicated shares?'''
#*(gouthamr)In the Tokyo summit, it was decided that we currently disallow migration of a share that has replicas. The replication relationships must be broken and reestablished after the migration is complete in order to migrate a share with replicas. 
# '''Do we need an API to initiate a sync? (resync or force-update)''' This is a great idea and one certain use case is during planned failovers. (gouthamr) 
# '''How to recover from promotion failures?''' (gouthamr) Currently, if some exception occurs at the driver level, the replicas are rendered useless. Should we provide escape hatches to avoid complete loss in case these are recoverable exceptions? The administrator has the ability to 'reset-state' and affect the 'status' of any replica. Currently, there exists no API to 'reset-replica-state' for a replica.

= Examples =

These examples assume that the replicas exist in different AZs.

== Writable replication example ==

# Administrator sets up backends in AZs b1 and b2 that have capability replication_type=writeable
# Administrator creates a new share_type called foo
# Administrator sets replication_type=writeable extra spec on share type foo
# User creates new share of type foo in AZ b1
# Share is created with replication_type=writeable, and 1 active replica in AZ b1
# User grants access on share to client1 in AZ b1, obtains the export location of the replica, mounts the share on a client, and starts to write data
# User add new replica of share in AZ b2
# A second replica is created in AZ b2 which initially has state out_of_sync
# Shortly afterwards, the replica state changes to active (after the replica finishes syncing with the original copy)
# The user grants access on the share to client2 in AZ b2, obtains the export location of the new replica, mounts the share, and sees the same data that client1 wrote
# Client2 writes some data to the share, which is immediately visible to client1

== Readable replication example ==

# Administrator sets up backends in AZs b1 and b2 that have capability replication_type=readable
# Administrator creates a new share_type called bar
# Administrator sets replication_type=readable extra spec on share type bar
# User creates new share of type bar in AZ b1
# Share is created with replication_type=readable, and 1 active replica in AZ b1
# User grants access on share to client1 in AZ b1, obtains the export location of the replica, mounts the share on a client, and starts to write data
# User add new replica of share in AZ b2
# A second replica is created in AZ b2 which initially has state out_of_sync
# Shortly afterwards, the replica state changes to in_sync (after the replica finishes syncing with the original copy)
# The user grants access on the share to client2 in AZ b2, obtains the export location of the new replica, mounts the share, and sees the same data that client1 wrote
# Client2 cannot write data to the share but continues to see updates

== Failover/failback example ==

(Continued from above)
# An outage occurs in AZ b1
# Administrator sends out a bulletin about the outage to his users "the power transformer in b1 turned to slag, it will be 12 hours before it can be replaced, please bear with us, yada yada"
# User notices that his application on client1 is no longer running, and investigates
# User finds out that client1 is gone, and reads a bulletin from the admin explaining why
# User notes that the b1 replica is his share is still active and the b2 replica is in_sync, while the state of the share is available
# User calls set active replica to AZ b2 on his share
# The share goes to state replication_change and access to the share is briefly lost on client2
# The state of replica b2 changes to active and the state of replica b1 changes to out_of_sync after Manila fails to contact the original primary. The state of the share changes back to available, and access is restored on client2
# User starts his application on client2, and the application recovers from an apparent crash, with a consistent copy of the application's data
# User application is back up and running, disaster is averted
# Eventually maintenance on the b1 AZ is completed, and all of the equipment is reactivated
# Administrator sends out a bulletin to his users about the outage ending "we replaced the transformer with a better one, everything is back online, thanks for your understanding, yada yada"
# Manila notices the out_of_sync replica is reachable and initiates a resync, bringing the b1 replica in_sync within a short time
# After some time, the users reads the bulletin and observes that his share is being replicated again. He decides to intentionally move back to b1.
# User gracefully shuts down his application, and flushes I/O
# User calls set active replica to AZ b1 on his share
# The share goes to state replication_change and no disruption is observed because the application is shut down
# The state of replica b1 changes to active and the state of replica b2 changes to in_sync after Manila reverses the replication again. The state of the share changes back to available
# User starts his application on client1, and the application starts cleanly, having been shut down gracefully

=Implementation Progress=
===Core Work===
[https://blueprints.launchpad.net/manila/+spec/manila-share-replication Blueprint for Core API/Scheduler implementation]
[https://blueprints.launchpad.net/manila/+spec/manilaclient-share-replication Blueprint for Client implementation]

===Ex Driver Implementation===
[https://blueprints.launchpad.net/manila/+spec/netapp-cdot-single-svm-replication Blueprint for cDOT Driver Implementation]

=Older Design Pages=
* Replication Use Cases: https://wiki.openstack.org/wiki/Manila/Replication_Use_Cases
* Design Notes: https://wiki.openstack.org/wiki/Manila/Replication_Design_Notes

Manila/design/manila-mitaka-data-replication

2015-12-07T19:08:26Z

Alex Meade: /* Manila Disaster Recovery / Data Replication Design */

=Manila Disaster Recovery / Data Replication Design=

==Introduction==

Replicated shares will be implemented in Manila without adding any new services or any new drivers. The code for creating replicated shares and for adding/removing replicas and other replication-related operations will all go into the existing share drivers. The most significant change required to allow this will be that share drivers will be responsible for communicating with ALL storage controllers necessary to achieve any replication tasks, even if that involves sending commands to other storage controllers in other AZs.

While we can't know how every storage controller works and how each vendor will implement replication, this approach should give driver writers all the flexibility needed to implement what they need with minimal added complexity in the Manila core. There are already examples of drivers reading config sections for other drivers when a operation requires communicating with 2 backends. The Manila Share Manager itself is expected to communicate all necessary backend details for share replicas that exist across AZs to a backend which it requests an operation upon.

Intra-AZ replication is supported, but in an ideal solution, AZs should be perceived as failure domains. So this feature tends to shine in an inter-AZ replication use case.

===Supported Replication Types===
There are 3 styles of DR that we would like to support in the long run:

# writable - Amazon EFS-style synchronously replicated shares where all replicas are writable. Promotion it not supported and not needed.
# readable - Mirror-style replication with a primary (writable) copy and one or more secondary (read-only) copies which can become writable after a promotion of the secondary.
# dr - Generalized replication with secondary copies that are inaccessible until after a promotion of one of the secondary copies.

===Replica States===
Each replica has a ''replica_state'' which has 4 possible values:

# active - All writable replicas are active.
# in_sync - Passive replica which is up to date with the active replica, and can be promoted to active.
# out_of_sync - Passive replica which has gone out of date, or new replica that is not yet up to date.
# error - Either the scheduler failed to schedule this replica or some irrecoverable damage occurred during promotion of a non-active replica. Drivers may set the ''replica_state'' to ''error'' if some irrecoverable damage to the replica is discovered at any point during its existence. (through the periodic ''replica_state'' update)
[[File:Manila instance replication states.png|center]]

===New share state===
replication_change - New transient state triggered by a change of the active replica. Access to the share is cut off while in this state.

===Promotion===
For '''readable''' and '''dr''' styles, we refer to the task of switching a non-active replica with the '''active''' replica as '''promotion'''. For the '''writable''' style of replication, '''promotion''' does not make sense since all replicas are '''active''' (writable) at all given points of time. Promotion of replicas with '''replica_state''' set to '''out_of_sync''' or '''error''' may be unsupported by the backend. However, we would like to allow the action as an administrator feature and such an attempt may be honored by backends if possible.

When multiple replicas exist, multiple replication relationships between shares may need to be redefined at the backend. If the driver fails at this stage, the replicas may be left at an inconsistent state. The manager will set all replica instances to have '''replica_state''' and '''status''' to '''error'''. Recovery from this stage would require administrator intervention.

==User Workflows==
====Creating a share supporting replication====
* Administrators can create a share type with extra-spec ''replication_type'', specifying the style of replication the backend supports.
* Users can use the share type to create a new share that allows/supports replication.
* A replicated share always starts out with one replica instance, the share itself. This should not be confused as the share having a replica already. The user can verify if the share has any replicas by requesting the details of the share.

====Creating a replica====
* POST to /share-replicas
User has to specify the share name/id of the share that is supposed to be replicated, an availability zone for the replica to exist in and optionally a share_network_id.
According to existing design, replicas of a share cannot exist in the same availability zone as the share itself.
* A newly created share_replica starts out in ''out_of_sync'' state and might transition to ''in_sync'' state when the driver reports this state.

====Listing and showing replicas====
* GET from /share-replicas
User can list all replicas and verify their ''status'' and ''replica_state''.
* GET from /share-replicas?share_id=<share_id>
User can list replicas for a particular share.
* GET from /share-replicas/<id>
User can view details of a replica.

====Promoting a non-active replica====
* POST to /share-replicas/action with body {'promote': None}
For replication types that permit promotion, the user can promote a replica with ''in_sync'' replica_state to ''active'' replica_state by initiating the ''promote_replica'' call.
* Only administrators can attempt promoting replicas with replica_state ''error'' or ''out_of_sync''.

====Deleting Replicas====
* DELETE to /share-replicas/<id>
The user can delete a replica. The last ''active'' replica cannot be deleted using this ''delete_replica'' call.

==System Workflows==
* Creating a share that supports replication
** Same process as creating a new share.
** The ''replication_type'' extra-spec from the share-type is copied over to the share''s data on the DB.
** Scheduler uses the ''replication_type'' extra-spec to filter available hosts and schedule the share on an appropriate backend that supports that specific style of ''replication_type''.

* Creating a replica:
** Create a share instance on the database.
** Update the share instance ''replica_state'' to ''out_of_sync''.
** Cast a call to the scheduler to find an appropriate host to schedule the replica on.
** In the scheduler, find weighted host for the replica.
*** If host cannot be chosen, update the replica's ''status'' and ''replica_state'' on the Database to ''error''. Throw an exception.
** Cast the create_share_replica call to the driver based on the weighted host selection.
** Prior to invoking the driver's call, collate information regarding existing ''active'' replica instance, existing share access rules and the replica's share_server, pass these to the driver while invoking ''create_replica''.
** The driver may return ''export_locations'' and ''replica_state''. If they are returned, the database is updated with these values.
** If the ''replication_type'' is ''writable'', the driver MUST return the ''replica_state'' set to ''active''.
** All the access rules for the new replica are set to ''active'' state on the database.
** The driver may throw an exception, update the ''status'' and ''replica_state'' of the replica to ''error''.

* Listing/showing shares:
** For shares that support replication, the result of listing/showing the share will have new fields, ''replication_type'': denoting the replication style supported and ''has_replicas'' to denote if the share has replicas.
** For replicated shares, the primary instance for the share must be preferred among those replicas that have a status ''replication_change'' or among those that have a replica_state set to ''active''.

* Listing replicas
** For listing, Grab share instances from the database that have a replica_state among {''in_sync'', ''active'', ''out_of_sync'', ''error''}.
** If share_id is provided, grab only share instances that belong to that share and have a replica_state among {''in_sync'', ''active'', ''out_of_sync'', ''error''}.
** Limits and offsets must be respected on the list calls.

* Promoting a non-active replica:
** If replica is already active, do nothing.
** Update the ''status'' of the replica being promoted to ''replication_change''.
** Grab all available replicas and invoke the appropriate driver''s promote_replica call by passing the available replicas and the new replica.
** If the driver throws an exception at this stage, there is a good chance that the replicas are somehow altered on the backend. Loop through the replicas and set their ''replica_state''s to ''error'' and leave the ''status'' unchanged. Also set the ''status'' of the replica that failed to promote to ''available'' as before this operation. The backend may choose to update the actual replica_state during the replica monitoring call.
** The driver may return an updated list of replicas. Update the ''export_locations'' and ''replica_state''s to the database.
** The ''status'' of the replica that was promoted should return to ''available'' from ''replication_change''.

* Periodic replica update:
** The share manager implements a looping call with default interval of 5 minutes to query from each driver and each backend the ''replica_state'' of all non-active replicas that are associated with them.
** The driver is allowed to set the ''replica_state'' to ''in_sync'', ''out_of_sync'', and in exceptional cases, ''error''.
** If the driver sets the ''replica_state'' of a replica to ''error'', it is assumed that some irrecoverable damage has occured to the replica instance. The ''status'' of the replica instance must be set to ''error'' as well.

* Deleting a share replica
** If the replica had no host, it is simply removed from the database.
** The ''status'' of the replica is set to ''deleting'' on the database and the appropriate driver method is called to delete the replica.
** If the driver fails to delete the replica, the ''status'' of the replica is updated to ''error_deleting'' on the database.
** If the driver returns without errors, the replica instance is removed from the database.

==Scheduler Impact==
* The host_manager must update the ''replication_type'' capability from the backend. The backends must report this capability at the host/pool level.
* The filter_scheduler must then match the ''replication_type'' capability with the capability that the replication share_type demands.
* The ''replication_type'' must be one of the styles mentioned [[#Supported Replication Styles|above]].

==DB Impact==
* Share Export Locations:
Preferred export locations will only be from the instances with ''replica_state'' set to ''active''.
def export_locations(self):
# TODO(gouthamr): Return AZ specific export locations for replicated
# shares
# NOTE(gouthamr): For a replicated share, export locations of the
# 'active' instances are taken, if 'available'.
all_export_locations = []
select_instances = list(filter(
lambda x: x['replica_state'] == constants.REPLICA_STATE_ACTIVE,
self.instances)) or self.instances
for instance in select_instances:
if instance['status'] == constants.STATUS_AVAILABLE:
for export_location in instance.export_locations:
all_export_locations.append(export_location['path'])
return all_export_locations

* Share Instance:
Preferred instance will be an instance with ''status'' set to ''replication_change'' or any instance with ''status'' set to ''available'' and ''replica_state'' set to ''active''.
def instance(self):
# NOTE(gouthamr): The order of preference: status 'replication_change',
# followed by 'available' and 'creating'. If replicated share and
# not undergoing a 'replication_change', only 'active' instances are
# preferred.
result = None
if len(self.instances) > 0:
order = [constants.STATUS_REPLICATION_CHANGE,
constants.STATUS_AVAILABLE, constants.STATUS_CREATING]
other_statuses = ([x['status'] for x in self.instances
if x['status'] not in order])
order.extend(other_statuses)
sorted_instances = sorted(
self.instances, key=lambda x: order.index(x['status']))
select_instances = sorted_instances
if (select_instances[0]['status'] !=
constants.STATUS_REPLICATION_CHANGE):
select_instances = (
list(filter(lambda x: x['replica_state'] ==
constants.REPLICA_STATE_ACTIVE,
sorted_instances)) or sorted_instances
)
result = select_instances[0]
return result

* New field on Share:
replication_type = Column(String(255), nullable=True)

* New field on ShareInstance
replica_state = Column(String(255), nullable=True)

==API Design==
GET /share-replicas/
GET /share-replicas?share-id=<share_id>
GET /share-replicas/<replica_id>

POST /share-replicas/<share-id>
Body:
{
'share_replica':
{
'availability_zone':<availability_zone_id>,
'share_id':<share_id>,
'share_network_id':<share_network_id>
}
}
POST /share-replicas/<replica-id>/action
Body:
{'os-promote_replica': null}

DELETE /share-replicas/<replica-id>

==Policies==

policy.json - All replication group policies should default to the default policy
"share_replica:get_all": "rule:default",
"share_replica:show": "rule:default",
"share_replica:create" : "rule:default",
"share_replica:delete": "rule:default",
"share_replica:promote": "rule:default"

==Driver API==
Only a single replication driver method will be called at a time for a given share as they are locked by share id.

def create_replica(self, context, active_replica, new_replica,
access_rules, share_server=None):
"""Replicate the active replica to a new replica on this backend.
:param context: Current context
:param active_replica: A current active replica instance dictionary.
EXAMPLE:
.. code::
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS1',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'active',
'availability_zone_id': 'e2c2db5c-cb2f-4697-9966-c06fb200cb80',
'export_locations': [
<models.ShareInstanceExportLocations>._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param new_replica: The share replica dictionary.
EXAMPLE:
.. code::
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS2',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'out_of_sync',
'availability_zone_id': 'f6e146d0-65f0-11e5-9d70-feff819cdc9f',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': 'e6155221-ea00-49ef-abf9-9f89b7dd900a',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param access_rules: A list of access rules that other instances of
the share already obey.
EXAMPLE:
.. code::
[ {
'id': 'f0875f6f-766b-4865-8b41-cccb4cdf1676',
'deleted' = False,
'share_id' = 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'access_type' = 'ip',
'access_to' = '172.16.20.1',
'access_level' = 'rw',
}]
:param share_server: <models.ShareServer>._as_dict() or None,
Share server of the replica being created.
:return: (export_locations, replica_state)
export_locations is a list of paths and replica_state is one of
active, in-sync, out-of-sync or error.
A backend supporting 'writable' type replication should return
'active' as the replica_state.
Export locations should be in the same format as returned by a
share_create. This list may be empty or None.
EXAMPLE:
.. code::
[{'id': 'uuid', 'export_locations': ['export_path']}]
"""
raise NotImplementedError()

def delete_replica(self, context, active_replica, replica,
share_server=None):
"""Delete a replica.
:param context:Current context
:param active_replica: A current active replica instance dictionary.
EXAMPLE:
.. code::
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS1',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'active',
'availability_zone_id': 'e2c2db5c-cb2f-4697-9966-c06fb200cb80',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param replica: Dictionary of the share replica being deleted.
EXAMPLE:
.. code::
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS2',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'in_sync',
'availability_zone_id': 'f6e146d0-65f0-11e5-9d70-feff819cdc9f',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '53099868-65f1-11e5-9d70-feff819cdc9f',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param share_server: <models.ShareServer>._as_dict() or None,
Share server of the replica to be deleted.
:return: None.
"""
raise NotImplementedError()

def promote_replica(self, context, replica_list, replica, access_rules, share_server=None):
"""Promote a replica to 'active' replica state.
:param context:Current context
:param replica_list: List of all replicas for a particular share.
This list also contains the replica to be promoted. The 'active'
replica will have its 'replica_state' attr set to 'active'.
EXAMPLE:
.. code::
[
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'replica_state': 'in-sync',
...
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
'share_server': <models.ShareServer>._as_dict() or None,
},
{
'id': '10e49c3e-aca9-483b-8c2d-1c337b38d6af',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'replica_state': 'active',
...
'share_server_id': 'f63629b3-e126-4448-bec2-03f788f76094',
'share_server': <models.ShareServer>._as_dict() or None,
},
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'replica_state': 'in-sync',
...
'share_server_id': '07574742-67ea-4dfd-9844-9fbd8ada3d87',
'share_server': <models.ShareServer>._as_dict() or None,
},
...
]
:param replica: Dictionary of the replica to be promoted.
EXAMPLE:
.. code::
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS2',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'in_sync',
'availability_zone_id': 'f6e146d0-65f0-11e5-9d70-feff819cdc9f',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '07574742-67ea-4dfd-9844-9fbd8ada3d87',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param access_rules: A list of access rules that other instances of
the share already obey.
EXAMPLE:
.. code::
[ {
'id': 'f0875f6f-766b-4865-8b41-cccb4cdf1676',
'deleted' = False,
'share_id' = 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'access_type' = 'ip',
'access_to' = '172.16.20.1',
'access_level' = 'rw',
}]
:param share_server: <models.ShareServer>._as_dict() or None,
Share server of the replica to be promoted.
:return: updated_replica_list or None
The driver can return the updated list as in the request
parameter. Changes that will be updated to the Database are:
'export_locations' and 'replica_state'.
:raises Exception
This can be any exception derived from BaseException. This is
re-raised by the manager after some necessary cleanup. If the
driver raises an exception during promotion, it is assumed
that all of the replicas of the share are in an inconsistent
state. Recovery is only possible through the periodic update
call and/or administrator intervention to correct the 'status'
of the affected replicas if they become healthy again.
"""
raise NotImplementedError()

def update_replica_state(self, context, replica, share_server=None):
"""Update the replica_state of a replica.
Drivers should fix replication relationships that were broken if
possible inside this method.
:param context:Current context
:param replica: Dictionary of the replica being updated.
EXAMPLE:
.. code::
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS1',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'active',
'availability_zone_id': 'e2c2db5c-cb2f-4697-9966-c06fb200cb80',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
}
:param share_server: <models.ShareServer>._as_dict() or None
:return: replica_state
replica_state - a str value denoting the replica_state that the
replica can have. Valid values are 'in_sync' and 'out_of_sync'
or None (to leave the current replica_state unchanged).
"""
raise NotImplementedError()

==Manila Client==
share-replica-create Creates a replica
share-replica-delete Remove one or more share replicas.
share-replica-list List share replicas.
share-replica-promote Promote replica to active replica.
share-replica-show Show details about a replica.

==First-Party Driver==
DR will require first-party driver support before it can be promoted from an experimental feature for two main reasons. 1. A feature is not truly open if only a single vendor can develop it due to it only being supported by a proprietary driver. 2. In order to ensure the quality of a feature, it must be testable in upstream gate.

=FAQs=
# '''How do we deal with Network issues in multi-SVM replication?''' If we choose to make replication a single-svm-only feature, the share-network API doesn't need to change. In order to support replication with share-networks, we also need to modify the share-network create API which allows creation of share networks with a table of AZ-to-subnet mappings. This approach allows us to keep a single share-network per share (with associated security service) while allowing the tenant to specify enough information that each share instance can be attached to the appropriate network in each AZ. Multi-AZ share networks would also be useful for non-replicated use cases. 
# '''Do we support Pool-level Replication?''' Some vendors have suggested that certain backends can mirror groups of shares or whole pools more efficiently than individual shares. This design only addresses the mirroring of individual shares. In the future, we may allow allow replication of groups of shares, but only if those groups are contained within a single tenant and defined by the tenant. 
# '''From where to where do we allow replication? Is it intra-cloud or inter-cloud? Do we allow replication to something that's not managed by Manila?''' Intra-cloud. Replicating to something outside of Manila allows a bit more freedom, but with significantly less value, because there's practically nothing we can do to automate the failover/failback portion of a disaster. For use cases involving replication outside of Manila, we would need to involve other tools with more breadth/scope to manage the process. 
# '''Who configures the replication? The admin? The end user? The manila scheduler?''' The end user. In the original design we presumed that the actual replication relationships should be hidden from the end user, but this doesn't match well with the concept of AZs that we are adding to Manila. If the users need to have control over which AZ the primary copy of their data lives in, then they also need to control where the other copies live. This means that the administrator's job is to ensure that for any share type that is replicated, it can be replicated from any AZ to any other AZ. 
# '''Is replication supported only intra-region or inter-region as well? How about intra-AZ vs inter-AZ?''' This design is for intra-region only. Inter-region replication is clearly a feature we want, but it would look pretty different from intra-region replication; this design does not cover that. Intra-AZ replication is supported, however inter-AZ replication is recommended as AZs should be perceived as failure domains.

=Unanswered Questions=
# '''Is there no way to achieve non-disruptive failover?''' (bswartz) I would love to find out that our initial intuition here is wrong, because it would change a lot of aspects of the design. It's worth spending time to brainstorm and research possibilities in this area. So far the most promising ideas involve:
#*Using VirtFS to mediate filesystem access and achieving non-disruptive failover that way
#*Using some kind of agent inside the guests to mediate file access 
# '''Can a replica have a different share_type? There is a valid use case where a user would want to create a share replica on backend with different capabilities than the one the original share resides on. For instance, replicas might need to be on a less expensive backend. In that case, can the replica have a different share_type altogether?''' "Currently", we inherit the share_type of the share and believe that replication has to be on symmetric terms, where both backends have similar capabilities. 
#'''Can we allow the driver to restrict replication support between available backends? Backends may support replication to other compatible backends only. Hence, they must report some sort of information to the scheduler so that when creating a replica for an existing share, the scheduler would use that information to schedule the creation of the replica. What information should this be?''' (gouthamr) We're investigating including 'driver_class_name' in a ReplicationFilter, including the possibility of backend reported configuration, 'replication_partners'. 
#'''How are access rules persisted across replicas/share instances?''' 
#*Do all replicas have the same access rules applied? (gouthamr): This is Currently being pursued
#*Should access rules be applied only to "active" replicas? 
#'''How does migration affect replicated shares?'''
#*(gouthamr)In the Tokyo summit, it was decided that we currently disallow migration of a share that has replicas. The replication relationships must be broken and reestablished after the migration is complete in order to migrate a share with replicas. 
# '''Do we need an API to initiate a sync? (resync or force-update)''' This is a great idea and one certain use case is during planned failovers. (gouthamr) 
# '''How to recover from promotion failures?''' (gouthamr) Currently, if some exception occurs at the driver level, the replicas are rendered useless. Should we provide escape hatches to avoid complete loss in case these are recoverable exceptions? The administrator has the ability to 'reset-state' and affect the 'status' of any replica. Currently, there exists no API to 'reset-replica-state' for a replica.

= Examples =

These examples assume that the replicas exist in different AZs.

== Writable replication example ==

# Administrator sets up backends in AZs b1 and b2 that have capability replication_type=writeable
# Administrator creates a new share_type called foo
# Administrator sets replication_type=writeable extra spec on share type foo
# User creates new share of type foo in AZ b1
# Share is created with replication_type=writeable, and 1 active replica in AZ b1
# User grants access on share to client1 in AZ b1, obtains the export location of the replica, mounts the share on a client, and starts to write data
# User add new replica of share in AZ b2
# A second replica is created in AZ b2 which initially has state out_of_sync
# Shortly afterwards, the replica state changes to active (after the replica finishes syncing with the original copy)
# The user grants access on the share to client2 in AZ b2, obtains the export location of the new replica, mounts the share, and sees the same data that client1 wrote
# Client2 writes some data to the share, which is immediately visible to client1

== Readable replication example ==

# Administrator sets up backends in AZs b1 and b2 that have capability replication_type=readable
# Administrator creates a new share_type called bar
# Administrator sets replication_type=readable extra spec on share type bar
# User creates new share of type bar in AZ b1
# Share is created with replication_type=readable, and 1 active replica in AZ b1
# User grants access on share to client1 in AZ b1, obtains the export location of the replica, mounts the share on a client, and starts to write data
# User add new replica of share in AZ b2
# A second replica is created in AZ b2 which initially has state out_of_sync
# Shortly afterwards, the replica state changes to in_sync (after the replica finishes syncing with the original copy)
# The user grants access on the share to client2 in AZ b2, obtains the export location of the new replica, mounts the share, and sees the same data that client1 wrote
# Client2 cannot write data to the share but continues to see updates

== Failover/failback example ==

(Continued from above)
# An outage occurs in AZ b1
# Administrator sends out a bulletin about the outage to his users "the power transformer in b1 turned to slag, it will be 12 hours before it can be replaced, please bear with us, yada yada"
# User notices that his application on client1 is no longer running, and investigates
# User finds out that client1 is gone, and reads a bulletin from the admin explaining why
# User notes that the b1 replica is his share is still active and the b2 replica is in_sync, while the state of the share is available
# User calls set active replica to AZ b2 on his share
# The share goes to state replication_change and access to the share is briefly lost on client2
# The state of replica b2 changes to active and the state of replica b1 changes to out_of_sync after Manila fails to contact the original primary. The state of the share changes back to available, and access is restored on client2
# User starts his application on client2, and the application recovers from an apparent crash, with a consistent copy of the application's data
# User application is back up and running, disaster is averted
# Eventually maintenance on the b1 AZ is completed, and all of the equipment is reactivated
# Administrator sends out a bulletin to his users about the outage ending "we replaced the transformer with a better one, everything is back online, thanks for your understanding, yada yada"
# Manila notices the out_of_sync replica is reachable and initiates a resync, bringing the b1 replica in_sync within a short time
# After some time, the users reads the bulletin and observes that his share is being replicated again. He decides to intentionally move back to b1.
# User gracefully shuts down his application, and flushes I/O
# User calls set active replica to AZ b1 on his share
# The share goes to state replication_change and no disruption is observed because the application is shut down
# The state of replica b1 changes to active and the state of replica b2 changes to in_sync after Manila reverses the replication again. The state of the share changes back to available
# User starts his application on client1, and the application starts cleanly, having been shut down gracefully

=Implementation Progress=
===Core Work===
[https://blueprints.launchpad.net/manila/+spec/manila-share-replication Blueprint for Core API/Scheduler implementation]
[https://blueprints.launchpad.net/manila/+spec/manilaclient-share-replication Blueprint for Client implementation]

===Ex Driver Implementation===
[https://blueprints.launchpad.net/manila/+spec/netapp-cdot-single-svm-replication Blueprint for cDOT Driver Implementation]

=Older Design Pages=
* Replication Use Cases: https://wiki.openstack.org/wiki/Manila/Replication_Use_Cases
* Design Notes: https://wiki.openstack.org/wiki/Manila/Replication_Design_Notes

Manila/design/manila-mitaka-data-replication

2015-12-04T15:51:12Z

Alex Meade: /* Driver API */

=Manila Disaster Recovery / Data Replication Design=

==Introduction==

Replicated shares will be implemented in Manila without adding any new services or any new drivers. The code for creating replicated shares and for adding/removing replicas and other replication-related operations will all go into the existing share drivers. The most significant change required to allow this will be that share drivers will be responsible for communicating with ALL storage controllers necessary to achieve any replication tasks, even if that involves sending commands to other storage controllers in other AZs.

While we can't know how every storage controller works and how each vendor will implement replication, this approach should give driver writers all the flexibility needed to implement what they need with minimal added complexity in the Manila core. There are already examples of drivers reading config sections for other drivers when a operation requires communicating with 2 backends. The Manila Share Manager itself is expected to communicate all necessary backend details for share replicas that exist across AZs to a backend which it requests an operation upon.

Intra-AZ replication is supported, but in an ideal solution, AZs should be perceived as failure domains. So this feature tends to shine in an inter-AZ replication use case.

===Supported Replication Types===
There are 3 styles of DR that we would like to support in the long run:

# writable - Amazon EFS-style synchronously replicated shares where all replicas are writable. Promotion it not supported and not needed.
# readable - Mirror-style replication with a primary (writable) copy and one or more secondary (read-only) copies which can become writable after a promotion of the secondary.
# dr - Generalized replication with secondary copies that are inaccessible until after a promotion of one of the secondary copies.

===Replica States===
Each replica has a ''replica_state'' which has 4 possible values:

# active - All writable replicas are active.
# in_sync - Passive replica which is up to date with the active replica, and can be promoted to active.
# out_of_sync - Passive replica which has gone out of date, or new replica that is not yet up to date.
# error - Either the scheduler failed to schedule this replica or some irrecoverable damage occurred during promotion of a non-active replica. Drivers may set the ''replica_state'' to ''error'' if some irrecoverable damage to the replica is discovered at any point during its existence. (through the periodic ''replica_state'' update)
[[File:Manila instance replication states.png|center]]

===New share state===
replication_change - New transient state triggered by a change of the active replica. Access to the share is cut off while in this state.

===Promotion===
For '''readable''' and '''dr''' styles, we refer to the task of switching a non-active replica with the '''active''' replica as '''promotion'''. For the '''writable''' style of replication, '''promotion''' does not make sense since all replicas are '''active''' (writable) at all given points of time. Promotion of replicas with '''replica_state''' set to '''out_of_sync''' or '''error''' may be unsupported by the backend. However, we would like to allow the action as an administrator feature and such an attempt may be honored by backends if possible.

When multiple replicas exist, multiple replication relationships between shares may need to be redefined at the backend. If the driver fails at this stage, the replicas may be left at an inconsistent state. The manager will set all replica instances to have '''replica_state''' and '''status''' to '''error'''. Recovery from this stage would require administrator intervention.

==User Workflows==
====Creating a share supporting replication====
* Administrators can create a share type with extra-spec ''replication_type'', specifying the style of replication the backend supports.
* Users can use the share type to create a new share that allows/supports replication.
* A replicated share always starts out with one replica instance, the share itself. This should not be confused as the share having a replica already. The user can verify if the share has any replicas by requesting the details of the share.

====Creating a replica====
* POST to /share-replicas
User has to specify the share name/id of the share that is supposed to be replicated, an availability zone for the replica to exist in and optionally a share_network_id.
According to existing design, replicas of a share cannot exist in the same availability zone as the share itself.
* A newly created share_replica starts out in ''out_of_sync'' state and might transition to ''in_sync'' state when the driver reports this state.

====Listing and showing replicas====
* GET from /share-replicas
User can list all replicas and verify their ''status'' and ''replica_state''.
* GET from /share-replicas?share_id=<share_id>
User can list replicas for a particular share.
* GET from /share-replicas/<id>
User can view details of a replica.

====Promoting a non-active replica====
* POST to /share-replicas/action with body {'promote': None}
For replication types that permit promotion, the user can promote a replica with ''in_sync'' replica_state to ''active'' replica_state by initiating the ''promote_replica'' call.
* Only administrators can attempt promoting replicas with replica_state ''error'' or ''out_of_sync''.

====Deleting Replicas====
* DELETE to /share-replicas/<id>
The user can delete a replica. The last ''active'' replica cannot be deleted using this ''delete_replica'' call.

==System Workflows==
* Creating a share that supports replication
** Same process as creating a new share.
** The ''replication_type'' extra-spec from the share-type is copied over to the share''s data on the DB.
** Scheduler uses the ''replication_type'' extra-spec to filter available hosts and schedule the share on an appropriate backend that supports that specific style of ''replication_type''.

* Creating a replica:
** Create a share instance on the database.
** Update the share instance ''replica_state'' to ''out_of_sync''.
** Cast a call to the scheduler to find an appropriate host to schedule the replica on.
** In the scheduler, find weighted host for the replica.
*** If host cannot be chosen, update the replica's ''status'' and ''replica_state'' on the Database to ''error''. Throw an exception.
** Cast the create_share_replica call to the driver based on the weighted host selection.
** Prior to invoking the driver's call, collate information regarding existing ''active'' replica instance, existing share access rules and the replica's share_server, pass these to the driver while invoking ''create_replica''.
** The driver may return ''export_locations'' and ''replica_state''. If they are returned, the database is updated with these values.
** If the ''replication_type'' is ''writable'', the driver MUST return the ''replica_state'' set to ''active''.
** All the access rules for the new replica are set to ''active'' state on the database.
** The driver may throw an exception, update the ''status'' and ''replica_state'' of the replica to ''error''.

* Listing/showing shares:
** For shares that support replication, the result of listing/showing the share will have new fields, ''replication_type'': denoting the replication style supported and ''has_replicas'' to denote if the share has replicas.
** For replicated shares, the primary instance for the share must be preferred among those replicas that have a status ''replication_change'' or among those that have a replica_state set to ''active''.

* Listing replicas
** For listing, Grab share instances from the database that have a replica_state among {''in_sync'', ''active'', ''out_of_sync'', ''error''}.
** If share_id is provided, grab only share instances that belong to that share and have a replica_state among {''in_sync'', ''active'', ''out_of_sync'', ''error''}.
** Limits and offsets must be respected on the list calls.

* Promoting a non-active replica:
** If replica is already active, do nothing.
** Update the ''status'' of the replica being promoted to ''replication_change''.
** Grab all available replicas and invoke the appropriate driver''s promote_replica call by passing the available replicas and the new replica.
** If the driver throws an exception at this stage, there is a good chance that the replicas are somehow altered on the backend. Loop through the replicas and set their ''replica_state''s to ''error'' and leave the ''status'' unchanged. Also set the ''status'' of the replica that failed to promote to ''available'' as before this operation. The backend may choose to update the actual replica_state during the replica monitoring call.
** The driver may return an updated list of replicas. Update the ''export_locations'' and ''replica_state''s to the database.
** The ''status'' of the replica that was promoted should return to ''available'' from ''replication_change''.

* Periodic replica update:
** The share manager implements a looping call with default interval of 5 minutes to query from each driver and each backend the ''replica_state'' of all non-active replicas that are associated with them.
** The driver is allowed to set the ''replica_state'' to ''in_sync'', ''out_of_sync'', and in exceptional cases, ''error''.
** If the driver sets the ''replica_state'' of a replica to ''error'', it is assumed that some irrecoverable damage has occured to the replica instance. The ''status'' of the replica instance must be set to ''error'' as well.

* Deleting a share replica
** If the replica had no host, it is simply removed from the database.
** The ''status'' of the replica is set to ''deleting'' on the database and the appropriate driver method is called to delete the replica.
** If the driver fails to delete the replica, the ''status'' of the replica is updated to ''error_deleting'' on the database.
** If the driver returns without errors, the replica instance is removed from the database.

==Scheduler Impact==
* The host_manager must update the ''replication_type'' capability from the backend. The backends must report this capability at the host/pool level.
* The filter_scheduler must then match the ''replication_type'' capability with the capability that the replication share_type demands.
* The ''replication_type'' must be one of the styles mentioned [[#Supported Replication Styles|above]].

==DB Impact==
* Share Export Locations:
Preferred export locations will only be from the instances with ''replica_state'' set to ''active''.
def export_locations(self):
# TODO(gouthamr): Return AZ specific export locations for replicated
# shares
# NOTE(gouthamr): For a replicated share, export locations of the
# 'active' instances are taken, if 'available'.
all_export_locations = []
select_instances = list(filter(
lambda x: x['replica_state'] == constants.REPLICA_STATE_ACTIVE,
self.instances)) or self.instances
for instance in select_instances:
if instance['status'] == constants.STATUS_AVAILABLE:
for export_location in instance.export_locations:
all_export_locations.append(export_location['path'])
return all_export_locations

* Share Instance:
Preferred instance will be an instance with ''status'' set to ''replication_change'' or any instance with ''status'' set to ''available'' and ''replica_state'' set to ''active''.
def instance(self):
# NOTE(gouthamr): The order of preference: status 'replication_change',
# followed by 'available' and 'creating'. If replicated share and
# not undergoing a 'replication_change', only 'active' instances are
# preferred.
result = None
if len(self.instances) > 0:
order = [constants.STATUS_REPLICATION_CHANGE,
constants.STATUS_AVAILABLE, constants.STATUS_CREATING]
other_statuses = ([x['status'] for x in self.instances
if x['status'] not in order])
order.extend(other_statuses)
sorted_instances = sorted(
self.instances, key=lambda x: order.index(x['status']))
select_instances = sorted_instances
if (select_instances[0]['status'] !=
constants.STATUS_REPLICATION_CHANGE):
select_instances = (
list(filter(lambda x: x['replica_state'] ==
constants.REPLICA_STATE_ACTIVE,
sorted_instances)) or sorted_instances
)
result = select_instances[0]
return result

* New field on Share:
replication_type = Column(String(255), nullable=True)

* New field on ShareInstance
replica_state = Column(String(255), nullable=True)

==API Design==
GET /share-replicas/
GET /share-replicas?share-id=<share_id>
GET /share-replicas/<replica_id>

POST /share-replicas/<share-id>
Body:
{
'share_replica':
{
'availability_zone':<availability_zone_id>,
'share_id':<share_id>,
'share_network_id':<share_network_id>
}
}
POST /share-replicas/<replica-id>/action
Body:
{'os-promote_replica': null}

DELETE /share-replicas/<replica-id>

==Policies==

policy.json - All replication group policies should default to the default policy
"share_replica:get_all": "rule:default",
"share_replica:show": "rule:default",
"share_replica:create" : "rule:default",
"share_replica:delete": "rule:default",
"share_replica:promote": "rule:default"

==Driver API==
Only a single replication driver method will be called at a time for a given share as they are locked by share id.

def create_replica(self, context, active_replica, new_replica,
access_rules, share_server=None):
"""Replicate the active replica to a new replica on this backend.
:param context: Current context
:param active_replica: A current active replica instance dictionary.
EXAMPLE:
.. code::
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS1',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'active',
'availability_zone_id': 'e2c2db5c-cb2f-4697-9966-c06fb200cb80',
'export_locations': [
<models.ShareInstanceExportLocations>._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param new_replica: The share replica dictionary.
EXAMPLE:
.. code::
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS2',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'out_of_sync',
'availability_zone_id': 'f6e146d0-65f0-11e5-9d70-feff819cdc9f',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': 'e6155221-ea00-49ef-abf9-9f89b7dd900a',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param access_rules: A list of access rules that other instances of
the share already obey.
EXAMPLE:
.. code::
[ {
'id': 'f0875f6f-766b-4865-8b41-cccb4cdf1676',
'deleted' = False,
'share_id' = 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'access_type' = 'ip',
'access_to' = '172.16.20.1',
'access_level' = 'rw',
}]
:param share_server: <models.ShareServer>._as_dict() or None,
Share server of the replica being created.
:return: (export_locations, replica_state)
export_locations is a list of paths and replica_state is one of
active, in-sync, out-of-sync or error.
A backend supporting 'writable' type replication should return
'active' as the replica_state.
Export locations should be in the same format as returned by a
share_create. This list may be empty or None.
EXAMPLE:
.. code::
[{'id': 'uuid', 'export_locations': ['export_path']}]
"""
raise NotImplementedError()

def delete_replica(self, context, active_replica, replica,
share_server=None):
"""Delete a replica.
:param context:Current context
:param active_replica: A current active replica instance dictionary.
EXAMPLE:
.. code::
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS1',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'active',
'availability_zone_id': 'e2c2db5c-cb2f-4697-9966-c06fb200cb80',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param replica: Dictionary of the share replica being deleted.
EXAMPLE:
.. code::
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS2',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'in_sync',
'availability_zone_id': 'f6e146d0-65f0-11e5-9d70-feff819cdc9f',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '53099868-65f1-11e5-9d70-feff819cdc9f',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param share_server: <models.ShareServer>._as_dict() or None,
Share server of the replica to be deleted.
:return: None.
"""
raise NotImplementedError()

def promote_replica(self, context, replica_list, replica, access_rules, share_server=None):
"""Promote a replica to 'active' replica state.
:param context:Current context
:param replica_list: List of all replicas for a particular share.
This list also contains the replica to be promoted. The 'active'
replica will have its 'replica_state' attr set to 'active'.
EXAMPLE:
.. code::
[
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'replica_state': 'in-sync',
...
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
'share_server': <models.ShareServer>._as_dict() or None,
},
{
'id': '10e49c3e-aca9-483b-8c2d-1c337b38d6af',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'replica_state': 'active',
...
'share_server_id': 'f63629b3-e126-4448-bec2-03f788f76094',
'share_server': <models.ShareServer>._as_dict() or None,
},
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'replica_state': 'in-sync',
...
'share_server_id': '07574742-67ea-4dfd-9844-9fbd8ada3d87',
'share_server': <models.ShareServer>._as_dict() or None,
},
...
]
:param replica: Dictionary of the replica to be promoted.
EXAMPLE:
.. code::
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS2',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'in_sync',
'availability_zone_id': 'f6e146d0-65f0-11e5-9d70-feff819cdc9f',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '07574742-67ea-4dfd-9844-9fbd8ada3d87',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param access_rules: A list of access rules that other instances of
the share already obey.
EXAMPLE:
.. code::
[ {
'id': 'f0875f6f-766b-4865-8b41-cccb4cdf1676',
'deleted' = False,
'share_id' = 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'access_type' = 'ip',
'access_to' = '172.16.20.1',
'access_level' = 'rw',
}]
:param share_server: <models.ShareServer>._as_dict() or None,
Share server of the replica to be promoted.
:return: updated_replica_list or None
The driver can return the updated list as in the request
parameter. Changes that will be updated to the Database are:
'export_locations' and 'replica_state'.
:raises Exception
This can be any exception derived from BaseException. This is
re-raised by the manager after some necessary cleanup. If the
driver raises an exception during promotion, it is assumed
that all of the replicas of the share are in an inconsistent
state. Recovery is only possible through the periodic update
call and/or administrator intervention to correct the 'status'
of the affected replicas if they become healthy again.
"""
raise NotImplementedError()

def update_replica_state(self, context, replica, share_server=None):
"""Update the replica_state of a replica.
Drivers should fix replication relationships that were broken if
possible inside this method.
:param context:Current context
:param replica: Dictionary of the replica being updated.
EXAMPLE:
.. code::
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS1',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'active',
'availability_zone_id': 'e2c2db5c-cb2f-4697-9966-c06fb200cb80',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
}
:param share_server: <models.ShareServer>._as_dict() or None
:return: replica_state
replica_state - a str value denoting the replica_state that the
replica can have. Valid values are 'in_sync' and 'out_of_sync'
or None (to leave the current replica_state unchanged).
"""
raise NotImplementedError()

==Manila Client==
share-replica-create Creates a replica
share-replica-delete Remove one or more share replicas.
share-replica-list List share replicas.
share-replica-promote Promote replica to active replica.
share-replica-show Show details about a replica.

=FAQs=
# '''How do we deal with Network issues in multi-SVM replication?''' If we choose to make replication a single-svm-only feature, the share-network API doesn't need to change. In order to support replication with share-networks, we also need to modify the share-network create API which allows creation of share networks with a table of AZ-to-subnet mappings. This approach allows us to keep a single share-network per share (with associated security service) while allowing the tenant to specify enough information that each share instance can be attached to the appropriate network in each AZ. Multi-AZ share networks would also be useful for non-replicated use cases. 
# '''Do we support Pool-level Replication?''' Some vendors have suggested that certain backends can mirror groups of shares or whole pools more efficiently than individual shares. This design only addresses the mirroring of individual shares. In the future, we may allow allow replication of groups of shares, but only if those groups are contained within a single tenant and defined by the tenant. 
# '''From where to where do we allow replication? Is it intra-cloud or inter-cloud? Do we allow replication to something that's not managed by Manila?''' Intra-cloud. Replicating to something outside of Manila allows a bit more freedom, but with significantly less value, because there's practically nothing we can do to automate the failover/failback portion of a disaster. For use cases involving replication outside of Manila, we would need to involve other tools with more breadth/scope to manage the process. 
# '''Who configures the replication? The admin? The end user? The manila scheduler?''' The end user. In the original design we presumed that the actual replication relationships should be hidden from the end user, but this doesn't match well with the concept of AZs that we are adding to Manila. If the users need to have control over which AZ the primary copy of their data lives in, then they also need to control where the other copies live. This means that the administrator's job is to ensure that for any share type that is replicated, it can be replicated from any AZ to any other AZ. 
# '''Is replication supported only intra-region or inter-region as well? How about intra-AZ vs inter-AZ?''' This design is for intra-region only. Inter-region replication is clearly a feature we want, but it would look pretty different from intra-region replication; this design does not cover that. Intra-AZ replication is supported, however inter-AZ replication is recommended as AZs should be perceived as failure domains.

=Unanswered Questions=
# '''Is there no way to achieve non-disruptive failover?''' (bswartz) I would love to find out that our initial intuition here is wrong, because it would change a lot of aspects of the design. It's worth spending time to brainstorm and research possibilities in this area. So far the most promising ideas involve:
#*Using VirtFS to mediate filesystem access and achieving non-disruptive failover that way
#*Using some kind of agent inside the guests to mediate file access 
# '''Can a replica have a different share_type? There is a valid use case where a user would want to create a share replica on backend with different capabilities than the one the original share resides on. For instance, replicas might need to be on a less expensive backend. In that case, can the replica have a different share_type altogether?''' "Currently", we inherit the share_type of the share and believe that replication has to be on symmetric terms, where both backends have similar capabilities. 
#'''Can we allow the driver to restrict replication support between available backends? Backends may support replication to other compatible backends only. Hence, they must report some sort of information to the scheduler so that when creating a replica for an existing share, the scheduler would use that information to schedule the creation of the replica. What information should this be?''' (gouthamr) We're investigating including 'driver_class_name' in a ReplicationFilter, including the possibility of backend reported configuration, 'replication_partners'. 
#'''How are access rules persisted across replicas/share instances?''' 
#*Do all replicas have the same access rules applied? (gouthamr): This is Currently being pursued
#*Should access rules be applied only to "active" replicas? 
#'''How does migration affect replicated shares?'''
#*(gouthamr)In the Tokyo summit, it was decided that we currently disallow migration of a share that has replicas. The replication relationships must be broken and reestablished after the migration is complete in order to migrate a share with replicas. 
# '''Do we need an API to initiate a sync? (resync or force-update)''' This is a great idea and one certain use case is during planned failovers. (gouthamr) 
# '''How to recover from promotion failures?''' (gouthamr) Currently, if some exception occurs at the driver level, the replicas are rendered useless. Should we provide escape hatches to avoid complete loss in case these are recoverable exceptions? The administrator has the ability to 'reset-state' and affect the 'status' of any replica. Currently, there exists no API to 'reset-replica-state' for a replica.

= Examples =

These examples assume that the replicas exist in different AZs.

== Writable replication example ==

# Administrator sets up backends in AZs b1 and b2 that have capability replication_type=writeable
# Administrator creates a new share_type called foo
# Administrator sets replication_type=writeable extra spec on share type foo
# User creates new share of type foo in AZ b1
# Share is created with replication_type=writeable, and 1 active replica in AZ b1
# User grants access on share to client1 in AZ b1, obtains the export location of the replica, mounts the share on a client, and starts to write data
# User add new replica of share in AZ b2
# A second replica is created in AZ b2 which initially has state out_of_sync
# Shortly afterwards, the replica state changes to active (after the replica finishes syncing with the original copy)
# The user grants access on the share to client2 in AZ b2, obtains the export location of the new replica, mounts the share, and sees the same data that client1 wrote
# Client2 writes some data to the share, which is immediately visible to client1

== Readable replication example ==

# Administrator sets up backends in AZs b1 and b2 that have capability replication_type=readable
# Administrator creates a new share_type called bar
# Administrator sets replication_type=readable extra spec on share type bar
# User creates new share of type bar in AZ b1
# Share is created with replication_type=readable, and 1 active replica in AZ b1
# User grants access on share to client1 in AZ b1, obtains the export location of the replica, mounts the share on a client, and starts to write data
# User add new replica of share in AZ b2
# A second replica is created in AZ b2 which initially has state out_of_sync
# Shortly afterwards, the replica state changes to in_sync (after the replica finishes syncing with the original copy)
# The user grants access on the share to client2 in AZ b2, obtains the export location of the new replica, mounts the share, and sees the same data that client1 wrote
# Client2 cannot write data to the share but continues to see updates

== Failover/failback example ==

(Continued from above)
# An outage occurs in AZ b1
# Administrator sends out a bulletin about the outage to his users "the power transformer in b1 turned to slag, it will be 12 hours before it can be replaced, please bear with us, yada yada"
# User notices that his application on client1 is no longer running, and investigates
# User finds out that client1 is gone, and reads a bulletin from the admin explaining why
# User notes that the b1 replica is his share is still active and the b2 replica is in_sync, while the state of the share is available
# User calls set active replica to AZ b2 on his share
# The share goes to state replication_change and access to the share is briefly lost on client2
# The state of replica b2 changes to active and the state of replica b1 changes to out_of_sync after Manila fails to contact the original primary. The state of the share changes back to available, and access is restored on client2
# User starts his application on client2, and the application recovers from an apparent crash, with a consistent copy of the application's data
# User application is back up and running, disaster is averted
# Eventually maintenance on the b1 AZ is completed, and all of the equipment is reactivated
# Administrator sends out a bulletin to his users about the outage ending "we replaced the transformer with a better one, everything is back online, thanks for your understanding, yada yada"
# Manila notices the out_of_sync replica is reachable and initiates a resync, bringing the b1 replica in_sync within a short time
# After some time, the users reads the bulletin and observes that his share is being replicated again. He decides to intentionally move back to b1.
# User gracefully shuts down his application, and flushes I/O
# User calls set active replica to AZ b1 on his share
# The share goes to state replication_change and no disruption is observed because the application is shut down
# The state of replica b1 changes to active and the state of replica b2 changes to in_sync after Manila reverses the replication again. The state of the share changes back to available
# User starts his application on client1, and the application starts cleanly, having been shut down gracefully

=Implementation Progress=
===Core Work===
[https://blueprints.launchpad.net/manila/+spec/manila-share-replication Blueprint for Core API/Scheduler implementation]
[https://blueprints.launchpad.net/manila/+spec/manilaclient-share-replication Blueprint for Client implementation]

===Ex Driver Implementation===
[https://blueprints.launchpad.net/manila/+spec/netapp-cdot-single-svm-replication Blueprint for cDOT Driver Implementation]

=Older Design Pages=
* Replication Use Cases: https://wiki.openstack.org/wiki/Manila/Replication_Use_Cases
* Design Notes: https://wiki.openstack.org/wiki/Manila/Replication_Design_Notes

Manila/design/manila-mitaka-data-replication

2015-12-02T21:18:35Z

Alex Meade: /* Listing and showing replicas */

=Manila Disaster Recovery / Data Replication Design=

==Introduction==

Replicated shares will be implemented in Manila without adding any new services or any new drivers. The code for creating replicated shares and for adding/removing replicas and other replication-related operations will all go into the existing share drivers. The most significant change required to allow this will be that share drivers will be responsible for communicating with ALL storage controllers necessary to achieve any replication tasks, even if that involves sending commands to other storage controllers in other AZs.

While we can't know how every storage controller works and how each vendor will implement replication, this approach should give driver writers all the flexibility needed to implement what they need with minimal added complexity in the Manila core. There are already examples of drivers reading config sections for other drivers when a operation requires communicating with 2 backends. The Manila Share Manager itself is expected to communicate all necessary backend details for share replicas that exist across AZs to a backend which it requests an operation upon.

Intra-AZ replication is supported, but in an ideal solution, AZs should be perceived as failure domains. So this feature tends to shine in an inter-AZ replication use case.

===Supported Replication Types===
There are 3 styles of DR that we would like to support in the long run:

# writable - Amazon EFS-style synchronously replicated shares where all replicas are writable. Promotion it not supported and not needed.
# readable - Mirror-style replication with a primary (writable) copy and one or more secondary (read-only) copies which can become writable after a promotion of the secondary.
# dr - Generalized replication with secondary copies that are inaccessible until after a promotion of one of the secondary copies.

===Replica States===
Each replica has a ''replica_state'' which has 4 possible values:

# active - All writable replicas are active.
# in_sync - Passive replica which is up to date with the active replica, and can be promoted to active.
# out_of_sync - Passive replica which has gone out of date, or new replica that is not yet up to date.
# error - Either the scheduler failed to schedule this replica or some irrecoverable damage occurred during promotion of a non-active replica. Drivers may set the ''replica_state'' to ''error'' if some irrecoverable damage to the replica is discovered at any point during its existence. (through the periodic ''replica_state'' update)
[[File:Manila instance replication states.png|center]]

===New share state===
replication_change - New transient state triggered by a change of the active replica. Access to the share is cut off while in this state.

===Promotion===
For '''readable''' and '''dr''' styles, we refer to the task of switching a non-active replica with the '''active''' replica as '''promotion'''. For the '''writable''' style of replication, '''promotion''' does not make sense since all replicas are '''active''' (writable) at all given points of time. Promotion of replicas with '''replica_state''' set to '''out_of_sync''' or '''error''' may be unsupported by the backend. However, we would like to allow the action as an administrator feature and such an attempt may be honored by backends if possible.

When multiple replicas exist, multiple replication relationships between shares may need to be redefined at the backend. If the driver fails at this stage, the replicas may be left at an inconsistent state. The manager will set all replica instances to have '''replica_state''' and '''status''' to '''error'''. Recovery from this stage would require administrator intervention.

==User Workflows==
====Creating a share supporting replication====
* Administrators can create a share type with extra-spec ''replication_type'', specifying the style of replication the backend supports.
* Users can use the share type to create a new share that allows/supports replication.
* A replicated share always starts out with one replica instance, the share itself. This should not be confused as the share having a replica already. The user can verify if the share has any replicas by requesting the details of the share.

====Creating a replica====
* POST to /share-replicas
User has to specify the share name/id of the share that is supposed to be replicated, an availability zone for the replica to exist in and optionally a share_network_id.
According to existing design, replicas of a share cannot exist in the same availability zone as the share itself.
* A newly created share_replica starts out in ''out_of_sync'' state and might transition to ''in_sync'' state when the driver reports this state.

====Listing and showing replicas====
* GET from /share-replicas
User can list all replicas and verify their ''status'' and ''replica_state''.
* GET from /share-replicas?share_id=<share_id>
User can list replicas for a particular share.
* GET from /share-replicas/<id>
User can view details of a replica.

====Promoting a non-active replica====
* POST to /share-replicas/action with body {'promote': None}
For replication types that permit promotion, the user can promote a replica with ''in_sync'' replica_state to ''active'' replica_state by initiating the ''promote_replica'' call.
* Only administrators can attempt promoting replicas with replica_state ''error'' or ''out_of_sync''.

====Deleting Replicas====
* DELETE to /share-replicas/<id>
The user can delete a replica. The last ''active'' replica cannot be deleted using this ''delete_replica'' call.

==System Workflows==
* Creating a share that supports replication
** Same process as creating a new share.
** The ''replication_type'' extra-spec from the share-type is copied over to the share''s data on the DB.
** Scheduler uses the ''replication_type'' extra-spec to filter available hosts and schedule the share on an appropriate backend that supports that specific style of ''replication_type''.

* Creating a replica:
** Create a share instance on the database.
** Update the share instance ''replica_state'' to ''out_of_sync''.
** Cast a call to the scheduler to find an appropriate host to schedule the replica on.
** In the scheduler, find weighted host for the replica.
*** If host cannot be chosen, update the replica's ''status'' and ''replica_state'' on the Database to ''error''. Throw an exception.
** Cast the create_share_replica call to the driver based on the weighted host selection.
** Prior to invoking the driver's call, collate information regarding existing ''active'' replica instance, existing share access rules and the replica's share_server, pass these to the driver while invoking ''create_replica''.
** The driver may return ''export_locations'' and ''replica_state''. If they are returned, the database is updated with these values.
** If the ''replication_type'' is ''writable'', the driver MUST return the ''replica_state'' set to ''active''.
** All the access rules for the new replica are set to ''active'' state on the database.
** The driver may throw an exception, update the ''status'' and ''replica_state'' of the replica to ''error''.

* Listing/showing shares:
** For shares that support replication, the result of listing/showing the share will have new fields, ''replication_type'': denoting the replication style supported and ''has_replicas'' to denote if the share has replicas.
** For replicated shares, the primary instance for the share must be preferred among those replicas that have a status ''replication_change'' or among those that have a replica_state set to ''active''.

* Listing replicas
** For listing, Grab share instances from the database that have a replica_state among {''in_sync'', ''active'', ''out_of_sync'', ''error''}.
** If share_id is provided, grab only share instances that belong to that share and have a replica_state among {''in_sync'', ''active'', ''out_of_sync'', ''error''}.
** Limits and offsets must be respected on the list calls.

* Promoting a non-active replica:
** If replica is already active, do nothing.
** Update the ''status'' of the replica being promoted to ''replication_change''.
** Grab all available replicas and invoke the appropriate driver''s promote_replica call by passing the available replicas and the new replica.
** If the driver throws an exception at this stage, there is a good chance that the replicas are somehow altered on the backend. Loop through the replicas and set their ''replica_state''s to ''error'' and leave the ''status'' unchanged. Also set the ''status'' of the replica that failed to promote to ''available'' as before this operation. The backend may choose to update the actual replica_state during the replica monitoring call.
** The driver may return an updated list of replicas. Update the ''export_locations'' and ''replica_state''s to the database.
** The ''status'' of the replica that was promoted should return to ''available'' from ''replication_change''.

* Periodic replica update:
** The share manager implements a looping call with default interval of 5 minutes to query from each driver and each backend the ''replica_state'' of all non-active replicas that are associated with them.
** The driver is allowed to set the ''replica_state'' to ''in_sync'', ''out_of_sync'', and in exceptional cases, ''error''.
** If the driver sets the ''replica_state'' of a replica to ''error'', it is assumed that some irrecoverable damage has occured to the replica instance. The ''status'' of the replica instance must be set to ''error'' as well.

* Deleting a share replica
** If the replica had no host, it is simply removed from the database.
** The ''status'' of the replica is set to ''deleting'' on the database and the appropriate driver method is called to delete the replica.
** If the driver fails to delete the replica, the ''status'' of the replica is updated to ''error_deleting'' on the database.
** If the driver returns without errors, the replica instance is removed from the database.

==Scheduler Impact==
* The host_manager must update the ''replication_type'' capability from the backend. The backends must report this capability at the host/pool level.
* The filter_scheduler must then match the ''replication_type'' capability with the capability that the replication share_type demands.
* The ''replication_type'' must be one of the styles mentioned [[#Supported Replication Styles|above]].

==DB Impact==
* Share Export Locations:
Preferred export locations will only be from the instances with ''replica_state'' set to ''active''.
def export_locations(self):
# TODO(gouthamr): Return AZ specific export locations for replicated
# shares
# NOTE(gouthamr): For a replicated share, export locations of the
# 'active' instances are taken, if 'available'.
all_export_locations = []
select_instances = list(filter(
lambda x: x['replica_state'] == constants.REPLICA_STATE_ACTIVE,
self.instances)) or self.instances
for instance in select_instances:
if instance['status'] == constants.STATUS_AVAILABLE:
for export_location in instance.export_locations:
all_export_locations.append(export_location['path'])
return all_export_locations

* Share Instance:
Preferred instance will be an instance with ''status'' set to ''replication_change'' or any instance with ''status'' set to ''available'' and ''replica_state'' set to ''active''.
def instance(self):
# NOTE(gouthamr): The order of preference: status 'replication_change',
# followed by 'available' and 'creating'. If replicated share and
# not undergoing a 'replication_change', only 'active' instances are
# preferred.
result = None
if len(self.instances) > 0:
order = [constants.STATUS_REPLICATION_CHANGE,
constants.STATUS_AVAILABLE, constants.STATUS_CREATING]
other_statuses = ([x['status'] for x in self.instances
if x['status'] not in order])
order.extend(other_statuses)
sorted_instances = sorted(
self.instances, key=lambda x: order.index(x['status']))
select_instances = sorted_instances
if (select_instances[0]['status'] !=
constants.STATUS_REPLICATION_CHANGE):
select_instances = (
list(filter(lambda x: x['replica_state'] ==
constants.REPLICA_STATE_ACTIVE,
sorted_instances)) or sorted_instances
)
result = select_instances[0]
return result

* New field on Share:
replication_type = Column(String(255), nullable=True)

* New field on ShareInstance
replica_state = Column(String(255), nullable=True)

==API Design==
GET /share-replicas/
GET /share-replicas?share-id=<share_id>
GET /share-replicas/<replica_id>

POST /share-replicas/<share-id>
Body:
{
'share_replica':
{
'availability_zone':<availability_zone_id>,
'share_id':<share_id>,
'share_network_id':<share_network_id>
}
}
POST /share-replicas/<replica-id>/action
Body:
{'os-promote_replica': null}

DELETE /share-replicas/<replica-id>

==Policies==

policy.json - All replication group policies should default to the default policy
"share_replica:get_all": "rule:default",
"share_replica:show": "rule:default",
"share_replica:create" : "rule:default",
"share_replica:delete": "rule:default",
"share_replica:promote": "rule:default"

==Driver API==

def create_replica(self, context, active_replica, new_replica,
access_rules, share_server=None):
"""Replicate the active replica to a new replica on this backend.
:param context:Current context
:param active_replica: A current active replica instance dictionary.
EXAMPLE:
.. code::
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS1',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'active',
'availability_zone_id': 'e2c2db5c-cb2f-4697-9966-c06fb200cb80',
'export_locations': [
<models.ShareInstanceExportLocations>._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param new_replica: The share replica dictionary.
EXAMPLE:
.. code::
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS2',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'out_of_sync',
'availability_zone_id': 'f6e146d0-65f0-11e5-9d70-feff819cdc9f',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': 'e6155221-ea00-49ef-abf9-9f89b7dd900a',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param access_rules: A list of access rules that other instances of
the share already obey.
EXAMPLE:
.. code::
[ {
'id': 'f0875f6f-766b-4865-8b41-cccb4cdf1676',
'deleted' = False,
'share_id' = 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'access_type' = 'ip',
'access_to' = '172.16.20.1',
'access_level' = 'rw',
}]
:param share_server: <models.ShareServer>._as_dict() or None,
Share server of the replica being created.
:return: (export_locations, replica_state)
export_locations is a list of paths and replica_state is one of
active, in-sync, out-of-sync or error.
A backend supporting 'writable' type replication should return
'active' as the replica_state.
Export locations should be in the same format as returned by a
share_create. This list may be empty or None.
EXAMPLE:
.. code::
[{'id': 'uuid', 'export_locations': ['export_path']}]
"""
raise NotImplementedError()

def delete_replica(self, context, active_replica, replica,
share_server=None):
"""Delete a replica.
:param context:Current context
:param active_replica: A current active replica instance dictionary.
EXAMPLE:
.. code::
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS1',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'active',
'availability_zone_id': 'e2c2db5c-cb2f-4697-9966-c06fb200cb80',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param replica: Dictionary of the share replica being deleted.
EXAMPLE:
.. code::
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS2',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'in_sync',
'availability_zone_id': 'f6e146d0-65f0-11e5-9d70-feff819cdc9f',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '53099868-65f1-11e5-9d70-feff819cdc9f',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param share_server: <models.ShareServer>._as_dict() or None,
Share server of the replica to be deleted.
:return: None.
"""
raise NotImplementedError()

def promote_replica(self, context, replica_list, replica, access_rules, share_server=None):
"""Promote a replica to 'active' replica state.
:param context:Current context
:param replica_list: List of all replicas for a particular share.
This list also contains the replica to be promoted. The 'active'
replica will have its 'replica_state' attr set to 'active'.
EXAMPLE:
.. code::
[
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'replica_state': 'in-sync',
...
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
'share_server': <models.ShareServer>._as_dict() or None,
},
{
'id': '10e49c3e-aca9-483b-8c2d-1c337b38d6af',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'replica_state': 'active',
...
'share_server_id': 'f63629b3-e126-4448-bec2-03f788f76094',
'share_server': <models.ShareServer>._as_dict() or None,
},
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'replica_state': 'in-sync',
...
'share_server_id': '07574742-67ea-4dfd-9844-9fbd8ada3d87',
'share_server': <models.ShareServer>._as_dict() or None,
},
...
]
:param replica: Dictionary of the replica to be promoted.
EXAMPLE:
.. code::
{
'id': 'e82ff8b6-65f0-11e5-9d70-feff819cdc9f',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS2',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'in_sync',
'availability_zone_id': 'f6e146d0-65f0-11e5-9d70-feff819cdc9f',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '07574742-67ea-4dfd-9844-9fbd8ada3d87',
'share_server': <models.ShareServer>._as_dict() or None,
}
:param access_rules: A list of access rules that other instances of
the share already obey.
EXAMPLE:
.. code::
[ {
'id': 'f0875f6f-766b-4865-8b41-cccb4cdf1676',
'deleted' = False,
'share_id' = 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'access_type' = 'ip',
'access_to' = '172.16.20.1',
'access_level' = 'rw',
}]
:param share_server: <models.ShareServer>._as_dict() or None,
Share server of the replica to be promoted.
:return: updated_replica_list or None
The driver can return the updated list as in the request
parameter. Changes that will be updated to the Database are:
'export_locations' and 'replica_state'.
:raises Exception
This can be any exception derived from BaseException. This is
re-raised by the manager after some necessary cleanup. If the
driver raises an exception during promotion, it is assumed
that all of the replicas of the share are in an inconsistent
state. Recovery is only possible through the periodic update
call and/or administrator intervention to correct the 'status'
of the affected replicas if they become healthy again.
"""
raise NotImplementedError()

def update_replica_status(self, context, replica, share_server=None):
"""Update the status and replica_state of a replica.
Drivers should fix replication relationships that were broken if
possible inside this method.
:param context:Current context
:param replica: Dictionary of the replica being updated.
EXAMPLE:
.. code::
{
'id': 'd487b88d-e428-4230-a465-a800c2cce5f8',
'share_id': 'f0e4bb5e-65f0-11e5-9d70-feff819cdc9f',
'deleted': False,
'host': 'openstack2@cmodeSSVMNFS1',
'status': 'available',
'scheduled_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'launched_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'terminated_at': None,
'replica_state': 'active',
'availability_zone_id': 'e2c2db5c-cb2f-4697-9966-c06fb200cb80',
'export_locations': [
models.ShareInstanceExportLocations._as_dict()
],
'share_network_id': '4ccd5318-65f1-11e5-9d70-feff819cdc9f',
'share_server_id': '4ce78e7b-0ef6-4730-ac2a-fd2defefbd05',
}
:param share_server: <models.ShareServer>._as_dict() or None
:return: replica_state
replica_state - a str value denoting the replica_state that the
replica can have. Valid values are 'in_sync' and 'out_of_sync'
or None (to leave the current replica_state unchanged).
"""
raise NotImplementedError()

==Manila Client==
share-replica-create Creates a replica
share-replica-delete Remove one or more share replicas.
share-replica-list List share replicas.
share-replica-promote Promote replica to active replica.
share-replica-show Show details about a replica.

=FAQs=
# '''How do we deal with Network issues in multi-SVM replication?''' If we choose to make replication a single-svm-only feature, the share-network API doesn't need to change. In order to support replication with share-networks, we also need to modify the share-network create API which allows creation of share networks with a table of AZ-to-subnet mappings. This approach allows us to keep a single share-network per share (with associated security service) while allowing the tenant to specify enough information that each share instance can be attached to the appropriate network in each AZ. Multi-AZ share networks would also be useful for non-replicated use cases. 
# '''Do we support Pool-level Replication?''' Some vendors have suggested that certain backends can mirror groups of shares or whole pools more efficiently than individual shares. This design only addresses the mirroring of individual shares. In the future, we may allow allow replication of groups of shares, but only if those groups are contained within a single tenant and defined by the tenant. 
# '''From where to where do we allow replication? Is it intra-cloud or inter-cloud? Do we allow replication to something that's not managed by Manila?''' Intra-cloud. Replicating to something outside of Manila allows a bit more freedom, but with significantly less value, because there's practically nothing we can do to automate the failover/failback portion of a disaster. For use cases involving replication outside of Manila, we would need to involve other tools with more breadth/scope to manage the process. 
# '''Who configures the replication? The admin? The end user? The manila scheduler?''' The end user. In the original design we presumed that the actual replication relationships should be hidden from the end user, but this doesn't match well with the concept of AZs that we are adding to Manila. If the users need to have control over which AZ the primary copy of their data lives in, then they also need to control where the other copies live. This means that the administrator's job is to ensure that for any share type that is replicated, it can be replicated from any AZ to any other AZ. 
# '''Is replication supported only intra-region or inter-region as well? How about intra-AZ vs inter-AZ?''' This design is for intra-region only. Inter-region replication is clearly a feature we want, but it would look pretty different from intra-region replication; this design does not cover that. Intra-AZ replication is supported, however inter-AZ replication is recommended as AZs should be perceived as failure domains.

=Unanswered Questions=
# '''Is there no way to achieve non-disruptive failover?''' (bswartz) I would love to find out that our initial intuition here is wrong, because it would change a lot of aspects of the design. It's worth spending time to brainstorm and research possibilities in this area. So far the most promising ideas involve:
#*Using VirtFS to mediate filesystem access and achieving non-disruptive failover that way
#*Using some kind of agent inside the guests to mediate file access 
# '''Can a replica have a different share_type? There is a valid use case where a user would want to create a share replica on backend with different capabilities than the one the original share resides on. For instance, replicas might need to be on a less expensive backend. In that case, can the replica have a different share_type altogether?''' "Currently", we inherit the share_type of the share and believe that replication has to be on symmetric terms, where both backends have similar capabilities. 
#'''Can we allow the driver to restrict replication support between available backends? Backends may support replication to other compatible backends only. Hence, they must report some sort of information to the scheduler so that when creating a replica for an existing share, the scheduler would use that information to schedule the creation of the replica. What information should this be?''' (gouthamr) We're investigating including 'driver_class_name' in a ReplicationFilter, including the possibility of backend reported configuration, 'replication_partners'. 
#'''How are access rules persisted across replicas/share instances?''' 
#*Do all replicas have the same access rules applied? (gouthamr): This is Currently being pursued
#*Should access rules be applied only to "active" replicas? 
#'''How does migration affect replicated shares?'''
#*(gouthamr)In the Tokyo summit, it was decided that we currently disallow migration of a share that has replicas. The replication relationships must be broken and reestablished after the migration is complete in order to migrate a share with replicas. 
# '''Do we need an API to initiate a sync? (resync or force-update)''' This is a great idea and one certain use case is during planned failovers. (gouthamr) 
# '''How to recover from promotion failures?''' (gouthamr) Currently, if some exception occurs at the driver level, the replicas are rendered useless. Should we provide escape hatches to avoid complete loss in case these are recoverable exceptions? The administrator has the ability to 'reset-state' and affect the 'status' of any replica. Currently, there exists no API to 'reset-replica-state' for a replica.

= Examples =

These examples assume that the replicas exist in different AZs.

== Writable replication example ==

# Administrator sets up backends in AZs b1 and b2 that have capability replication_type=writeable
# Administrator creates a new share_type called foo
# Administrator sets replication_type=writeable extra spec on share type foo
# User creates new share of type foo in AZ b1
# Share is created with replication_type=writeable, and 1 active replica in AZ b1
# User grants access on share to client1 in AZ b1, obtains the export location of the replica, mounts the share on a client, and starts to write data
# User add new replica of share in AZ b2
# A second replica is created in AZ b2 which initially has state out_of_sync
# Shortly afterwards, the replica state changes to active (after the replica finishes syncing with the original copy)
# The user grants access on the share to client2 in AZ b2, obtains the export location of the new replica, mounts the share, and sees the same data that client1 wrote
# Client2 writes some data to the share, which is immediately visible to client1

== Readable replication example ==

# Administrator sets up backends in AZs b1 and b2 that have capability replication_type=readable
# Administrator creates a new share_type called bar
# Administrator sets replication_type=readable extra spec on share type bar
# User creates new share of type bar in AZ b1
# Share is created with replication_type=readable, and 1 active replica in AZ b1
# User grants access on share to client1 in AZ b1, obtains the export location of the replica, mounts the share on a client, and starts to write data
# User add new replica of share in AZ b2
# A second replica is created in AZ b2 which initially has state out_of_sync
# Shortly afterwards, the replica state changes to in_sync (after the replica finishes syncing with the original copy)
# The user grants access on the share to client2 in AZ b2, obtains the export location of the new replica, mounts the share, and sees the same data that client1 wrote
# Client2 cannot write data to the share but continues to see updates

== Failover/failback example ==

(Continued from above)
# An outage occurs in AZ b1
# Administrator sends out a bulletin about the outage to his users "the power transformer in b1 turned to slag, it will be 12 hours before it can be replaced, please bear with us, yada yada"
# User notices that his application on client1 is no longer running, and investigates
# User finds out that client1 is gone, and reads a bulletin from the admin explaining why
# User notes that the b1 replica is his share is still active and the b2 replica is in_sync, while the state of the share is available
# User calls set active replica to AZ b2 on his share
# The share goes to state replication_change and access to the share is briefly lost on client2
# The state of replica b2 changes to active and the state of replica b1 changes to out_of_sync after Manila fails to contact the original primary. The state of the share changes back to available, and access is restored on client2
# User starts his application on client2, and the application recovers from an apparent crash, with a consistent copy of the application's data
# User application is back up and running, disaster is averted
# Eventually maintenance on the b1 AZ is completed, and all of the equipment is reactivated
# Administrator sends out a bulletin to his users about the outage ending "we replaced the transformer with a better one, everything is back online, thanks for your understanding, yada yada"
# Manila notices the out_of_sync replica is reachable and initiates a resync, bringing the b1 replica in_sync within a short time
# After some time, the users reads the bulletin and observes that his share is being replicated again. He decides to intentionally move back to b1.
# User gracefully shuts down his application, and flushes I/O
# User calls set active replica to AZ b1 on his share
# The share goes to state replication_change and no disruption is observed because the application is shut down
# The state of replica b1 changes to active and the state of replica b2 changes to in_sync after Manila reverses the replication again. The state of the share changes back to available
# User starts his application on client1, and the application starts cleanly, having been shut down gracefully

=Implementation Progress=
===Core Work===
[https://blueprints.launchpad.net/manila/+spec/manila-share-replication Blueprint for Core API/Scheduler implementation]
[https://blueprints.launchpad.net/manila/+spec/manilaclient-share-replication Blueprint for Client implementation]

===Ex Driver Implementation===
[https://blueprints.launchpad.net/manila/+spec/netapp-cdot-single-svm-replication Blueprint for cDOT Driver Implementation]

=Older Design Pages=
* Replication Use Cases: https://wiki.openstack.org/wiki/Manila/Replication_Use_Cases
* Design Notes: https://wiki.openstack.org/wiki/Manila/Replication_Design_Notes

Manila/Meetings

2015-11-12T14:32:49Z

Alex Meade: /* Next meeting */

= Weekly Manila team meeting =
'''NOTE MEETING TIME: Thursday at 15:00 UTC'''

If you're interested in management of shared filesystems for OpenStack, we have a weekly meetings in <code><nowiki>#openstack-meeting-alt</nowiki></code>, on Thursdays at 15:00 UTC. Please feel free to add items to the agenda below. NOTE: When adding topics please include your IRC name so we know who's topic it is and how to get more info.

== Next meeting ==
'''NOTE:''' ''Include your IRC nickname next to agenda items so that you can be called upon in the meeting and arrive at the meeting promptly if placing items in agenda. You might want to put this on your calendar if you are adding items.''

'''NOTE:''' USA-based attendees should remember to double check the meeting time as the end of daylight savings time will move the meeting earlier in your day.

'''Scheduled for 12 Nov, 2015, 15:00 UTC'''

# Which method is better for Manila QoS(independent way or just use extra specs )
#* link:http://paste.openstack.org/show/477677/
#*If use extra specs to config QoS, Does we first need add “QoS_support=true/false” in share_type. Does we need to add QoS_support flag in common capabilities(https://github.com/openstack/manila/blob/master/doc/source/devref/capabilities_and_extra_specs.rst)?
eg: share_type: qos_support='<is> True' or '<is> False'
netapp:max_iops=100 or qos:netapp:max_iops=100 or qos:max_iops=100
driver update: qos_support=true
#** An alternative could be to have the driver report a range

# Manila DR update
#* Still no reviews on https://review.openstack.org/#/c/238572/
#* First-party driver implementation? Which driver, current generic or new solution? When do we need this implemented and who wants to do it?
#* Replication_change share state - This is the state of the share when any of it's replicas are being promoted. Is 'replication_change' good or is there a better phrase?
# Manila Driver minimum requirements document update (ganso)
# Open Discussion

== Previous meetings ==

'''Scheduled for 5 Nov, 2015, 15:00 UTC'''
# Decide on read-only access rules as a required feature
# Manila DR - Design/Code feedback (gouthamr)
#* [https://wiki.openstack.org/wiki/Manila/design/manila-mitaka-data-replication Design Document]
#* [https://review.openstack.org/#/c/238572/ Core Implementation on Gerrit]
#* [https://review.openstack.org/#/c/235448/ Client Implementation on Gerrit]

'''Scheduled for 1 Oct, 2015, 15:00 UTC'''
# Liberty RC2 status
#* https://launchpad.net/manila/+milestone/liberty-rc2
#* https://review.openstack.org/#/q/project:openstack/manila+branch:stable/liberty,n,z
# Mitaka Design Summit
#* https://etherpad.openstack.org/p/manila-mitaka-summit-topics
# Vendor driver docs
#* https://github.com/openstack/manila/blob/master/doc/source/devref/share_back_ends_feature_support_mapping.rst
# Manila Client Nova Style Microversion Support (cfouts)
#* https://review.openstack.org/#/c/229142/
#* http://docs.openstack.org/developer/manila/devref/api_microversion_dev.html

'''Scheduled for 24 Sep, 2015, 15:00 UTC'''
# 3rd-party CI status
#* http://ec2-54-67-102-119.us-west-1.compute.amazonaws.com:5000/?project=openstack/manila&user=&timeframe=72&start=&end=&page_size=500
#* http://ci-watch.tintri.com/project?project=manila
# Liberty RC status
# Support for QoS in Manila (zhongjun2)
#* https://blueprints.launchpad.net/manila/+spec/manila-support-qos
#* https://wiki.openstack.org/wiki/Manila/QoS
# Reminder for backend driver maintainers to update feature support table (toabctl)
#* http://docs.openstack.org/developer/manila/devref/share_back_ends_feature_support_mapping.html
# Regression when using NetApp - license issue with netapp_lib (toabctl)
#* https://bugs.launchpad.net/cinder/+bug/1499334
# Mitaka Design Summit
#* Propose topics - https://etherpad.openstack.org/p/manila-mitaka-summit-topics
# Vendor driver docs
# Open Discussion

http://eavesdrop.openstack.org/meetings/manila/

Manila/Meetings

2015-11-12T13:08:25Z

Alex Meade: /* Next meeting */

= Weekly Manila team meeting =
'''NOTE MEETING TIME: Thursday at 15:00 UTC'''

If you're interested in management of shared filesystems for OpenStack, we have a weekly meetings in <code><nowiki>#openstack-meeting-alt</nowiki></code>, on Thursdays at 15:00 UTC. Please feel free to add items to the agenda below. NOTE: When adding topics please include your IRC name so we know who's topic it is and how to get more info.

== Next meeting ==
'''NOTE:''' ''Include your IRC nickname next to agenda items so that you can be called upon in the meeting and arrive at the meeting promptly if placing items in agenda. You might want to put this on your calendar if you are adding items.''

'''NOTE:''' USA-based attendees should remember to double check the meeting time as the end of daylight savings time will move the meeting earlier in your day.

'''Scheduled for 12 Nov, 2015, 15:00 UTC'''

# Which method is better for Manila QoS(independent way or just use extra specs )
#* link:http://paste.openstack.org/show/477677/
#*If use extra specs to config QoS, Does we first need add “QoS_support=true/false” in share_type. Does we need to add QoS_support flag in common capabilities?
# Manila DR update
#* Still no reviews on https://review.openstack.org/#/c/238572/
#* First-party driver implementation? Which driver, current generic or new solution? When do we need this implemented and who wants to do it?
#* Replication_change share state - This is the state of the share when any of it's replicas are being promoted. Is 'replication_change' good or is there a better phrase?
# Open Discussion

== Previous meetings ==

'''Scheduled for 5 Nov, 2015, 15:00 UTC'''
# Decide on read-only access rules as a required feature
# Manila DR - Design/Code feedback (gouthamr)
#* [https://wiki.openstack.org/wiki/Manila/design/manila-mitaka-data-replication Design Document]
#* [https://review.openstack.org/#/c/238572/ Core Implementation on Gerrit]
#* [https://review.openstack.org/#/c/235448/ Client Implementation on Gerrit]

'''Scheduled for 1 Oct, 2015, 15:00 UTC'''
# Liberty RC2 status
#* https://launchpad.net/manila/+milestone/liberty-rc2
#* https://review.openstack.org/#/q/project:openstack/manila+branch:stable/liberty,n,z
# Mitaka Design Summit
#* https://etherpad.openstack.org/p/manila-mitaka-summit-topics
# Vendor driver docs
#* https://github.com/openstack/manila/blob/master/doc/source/devref/share_back_ends_feature_support_mapping.rst
# Manila Client Nova Style Microversion Support (cfouts)
#* https://review.openstack.org/#/c/229142/
#* http://docs.openstack.org/developer/manila/devref/api_microversion_dev.html

'''Scheduled for 24 Sep, 2015, 15:00 UTC'''
# 3rd-party CI status
#* http://ec2-54-67-102-119.us-west-1.compute.amazonaws.com:5000/?project=openstack/manila&user=&timeframe=72&start=&end=&page_size=500
#* http://ci-watch.tintri.com/project?project=manila
# Liberty RC status
# Support for QoS in Manila (zhongjun2)
#* https://blueprints.launchpad.net/manila/+spec/manila-support-qos
#* https://wiki.openstack.org/wiki/Manila/QoS
# Reminder for backend driver maintainers to update feature support table (toabctl)
#* http://docs.openstack.org/developer/manila/devref/share_back_ends_feature_support_mapping.html
# Regression when using NetApp - license issue with netapp_lib (toabctl)
#* https://bugs.launchpad.net/cinder/+bug/1499334
# Mitaka Design Summit
#* Propose topics - https://etherpad.openstack.org/p/manila-mitaka-summit-topics
# Vendor driver docs
# Open Discussion

http://eavesdrop.openstack.org/meetings/manila/

Manila/Meetings

2015-11-12T12:53:52Z

Alex Meade: /* Next meeting */

= Weekly Manila team meeting =
'''NOTE MEETING TIME: Thursday at 15:00 UTC'''

If you're interested in management of shared filesystems for OpenStack, we have a weekly meetings in <code><nowiki>#openstack-meeting-alt</nowiki></code>, on Thursdays at 15:00 UTC. Please feel free to add items to the agenda below. NOTE: When adding topics please include your IRC name so we know who's topic it is and how to get more info.

== Next meeting ==
'''NOTE:''' ''Include your IRC nickname next to agenda items so that you can be called upon in the meeting and arrive at the meeting promptly if placing items in agenda. You might want to put this on your calendar if you are adding items.''

'''NOTE:''' USA-based attendees should remember to double check the meeting time as the end of daylight savings time will move the meeting earlier in your day.

'''Scheduled for 12 Nov, 2015, 15:00 UTC'''

# Which method is better for Manila QoS(independent way or just use extra specs )
#* link:http://paste.openstack.org/show/477677/
#*If use extra specs to config QoS, Does we first need add “QoS_support=true/false” in share_type. Does we need to add QoS_support flag in common capabilities?
# Manila DR update
#* Still no reviews on https://review.openstack.org/#/c/238572/
#* First-party driver implementation? Which driver, current generic or new solution? When do we need this implemented and who wants to do it?
# Open Discussion

== Previous meetings ==

'''Scheduled for 5 Nov, 2015, 15:00 UTC'''
# Decide on read-only access rules as a required feature
# Manila DR - Design/Code feedback (gouthamr)
#* [https://wiki.openstack.org/wiki/Manila/design/manila-mitaka-data-replication Design Document]
#* [https://review.openstack.org/#/c/238572/ Core Implementation on Gerrit]
#* [https://review.openstack.org/#/c/235448/ Client Implementation on Gerrit]

'''Scheduled for 1 Oct, 2015, 15:00 UTC'''
# Liberty RC2 status
#* https://launchpad.net/manila/+milestone/liberty-rc2
#* https://review.openstack.org/#/q/project:openstack/manila+branch:stable/liberty,n,z
# Mitaka Design Summit
#* https://etherpad.openstack.org/p/manila-mitaka-summit-topics
# Vendor driver docs
#* https://github.com/openstack/manila/blob/master/doc/source/devref/share_back_ends_feature_support_mapping.rst
# Manila Client Nova Style Microversion Support (cfouts)
#* https://review.openstack.org/#/c/229142/
#* http://docs.openstack.org/developer/manila/devref/api_microversion_dev.html

'''Scheduled for 24 Sep, 2015, 15:00 UTC'''
# 3rd-party CI status
#* http://ec2-54-67-102-119.us-west-1.compute.amazonaws.com:5000/?project=openstack/manila&user=&timeframe=72&start=&end=&page_size=500
#* http://ci-watch.tintri.com/project?project=manila
# Liberty RC status
# Support for QoS in Manila (zhongjun2)
#* https://blueprints.launchpad.net/manila/+spec/manila-support-qos
#* https://wiki.openstack.org/wiki/Manila/QoS
# Reminder for backend driver maintainers to update feature support table (toabctl)
#* http://docs.openstack.org/developer/manila/devref/share_back_ends_feature_support_mapping.html
# Regression when using NetApp - license issue with netapp_lib (toabctl)
#* https://bugs.launchpad.net/cinder/+bug/1499334
# Mitaka Design Summit
#* Propose topics - https://etherpad.openstack.org/p/manila-mitaka-summit-topics
# Vendor driver docs
# Open Discussion

http://eavesdrop.openstack.org/meetings/manila/

Manila/Meetings

2015-11-12T12:51:09Z

Alex Meade: /* Weekly Manila team meeting */

= Weekly Manila team meeting =
'''NOTE MEETING TIME: Thursday at 15:00 UTC'''

If you're interested in management of shared filesystems for OpenStack, we have a weekly meetings in <code><nowiki>#openstack-meeting-alt</nowiki></code>, on Thursdays at 15:00 UTC. Please feel free to add items to the agenda below. NOTE: When adding topics please include your IRC name so we know who's topic it is and how to get more info.

== Next meeting ==
'''NOTE:''' ''Include your IRC nickname next to agenda items so that you can be called upon in the meeting and arrive at the meeting promptly if placing items in agenda. You might want to put this on your calendar if you are adding items.''

'''NOTE:''' USA-based attendees should remember to double check the meeting time as the end of daylight savings time will move the meeting earlier in your day.

'''Scheduled for 12 Nov, 2015, 15:00 UTC'''

# Which method is better for Manila QoS(independent way or just use extra specs )
#* link:http://paste.openstack.org/show/477677/
#*If use extra specs to config QoS, Does we first need add “QoS_support=true/false” in share_type. Does we need to add QoS_support flag in common capabilities?
# Manila DR update
#* First-party driver implementation? Which driver, current generic or new solution? When do we need this implemented and who wants to do it?
# Open Discussion

== Previous meetings ==

'''Scheduled for 5 Nov, 2015, 15:00 UTC'''
# Decide on read-only access rules as a required feature
# Manila DR - Design/Code feedback (gouthamr)
#* [https://wiki.openstack.org/wiki/Manila/design/manila-mitaka-data-replication Design Document]
#* [https://review.openstack.org/#/c/238572/ Core Implementation on Gerrit]
#* [https://review.openstack.org/#/c/235448/ Client Implementation on Gerrit]

'''Scheduled for 1 Oct, 2015, 15:00 UTC'''
# Liberty RC2 status
#* https://launchpad.net/manila/+milestone/liberty-rc2
#* https://review.openstack.org/#/q/project:openstack/manila+branch:stable/liberty,n,z
# Mitaka Design Summit
#* https://etherpad.openstack.org/p/manila-mitaka-summit-topics
# Vendor driver docs
#* https://github.com/openstack/manila/blob/master/doc/source/devref/share_back_ends_feature_support_mapping.rst
# Manila Client Nova Style Microversion Support (cfouts)
#* https://review.openstack.org/#/c/229142/
#* http://docs.openstack.org/developer/manila/devref/api_microversion_dev.html

'''Scheduled for 24 Sep, 2015, 15:00 UTC'''
# 3rd-party CI status
#* http://ec2-54-67-102-119.us-west-1.compute.amazonaws.com:5000/?project=openstack/manila&user=&timeframe=72&start=&end=&page_size=500
#* http://ci-watch.tintri.com/project?project=manila
# Liberty RC status
# Support for QoS in Manila (zhongjun2)
#* https://blueprints.launchpad.net/manila/+spec/manila-support-qos
#* https://wiki.openstack.org/wiki/Manila/QoS
# Reminder for backend driver maintainers to update feature support table (toabctl)
#* http://docs.openstack.org/developer/manila/devref/share_back_ends_feature_support_mapping.html
# Regression when using NetApp - license issue with netapp_lib (toabctl)
#* https://bugs.launchpad.net/cinder/+bug/1499334
# Mitaka Design Summit
#* Propose topics - https://etherpad.openstack.org/p/manila-mitaka-summit-topics
# Vendor driver docs
# Open Discussion

http://eavesdrop.openstack.org/meetings/manila/

ThirdPartySystems/NetApp FC CI

2015-10-09T14:17:35Z

Alex Meade:

{{ThirdPartySystemInfo|name=NetApp FC CI
|account=netapp-fc-ci
|contact= xdl-openstack-jenkins@netapp.com, andrew.kerr@netapp.com (akerr), alex.meade@netapp.com (ameade)
|intent=Test Openstack with NetApp Fibre Channel Cinder drivers using storage arrays
|structure=Nodepool, Zuul, Jenkins, modified upstream components with local configuration
|method=Using Zuul to connect to Gerrit
|programs=Cinder
|status= transitioning to be merged with the 'netapp-ci' system}}

Manila/Replication Use Cases

2015-10-07T18:56:26Z

Alex Meade: /* Unanswered Questions */

== Answered Questions ==

Q: From where to where do we allow replication? Is it intra-cloud or inter-cloud? Do we allow replication to something that's not managed by Manila?

A: Intra-cloud. Replicating to something outside of Manila allows a bit more freedom, but with significantly less value, because there's practically nothing we can do to automate the failover/failback portion of a disaster. For use cases involving replication outside of Manila, we would need to involve other tools with more breadth/scope to manage the process.

Q: Do we support unplanned failovers? Do we support planned failovers? Are failovers disruptive or not?

A: Failovers can be planned or unplanned, but they are always disruptive at the data level. With application integration, they could be made nondisruptive at the application level, but unfortunately we've chosen no to use any intermediary technology (like virtfs) in the data path, which means we have no options for non disruptive failovers.

Q: Who configures the replication? The admin? The end user? The manila scheduler?

A: The end user. In the original design we presumed that the actual replication relationships should be hidden from the end user, but this doesn't match well with the concept of AZs that we are adding to Manila. If the users need to have control over which AZ the primary copy of their data lives in, then they also need to control where the other copies live. This means that the administrator's job is to ensure that for any share type that is replicated, it can be replicated from any AZ to any other AZ.

Q: Who triggers a failover? Is it a manual button the admin presses? Can Manila failover automatically? If so, when? Can the end users control failovers at all?

A: Failovers are manual, triggered by either an administrator or a user. Generally speaking it's more appropriate for the administrator to initiate a failover because the administrator has more knowledge about the nature of an outage. However, it's also essential for end users to test failovers so they need the capability to initiate failovers of shares themselves.

Q: During replication (before failover) is the secondary even visible/accessible?

A: Yes, but possibly with significant limitations. Some backends may not support accessing the secondary side of a replicated share. Some backends may allow access, but read-only. We know of at least one backend that can support write access to the secondary (in which case calling it a secondary isn't really accurate because it's more of an active-active relationship). Amazon's EFS has the model of active-active replication so it's something we don't want to disallow.

Q: What is the granularity of the failover? Whole backend? Single pools?

A: Individual shares. There's no technical reason to prevent failover/failback on share-by-share basis. To make the administrator's life easier, we also have to support whole-backend failover (could be essential to minimize downtime in an actual disaster). The ability to do single-share failover is nice because it allows testing of the DR system without triggering an outage that affects users, since failovers are disruptive.

== Unanswered Questions ==

There are some major unanswered questions (or areas of investigation).

1) Is there no way to achieve nondisruptive failover? I would love to find out that our initial intuition here is wrong, because it would change a lot of aspects of the design. It's worth spending time to brainstorm and research possibilities in this area. So far the most promising ideas involve:
* Using VirtFS to mediate filesystem access and achieving non-disruptive failover that way
* Using some kind of agent inside the guests to mediate file access

2) How do we deal with recovery after a disaster and failover? Assuming a successful failover, and a repair of the original primary, failing back will cause another outage. How can we orchestrate that to minimize pain and suffering?

3) Assuming the disruptive aspect of failovers is unavoidable, how can we invest to make them less painful at the application level?
Application quiescing and mount automation could make failovers nondisruptive for a least a select few applications.

4) Can the replica have a different share_type?

There is a valid use case where a user would want to create a share replica on backend with different capabilities than the one the original share resides on. For instance, replicas might need to be on a less expensive backend. In that case, can the replica have a different share_type altogether?

"Currently", we inherit the share_type of the share and believe that replication has to be on symmetric terms, where both backends have similar capabilities.

5) Can we allow the driver to restrict replication support between available backends?

Backends may support replication to other compatible backends only. Hence, they must report some sort of information to the scheduler so that when creating a replica for an existing share, the scheduler would use that information to schedule the creation of the replica. What information should this be?

6) How are access rules persisted across replicas/share instances?
* Do all replicas have the same access rules applied? ''(Currently being pursued)''
* Should access rules be applied only to "active" replicas?

7) How does migration affect replicated shares?

8) API Endpoints:
* Current Endpoint Design: https://wiki.openstack.org/wiki/Manila/Replication_API_Design#New_APIs

9) What export locations should appear when showing a replicated share?
* All export locations from available "active" replicas ''(Currently being pursued)''
* Export locations of active instances
* Export locations that are within the AZ of the active instance

10) Where do we store the "replication_change" status?
* On the instance "status" for the instance we are promoting ''(Currently being pursued)''
* On all the replica instances of the replicated share
* As a new status on the share itself

11) 'replica_state' field should be 'healthy' or 'error' as current options 'in-sync' and 'out-of-sync' may signify synchronous and async replication.
* Discussions with Ben in IRC
** "the idea is that your redefine in-sync to mean "within the RPO guarantee of the async replication""
** The user should know if it's async or sync replication by the 'description' in the share type.
** "the main thing that in-sync/out-of-sync is supposed to communicate is whether the replica is "good enough" that you could fail over to it"
12) 'replication' field on the share should qualify whether the replication is sync or async, so the options should be ["readable_sync", "readable_async", "writable", "dr_sync", "dr_async"]
* Discussions with Ben in IRC
** The user should know if it's async or sync replication by the 'description' in the share type.

Manila/Replication API Design

2015-09-23T16:57:59Z

Alex Meade: /* New APIs */

= Design =

== Intro ==

The Manila DR API will be implemented as an extension to the Manila API (not part of core) initially, because we want to prove the concept without investing heavily in support in reference implementation (generic driver) or automated testing/CI.

== Replication styles ==

There are 3 styles of DR that we would like to support in the long run:
# writeable - Amazon EFS-style synchronously replicated shares where all replicas are writeable. Failover it not supported and not needed.
# readable - Mirror-style replication with a primary (writeable) copy and one or more secondary (read-only) copies which can become writeable after a failover.
# dr - Generalized replication with secondary copies that are inaccessible until after a failover.

== Replica states ==

Each replica has a state which has 3 possible values:
# active - All writeable replicas are active
# in_sync - Passive replica which is up to date with the active replica, and can be promoted to active
# out_of_sync - Passive replica which has gone out of date, or new replica that is not yet up to date

== New APIs ==

We will implement a Manila extension that includes several new APIs needed to support replicated shares.

* List share replicas - Takes a share ID. Returns a table of replicas with details. Must be a replicated share. Details include AZ, replica state (active, in_sync, out_of_sync) and export locations.
** GET /shares/<share-id>/replicas
* Show share replica details
** GET /shares/<share-id>/replicas/<replica-id>
* Add share replica - Takes a share ID, and an AZ. The share must not already have a replica in the specified AZ. Returns replica UUID (actually share instance UUID).
** POST /shares/<share-id>/replicas
* Remove share replica - Takes a replica UUID. Deletes the replica, regardless of state. Must not be the only active replica.
** DELETE /shares/<share-id>/replicas/<replica-id>
* Set active replica - Takes a replica UUID. Make that replica active. The state of the replica must be in_sync.
** POST /shares/<share-id>/action
** {'os-promote_replica': {'replica_id': <replica-id>}}

== New share states ==

* replication_change - New transient state triggered by a change of the active replica. Access to the share is cut off while in this state.

== Changes to existing APIs ==

* Share type APIs will have a new user-visible extra spec - replication=writeable/readable/dr. The absence of this extra spec indicates non-replicated shares and the presence of the extra spec will indicate that the share is replicated with the given style.
* Share create will create a replicated share if the share type is has replication extra spec. The style of replication is determined by the share type's replication vale.
* Share list/details APIs will return the replication style (writeable, readable, dr) and a flag if more than one replica exists.
* Create snapshot - creates snapshots of all the replicas
* Delete share/snapshot - deletes ALL replicas of the share/snapshot
* Migrate/retype/etc - only the primary replica is considered as the source

== Network issues with multi-SVM and replication ==

!!OPTIONAL!!

If we choose to make replication a single-svm-only feature, the share-network
API doesn't need to change. In order to support replication with share-networks, we also
need to modify the share-network create API which allows creation of share networks with
a table of AZ-to-subnet mappings. This approach allows us to keep a
single share-network per share (with associated security service) while allowing the
tenant to specify enough information that each share instance can be attached to the
appropriate network in each AZ. Multi-AZ share networks would also be useful for
non-replicated use cases.

= Examples =

== Writable replication example ==

# Administrator sets up backends in AZs b1 and b2 that have capability replication=writeable
# Administrator creates a new share_type called foo
# Administrator sets replication=writeable extra spec on share type foo
# User creates new share of type foo in AZ b1
# Share is created with replication=writeable, and 1 active replica in AZ b1
# User grants access on share to client1 in AZ b1, obtains the export location of the replica, mounts the share on a client, and starts to write data
# User add new replica of share in AZ b2
# A second replica is created in AZ b2 which initially has state out_of_sync
# Shortly afterwards, the replica state changes to active (after the replica finishes syncing with the original copy)
# The user grants access on the share to client2 in AZ b2, obtains the export location of the new replica, mounts the share, and sees the same data that client1 wrote
# Client2 writes some data to the share, which is immediately visible to client1

== Readable replication example ==

# Administrator sets up backends in AZs b1 and b2 that have capability replication=readable
# Administrator creates a new share_type called bar
# Administrator sets replication=readable extra spec on share type bar
# User creates new share of type bar in AZ b1
# Share is created with replication=readable, and 1 active replica in AZ b1
# User grants access on share to client1 in AZ b1, obtains the export location of the replica, mounts the share on a client, and starts to write data
# User add new replica of share in AZ b2
# A second replica is created in AZ b2 which initially has state out_of_sync
# Shortly afterwards, the replica state changes to in_sync (after the replica finishes syncing with the original copy)
# The user grants access on the share to client2 in AZ b2, obtains the export location of the new replica, mounts the share, and sees the same data that client1 wrote
# Client2 cannot write data to the share but continues to see updates

== Failover/failback example ==

(Continued from above)
# An outage occurs in AZ b1
# Administrator sends out a bulletin about the outage to his users "the power transformer in b1 turned to slag, it will be 12 hours before it can be replaced, please bear with us, yada yada"
# User notices that his application on client1 is no longer running, and investigates
# User finds out that client1 is gone, and reads a bulletin from the admin explaining why
# User notes that the b1 replica is his share is still active and the b2 replica is in_sync, while the state of the share is available
# User calls set active replica to AZ b2 on his share
# The share goes to state replication_change and access to the share is briefly lost on client2
# The state of replica b2 changes to active and the state of replica b1 changes to out_of_sync after Manila fails to contact the original primary. The state of the share changes back to available, and access is restored on client2
# User starts his application on client2, and the application recovers from an apparent crash, with a consistent copy of the application's data
# User application is back up and running, disaster is averted
# Eventually maintenance on the b1 AZ is completed, and all of the equipment is reactivated
# Administrator sends out a bulletin to his users about the outage ending "we replaced the transformer with a better one, everything is back online, thanks for your understanding, yada yada"
# Manila notices the out_of_sync replica is reachable and initiates a resync, bringing the b1 replica in_sync within a short time
# After some time, the users reads the bulletin and observes that his share is being replicated again. He decides to intentionally move back to b1.
# User gracefully shuts down his application, and flushes I/O
# User calls set active replica to AZ b1 on his share
# The share goes to state replication_change and no disruption is observed because the application is shut down
# The state of replica b1 changes to active and the state of replica b2 changes to in_sync after Manila reverses the replication again. The state of the share changes back to available
# User starts his application on client1, and the application starts cleanly, having been shut down gracefully

= Notes =

== Pool-level Replication ==

Some vendors have suggested that certain backends can mirror groups of shares or whole pools more efficiently than individual shares. This design only addresses the mirroring of individual shares. In the future, we may allow allow replication of groups of shares, but only if those groups are contained within a single tenant and defined by the tenant.

Manila/design/manila-liberty-consistency-groups/api-schema

2015-08-15T20:00:24Z

Alex Meade: /* Detailed CG */

= Resource schemas=

== Simple CG ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer"
},
"name": {
"description": "A non-unique name for a CG",
"type": "string"
},
},
"required": ["id", "name"]
}

== Detailed CG ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "string",
'maxLength': 36,
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"share_network_id": {
"description": "The share network to provision the CG and all of it's shares into",
"type": "string",
'maxLength': 36,
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"share_server_id": {
"description": "The share server on which the CG is provisioned, visible only to admins",
"type": "string",
'maxLength': 36,
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"created_at": {
"description": "The time at which the CG was created",
"type": "string"
},
"status": {
"description": "Status of the CG",
"type": "string",
"enum": ["creating", "available", "deleting", "error"]
},
"project_id": {
"description": "The owner of the CG",
"type": "string"
},
"host": {
"description": "The backend where the CG lives",
"type": "string"
},
"source_cgsnapshot_id": {
"description": "The uuid of the cgsnapshot from which this CG was created.",
"type": ['null', 'string'],
},
"share_types": {
"description": "The share types that shares in the CG are allowed to be a part of",
"type": "array",
"items": {
"type": "string"
},
},
},
"required": ["id", "name", "description", "created_at", "status", "project_id", "host"]
}

== Simple CG Snapshot ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CGsnapshot",
"type": "string"
},
},
"required": ["id", "name"]
}

== Detailed CG Snapshot ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CG snapshot",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG snapshot",
"type": "string"
},
"created_at": {
"description": "The time at which the CG snapshot was created",
"type": "string"
},
"status": {
"description": "Status of the CG snapshot",
"type": "string",
"enum": ["creating", "available", "deleting", "error"]
},
"project_id": {
"description": "The owner of the CG snapshot",
"type": "string"
},
"consistency_group_id": {
"description": "The uuid of the consistency group from which this CG snapshot was created.",
"type": ['null', 'string'],
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
},
"required": ["id", "name", "description", "created_at", "status", "project_id"]
}

== CG Snapshot member ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot Member",
"description": "A resource representing a member of a CGSnapshot",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"cgsnapshot_id": {
"description": "The unique identifier for the parent cg_snapshot",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"share_id": {
"description": "The unique identifier for the share this snapshot was created from.",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"share_type_id": {
"description": "The unique identifier for the share_type of the share this snapshot was created from.",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"size": {
"description": "The storage size of this snapshot",
"type": "integer",
},
"share_protocol": {
"description": "The share protocol of the parent share",
"type": "string",
},
},
}

= Request schemas =

== POST /os-consistency-groups ==
{ "consistency_group" {
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"source_cgsnapshot_id": {
"description": "The uuid of the cgsnapshot from which this CG was created. Cannot be supplied when 'share_types' is provided.",
"type": ['null', 'string'],
},
"share_types": {
"description": "The share types that shares in the CG are allowed to be a part of. Cannot be supplied when 'source_cgsnapshot_id' is provided.",
"type": "string"
},
},
"required": []
}
}

Response: {'consistency_group": detailed CG}

== GET /os-consistency-groups ==
Response:
{'consistency_groups": [Simple CG]}

== GET /os-consistency-groups/detail ==
Response:
{'consistency_groups": [Detailed CG]}

== GET /os-consistency-groups/<id> ==
Response: {'consistency_group": Detailed CG}

== PUT /os-consistency-groups/<id> ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"required": []
}

Response: {'consistency_group": Detailed CG}

== POST /os-cgsnapshots ==
{'cgsnapshot": {
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"consistency_group_id": {
"description": "The uuid of the CG from which to create this CG snapshot.",
"type": 'string',
},
},
"required": ['consistency_group_id']
}}
Response: {'cgsnapshot": detailed CGSnapshot}

== PUT /os-cgsnapshots/<id> ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"required": []
}

Response: {'cgsnapshot": Detailed CGSnapshot}

Manila/design/manila-liberty-consistency-groups

2015-08-15T19:59:04Z

Alex Meade:

= Consistency Groups =
This document describes the design choices and steps for the consistency group feature in core Manila.

Consistency groups are a mechanism to allow multiple filesystem shares to be guaranteed snapshots are able to be created at the exact same point in time. For example, a database may have its tables, logs, and configuration on separate volumes. If we wanted to restore the database from a previous point in time, it would only make sense to restore the logs, tables, and configuration together from the exact same point in time.

= Manila Core Changes =
The core manila changes are very similar to what has been done in Cinder. At a high level, changes in the DB, share service, drivers, scheduler service and api service will need to be changed.

Phase 1:
* Create/Delete CGs
* Create a share within a CG
* Delete a share that is in a CG
* Snapshot entire CGs
* Create CG from a CGSnapshot

Potential future features:
* Add/remove shares in a CG
* CG migration
* Replication of a CG

==Manila vs Cinder==
* Different CLI syntax
* Restful api tweaks
** The api will be an 'experimental' microversion in the Liberty release
** Change /create_from_src to just be a cgsnapshot_id in the body on POST /consistency-groups
** Deletes to use the DELETE HTTP verb, os-force_delete is supported via the admin actions extension (/action)
* At the Driver API, cgsnapshot objects and cg objects (with all the information about what volumes are in the CG) need to be passed to the driver instead of just the cg/cgsnapshot name. This is important for backends that aren't managing a CG construct such as NetApp cDOT as they need to do things such as loop through the snapshots in a cg to delete them.
* The default policy in policy.json for consistency group operations is the default policy versus in Cinder is it 'nobody'
* Unlike Cinder, snapshots in a CGsnapshot are not the same as a normal snapshot. A CGsnapshot is treated as a single unit instead of a collection of snapshots.
* consistency_group_support scheduler capability has multiple values instead of boolean. This is to accommodate backends that require all volumes in a CG to be on the same pool vs. just the same host. It also allows for adding additional values in the future if a backend can support some other paradigm like a CG spanning multiple backends.

== User Workflows ==
* Snapshot multiple shares
*# Create a Consistency Group
*# Create share(s) with a CG specified
*# Create a CGsnapshot

* Create a CG
*# POST to /consistency-group
*#* specifiy share_types or the default share type will be used

* Copy a consistency group
*# Create a CGsnapshot
*# Create a CG with the CGSnapshot id

* Delete a CG
*# Delete all CG snapshots of the CG
*# Delete all shares in the CG
*# Delete the CG (Only empty CGs can be deleted)

* List shares in a given CG
*# GET /shares?consistencygroup_id=<cg_id>

* List the shares that were captured in a given CGSnapshot
*# GET /cgsnapshots/<id>/members

== System Workflows ==
* Creating a CG
*# Create the CG in the DB
*# Cast the consistency group creation to the scheduler
*## Choose a host for the CG
*# Cast the creation to the share service
*## Get the CG info from DB
*## Call the driver
*## Update database with info returned from driver (ex: CG status)

* Creating a CG from a cgsnapshot
*# Create a new CG in the DB
*# Create new shares from CG snapshot members in the DB
*# Schedule the CG create to the scheduler with the orig CG host/pool+share_type (depending on capability)
*# Call the driver with all info about source shares, new shares, cg, and cgsnapshot info.
*# Update CGsnapshot status and all shares status' to available

* Creating a cg snapshot
*# Create cgsnapshot entity in DB
*# For all shares in CG, create cgsnapshot_member in DB
*# cast create_cgsnapshot to the driver

* Adding a share to a CG
** On share creation:
**# Cast to scheduler with the host of the CG
**#* if perpool capable: Goes to the same pool as CG
**#* if per backend: goes to a pool on the same host that matches a CG share type

* Deleting a share (in a CG)
** Not allowed if the CG has snapshots of the share ( check if the share_id is in the CGSnapshotMembers tables)

== Scheduler ==
manila/scheduler/filter_scheduler.py should now look for consistency_group_support as a capability.

consistency_group_support
default: None 
Values:
* None - No support for CGs
* host - shares in a CG must be on pool(s) on the same host that also match the CG share type
* pool - shares in a CG must live in the same pool as the CG

==DB==
* New Tables:
class ConsistencyGroup(BASE, ManilaBase):
"""Represents a consistencygroup."""
__tablename__ = 'consistency_groups'
id = Column(String(36), primary_key=True)

user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)
deleted = Column(String(36), default='False')

host = Column(String(255))
name = Column(String(255))
description = Column(String(255))
status = Column(String(255))
source_cgsnapshot_id = Column(String(36))
share_network_id = Column(String(36), ForeignKey('share_networks.id'),
nullable=True)
share_server_id = Column(String(36), ForeignKey('share_servers.id'),
nullable=True)

class ConsistencyGroupShareTypeMapping(BASE, ManilaBase):
"""Represents the share types in a consistency group"""
__tablename__ = 'consistency_group_share_type_mappings'
id = Column(String(36), primary_key=True)
deleted = Column(String(36), default='False')
consistency_group_id = Column(String(36),
ForeignKey('consistency_groups.id'),
nullable=False)
share_type_id = Column(String(36),
ForeignKey('share_types.id'),
nullable=False)
consistency_group = orm.relationship(
ConsistencyGroup,
backref="share_types",
foreign_keys=consistency_group_id,
primaryjoin=('and_('
'ConsistencyGroupShareTypeMapping.consistency_group_id '
'== ConsistencyGroup.id,'
'ConsistencyGroupShareTypeMapping.deleted == "False")')
)

class CGSnapshot(BASE, ManilaBase):
"""Represents a cgsnapshot."""
__tablename__ = 'cgsnapshots'
id = Column(String(36), primary_key=True)

consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'))
user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)
deleted = Column(String(36), default='False')

name = Column(String(255))
description = Column(String(255))
status = Column(String(255))

consistency_group = orm.relationship(
ConsistencyGroup,
backref="cgsnapshots",
foreign_keys=consistency_group_id,
primaryjoin=('and_('
'CGSnapshot.consistency_group_id == ConsistencyGroup.id,'
'CGSnapshot.deleted == "False")')
)

class CGSnapshotMembers(BASE, ManilaBase):
__tablename__ = 'cgsnapshot_members'
id = Column(String(36), primary_key=True)
cgsnapshot_id = Column(String(36), ForeignKey('cgsnapshots.id'))
share_id = Column(String(36), ForeignKey('shares.id'))
size = Column(Integer)
status = Column(String(255))
share_proto = Column(String(255))
share_type_id = Column(String(36), ForeignKey('share_types.id'),
nullable=True)
user_id = Column(String(255))
project_id = Column(String(255))
deleted = Column(String(36), default='False')

cgsnapshot = relationship(
ConsistencyGroup,
backref="cgsnapshot_members",
foreign_keys=cgsnapshot_id,
primaryjoin='CGSnapshotMembers.cgsnapshot_id == CGSnapshot.id')

share = orm.relationship(Share, backref="cgsnapshot_members",
foreign_keys=share_id,
primaryjoin='and_('
'CGSnapshotMember.share_id == Share.id,'
'CGSnapshotMember.deleted == "False")')

* New fields in Share
consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'), nullable=True)
consistency_group = relationship(
ConsistencyGroup,
backref="shares",
foreign_keys=consistency_group_id,
primaryjoin='Share.consistency_group_id == ConsistencyGroup.id')

source_cgsnapshot_member_id = Column(String(36), nullable=True)

==API==
Schemas: https://wiki.openstack.org/wiki/Manila/design/manila-liberty-consistency-groups/api-schema

'''GET''' /consistency-groups/
'''GET''' /consistency-groups/detail
'''POST''' /consistency-groups
'''GET''' /consistency-groups/<id>
'''DELETE''' consistency-groups/<id>
'''PUT''' /consistency-groups/<id>

'''GET''' /cgsnapshots/
'''GET''' /cgsnapshots/detail
'''POST''' /cgsnapshots
'''PUT''' /cgsnapshots/<id>
'''GET''' /cgsnapshots/<id>
'''GET''' /cgsnapshots/<id>/members
'''DELETE''' /cgsnapshots/<id>

Admin Actions
'''POST''' /consistency-groups/<id>/action # os-reset-status and os-force-delete
Body:
{"os-reset_status": { "status": "available"}}
'''POST''' /cgsnapshots/<id>/action # os-reset-status and os-force-delete
Body:
{"os-force_delete": null}

Updates to Share resource
Adds consistency_group_id to POST/shares
Adds source_cgsnapshot_member_id
Add consistency_group_id query filter to /shares

==Policies==
policy.json - All consistency group policies should default to the default policy
"consistency_group:create" : "",
"consistency_group:delete": "",
"consistency_group:update": "",
"consistency_group:get": "",
"consistency_group:get_all": "",

"consistency_group:create_cgsnapshot" : "",
"consistency_group:delete_cgsnapshot": "",
"consistency_group:get_cgsnapshot": "",
"consistency_group:get_all_cgsnapshots": "",

==Driver API==

update_share_stats should now return 'consistency_group_support' pool/host/None.

def create_consistency_group(self, context, cg_dict, share_server=None):
"""Create a consistency group.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_consistency_group_from_cgsnapshot(self, context, cg_dict,
cgsnapshot_dict, share_server=None):
"""Create a consistency group from a cgsnapshot.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'shares': [<models.Share>], # The new shares being created
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:param cgsnapshot_dict:
EXAMPLE:
{'status': 'available',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': '0',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'consistency_group_id': '4b04fdc3-00b9-4909-ba1a-06e9b3f88b67',
'cgsnapshot_members': [
{'status': 'available',
'share_type_id': '1a9ed31e-ee70-483d-93ba-89690e028d7f',
'share_id': 'e14b5174-e534-4f35-bc4f-fe81c1575d6f',
'user_id': 'a0314a441ca842019b0952224aa39192',
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share': <models.Share>,
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share_proto': 'NFS',
'project_id': '13c0be6290934bd98596cfa004650049',
'cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'deleted_at': None,
'id': '6813e06b-a8f5-4784-b17d-f3e91afa370e',
'size': 1
}
],
'deleted_at': None,
'id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.

share_update_list - a list of dictionaries containing dicts for
every share created in the CG. Any share dicts should at a minimum
contain the 'id' key and 'export_locations'. Export locations
should be in the same format as returned by a share_create. This
list may be empty or None.
EXAMPLE:
[
{'id': 'uuid', 'export_locations': ['export_path']}
]
"""
raise NotImplementedError()

def delete_consistency_group(self, context, cg_dict, share_server=None):
"""Delete a consistency group

:param context:
:param cg_dict:
:return: cg_model_update
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_cgsnapshot(self, context, snap_dict, share_server=None):
"""Create a consistency group snapshot.

:param context:
:param snap_dict:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.

member_update_list - a list of dictionaries containing for every
member of the cgsnapshot. Each dict should contains values to be
updated for teh CGSnapshotMember in the database. This list may be
empty or None.
"""
raise NotImplementedError()

def delete_cgsnapshot(self, context, snap_dict, share_server=None):
"""Delete a consistency group snapshot

:param context:
:param snap_dict:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.
"""
raise NotImplementedError()

== Manila Client ==
* Do we want to alias the commands so that there are Cinder consistent commands and the improved names? (I.E cg-create = consisgroup-create) Perhaps we alias the cinderclient commands to be more like this?
cg-snapshot-create Creates a cgsnapshot.
cg-snapshot-delete Removes one or more cgsnapshots.
cg-snapshot-list Lists all cgsnapshots.
cg-snapshot-show Shows cgsnapshot details.

cg-create Creates a consistency group (--cgsnapshot to create from existing cg snapshot)
cg-delete Removes one or more consistency groups.
cg-list Lists all consistencygroups.
cg-show Shows details of a consistency group.
cg-update Updates a consistencygroup.

create --consistency-group Creates a share and puts it into consistency group

= Outstanding Questions / FAQ =
# How do we avoid confusion if no backends in a deployment support CGs?
#* It appears that Cinder has the policies default to 'nobody' but this seems odd, is there a better way?
#** (ameade) I can't think of a way to have devstack set the policies automatically either.
#** (ameade) we have chosen to go with an os-consistency-groups extension for now
# Do we need a way to fake CGs with the generic driver?
#* <strike> NO, but it will not be a core feature if it cannot be supported in the generic driver.</strike>
#* Ben says that we should fake it in the generic driver so it can be tested by the gate.
# Do we need a safeguard for deleting shares that are in a CG? (I.E. protected shares?) Are shares in a CG more valuable than normal shares?
# Should we overload the Snapshots table to include snapshots in a CGSnapshot to avoid adding a new db table? Or continue with having a new table?
# Why not have the cg delete all of the shares in it when deleting the cg?
#* This could cause odd state to occur if the backend fails when deleting shares and it will not be obvious where it failed. This is also extra complexity in the implementation that could instead be hidden by a client.

= Notes on Cinder Impl =

* A consistency group may be specified on volume create
* When a consistency group is created it is scheduled to a pool and all volume created in the CG will go to that pol

Cinder cli:
cgsnapshot-create Creates a cgsnapshot and snapshots of every volume in the CG that show up in snapshot-list.
cgsnapshot-delete Removes one or more cgsnapshots.
cgsnapshot-list Lists all cgsnapshots.
cgsnapshot-show Shows cgsnapshot details. <-- /cgsnapshots/detail?all_tenants=1&name=blah
consisgroup-create Creates a consistency group.
consisgroup-create-from-src Creates a consistency group from a cgsnapshot filled with volumes from the snapshots in the cg.
consisgroup-delete Removes one or more consistency groups.
consisgroup-list Lists all consistencygroups.
consisgroup-show Shows details of a consistency group. <-- /consistencygroups/detail?all_tenants=1&name=blah
consisgroup-update Updates a consistencygroup.

create --consisgroup-id

Cinder API:
'''GET''' /consistencygroups/detail
'''POST''' /consistencygroups
{'consistencygroup':{
'name': <string>,
'description': <string>,
'volume_types': <string>,
'availability_zone': <string>,
}}
'''POST''' /consistencygroups/create_from_src
{'consistencygroup-from-src':{
'name': <string>,
'description': <string>,
'cgsnapshot_id': <string>,
}}
'''GET''' /consistencygroups/<id>
'''POST''' consistencygroups/<id>/delete
{"consistencygroup": {"force": false}}
'''PUT''' /consistencygroups/<id>
{'consistencygroup':{
'name': <string>,
'description': <string>,
'add_volumes': <string>,
'remove_volumes': <string>,
}}

'''GET''' /cgsnapshots/detail
'''GET''' /cgsnapshots/<id>
'''POST''' /cgsnapshots
{'cgsnapshot':{
'consistencygroup_id': <string>,
}}
'''DELETE''' /cgsnapshots/<id>

Adds consistencygroup_id to POST/volumes

Xings commit message:
1) Create a CG, specifying all volume types that can be supported by this
CG. The scheduler chooses a backend that supports all specified volume types.
The CG will be empty when it is first created. Backend needs to report
consistencygroup_support = True. Volume type can have the following in
extra specs: {'capabilities:consistencygroup_support': '<is> True'}.
If consistencygroup_support is not in volume type extra specs, it will be
added to filter_properties by the scheduler to make sure that the scheduler
will select the backend which reports consistency group support capability.

Create CG CLI:
cinder consisgroup-create --volume-type type1,type2 mycg1

This will add a CG entry in the new consistencygroups table.

2) After the CG is created, create a new volume and add to the CG.
Repeat until all volumes are created for the CG.

Create volume CLI (with CG):
cinder create --volume-type type1 --consisgroup-id <CG uuid> 10

This will add a consistencygroup_id foreign key in the new volume
entry in the db.

3) Create a snapshot of the CG (cgsnapshot).

Create cgsnapshot CLI:
cinder cgsnapshot-create <CG uuid>

This will add a cgsnapshot entry in the new cgsnapshots table, create
snapshot for each volume in the CG, and add a cgsnapshot_id foreign key
in each newly created snapshot entry in the db.

===Questions===
# Why does policy.json default to 'nobody' being able to perform CG commands?
# Why are volume types specified when creating a CG?
#* This is because all volumes within a CG need to be on the same backend in order for the backend to honor the CG. This is important to note in the docs so that conflicting volume-types are not specified.
# What occurs when a volume on a different backend is added to a CG via the API?
#* Currently the API ensures that the volume volume_type matches a supported volume_type of the CG and that the CG and volume are on the same host (but can be on different pools).
# Is a volume created with a CG going to end up on the same host?
#* Yes, if a volume has a CG it will be cast to the same pool as the CG.
#* What if the pool is full but there are other available pools on the same backend? Should this fail?
# What happens if a driver does not support create_from_src?
#* The CG goes to ERROR a and volumes are created from the snapshot and also in ERROR status
#* A volume in a consistency group cannot be deleted and a CG with a volume cannot be deleted, therefore a volume must be removed from the CG before it can be deleted. But what if you don't support modifying a CG?
# Why do we need a force delete for CGs?
# What do I do if I do not want to see Snapshots in a CG when doing snapshot-list?
#* I suppose API filtering could allow for someone to not see all of the snapshots if they do not want to.

=== Cinder Bugs to file===
# What happens when you force delete a CG?
#* The volumes remain but still think they are in a CG, which means you still cannot delete the volume
# consisgroup-create-from-src says cgsnapshot is and optional param in cinderclient
# Creating a volume in a CG may not end up in the cg. The only check that is performed is that the volume-type of the volume is supported by the CG but volume-types are not 1-to-1 with backends. There could be backend A and B that both match with volume type 1. A cg is created with volume type 1 and ends up on backend A. Then a volume is created with the CG specified but ends up on backend B

=== Main impl ===
299b99437c2de84222fd585f06c2d05ca213225b ConsistencyGroup: Return 400 instead of 500 for invalid body
adb4c80be82caacad83f1366a4b34e5653fd5dab Create Consistency Group from CG Snapshot API
1a62a6e60fda73bf31256fbf684dc03bb6cf0038 Modify Consistency Group API
fc7a4384be2ccf8ff24f6c0f72c681ad9133801a Add tests for consistency groups DB migration
9082273305b0b9c117eb677a0e34c2ffde4e66f0 Volume types need to be specified when creating CG
764c5ec749821d36bb0215dc6002d3caea95d3b1 Delete consistency group failed
cf961f83ac323dfad1fa5e227d1e502a17529ecc Consistency Groups

=== Ex driver impl ===
8d5b795b37ed021c2639066689645f8aa0b1012f PureISCSIDriver consistency group updates.
39c1a8e08dc854e22ada315a46c20c99df2facf8 Add support to PureISCSIDriver for Consistency Groups
92a817708a3938b1b734d2caaa206b310996d8d0 EMC VMAX driver Kilo update

Manila/design/manila-liberty-consistency-groups

2015-08-15T19:58:29Z

Alex Meade: API extension -> experimental api

= Consistency Groups =
This document describes the design choices and steps for the consistency group feature in core Manila.

Consistency groups are a mechanism to allow multiple filesystem shares to be guaranteed snapshots are able to be created at the exact same point in time. For example, a database may have its tables, logs, and configuration on separate volumes. If we wanted to restore the database from a previous point in time, it would only make sense to restore the logs, tables, and configuration together from the exact same point in time.

= Manila Core Changes =
The core manila changes are very similar to what has been done in Cinder. At a high level, changes in the DB, share service, drivers, scheduler service and api service will need to be changed.

Phase 1:
* Create/Delete CGs
* Create a share within a CG
* Delete a share that is in a CG
* Snapshot entire CGs
* Create CG from a CGSnapshot

Potential future features:
* Add/remove shares in a CG
* CG migration
* Replication of a CG

==Manila vs Cinder==
* Different CLI syntax
* Restful api tweaks
** The api will be an 'experimental' microversion in the Liberty release
** Change /create_from_src to just be a cgsnapshot_id in the body on POST /consistency-groups
** Deletes to use the DELETE HTTP verb, os-force_delete is supported via the admin actions extension (/action)
* At the Driver API, cgsnapshot objects and cg objects (with all the information about what volumes are in the CG) need to be passed to the driver instead of just the cg/cgsnapshot name. This is important for backends that aren't managing a CG construct such as NetApp cDOT as they need to do things such as loop through the snapshots in a cg to delete them.
* The default policy in policy.json for consistency group operations is the default policy versus in Cinder is it 'nobody'
* Unlike Cinder, snapshots in a CGsnapshot are not the same as a normal snapshot. A CGsnapshot is treated as a single unit instead of a collection of snapshots.
* consistency_group_support scheduler capability has multiple values instead of boolean. This is to accommodate backends that require all volumes in a CG to be on the same pool vs. just the same host. It also allows for adding additional values in the future if a backend can support some other paradigm like a CG spanning multiple backends.

== User Workflows ==
* Snapshot multiple shares
*# Create a Consistency Group
*# Create share(s) with a CG specified
*# Create a CGsnapshot

* Create a CG
*# POST to /consistency-group
*#* specifiy share_types or the default share type will be used

* Copy a consistency group
*# Create a CGsnapshot
*# Create a CG with the CGSnapshot id

* Delete a CG
*# Delete all CG snapshots of the CG
*# Delete all shares in the CG
*# Delete the CG (Only empty CGs can be deleted)

* List shares in a given CG
*# GET /shares?consistencygroup_id=<cg_id>

* List the shares that were captured in a given CGSnapshot
*# GET /cgsnapshots/<id>/members

== System Workflows ==
* Creating a CG
*# Create the CG in the DB
*# Cast the consistency group creation to the scheduler
*## Choose a host for the CG
*# Cast the creation to the share service
*## Get the CG info from DB
*## Call the driver
*## Update database with info returned from driver (ex: CG status)

* Creating a CG from a cgsnapshot
*# Create a new CG in the DB
*# Create new shares from CG snapshot members in the DB
*# Schedule the CG create to the scheduler with the orig CG host/pool+share_type (depending on capability)
*# Call the driver with all info about source shares, new shares, cg, and cgsnapshot info.
*# Update CGsnapshot status and all shares status' to available

* Creating a cg snapshot
*# Create cgsnapshot entity in DB
*# For all shares in CG, create cgsnapshot_member in DB
*# cast create_cgsnapshot to the driver

* Adding a share to a CG
** On share creation:
**# Cast to scheduler with the host of the CG
**#* if perpool capable: Goes to the same pool as CG
**#* if per backend: goes to a pool on the same host that matches a CG share type

* Deleting a share (in a CG)
** Not allowed if the CG has snapshots of the share ( check if the share_id is in the CGSnapshotMembers tables)

== Scheduler ==
manila/scheduler/filter_scheduler.py should now look for consistency_group_support as a capability.

consistency_group_support
default: None 
Values:
* None - No support for CGs
* host - shares in a CG must be on pool(s) on the same host that also match the CG share type
* pool - shares in a CG must live in the same pool as the CG

==DB==
* New Tables:
class ConsistencyGroup(BASE, ManilaBase):
"""Represents a consistencygroup."""
__tablename__ = 'consistency_groups'
id = Column(String(36), primary_key=True)

user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)
deleted = Column(String(36), default='False')

host = Column(String(255))
name = Column(String(255))
description = Column(String(255))
status = Column(String(255))
source_cgsnapshot_id = Column(String(36))
share_network_id = Column(String(36), ForeignKey('share_networks.id'),
nullable=True)

class ConsistencyGroupShareTypeMapping(BASE, ManilaBase):
"""Represents the share types in a consistency group"""
__tablename__ = 'consistency_group_share_type_mappings'
id = Column(String(36), primary_key=True)
deleted = Column(String(36), default='False')
consistency_group_id = Column(String(36),
ForeignKey('consistency_groups.id'),
nullable=False)
share_type_id = Column(String(36),
ForeignKey('share_types.id'),
nullable=False)
consistency_group = orm.relationship(
ConsistencyGroup,
backref="share_types",
foreign_keys=consistency_group_id,
primaryjoin=('and_('
'ConsistencyGroupShareTypeMapping.consistency_group_id '
'== ConsistencyGroup.id,'
'ConsistencyGroupShareTypeMapping.deleted == "False")')
)

class CGSnapshot(BASE, ManilaBase):
"""Represents a cgsnapshot."""
__tablename__ = 'cgsnapshots'
id = Column(String(36), primary_key=True)

consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'))
user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)
deleted = Column(String(36), default='False')

name = Column(String(255))
description = Column(String(255))
status = Column(String(255))

consistency_group = orm.relationship(
ConsistencyGroup,
backref="cgsnapshots",
foreign_keys=consistency_group_id,
primaryjoin=('and_('
'CGSnapshot.consistency_group_id == ConsistencyGroup.id,'
'CGSnapshot.deleted == "False")')
)

class CGSnapshotMembers(BASE, ManilaBase):
__tablename__ = 'cgsnapshot_members'
id = Column(String(36), primary_key=True)
cgsnapshot_id = Column(String(36), ForeignKey('cgsnapshots.id'))
share_id = Column(String(36), ForeignKey('shares.id'))
size = Column(Integer)
status = Column(String(255))
share_proto = Column(String(255))
share_type_id = Column(String(36), ForeignKey('share_types.id'),
nullable=True)
user_id = Column(String(255))
project_id = Column(String(255))
deleted = Column(String(36), default='False')

cgsnapshot = relationship(
ConsistencyGroup,
backref="cgsnapshot_members",
foreign_keys=cgsnapshot_id,
primaryjoin='CGSnapshotMembers.cgsnapshot_id == CGSnapshot.id')

share = orm.relationship(Share, backref="cgsnapshot_members",
foreign_keys=share_id,
primaryjoin='and_('
'CGSnapshotMember.share_id == Share.id,'
'CGSnapshotMember.deleted == "False")')

* New fields in Share
consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'), nullable=True)
consistency_group = relationship(
ConsistencyGroup,
backref="shares",
foreign_keys=consistency_group_id,
primaryjoin='Share.consistency_group_id == ConsistencyGroup.id')

source_cgsnapshot_member_id = Column(String(36), nullable=True)

==API==
Schemas: https://wiki.openstack.org/wiki/Manila/design/manila-liberty-consistency-groups/api-schema

'''GET''' /consistency-groups/
'''GET''' /consistency-groups/detail
'''POST''' /consistency-groups
'''GET''' /consistency-groups/<id>
'''DELETE''' consistency-groups/<id>
'''PUT''' /consistency-groups/<id>

'''GET''' /cgsnapshots/
'''GET''' /cgsnapshots/detail
'''POST''' /cgsnapshots
'''PUT''' /cgsnapshots/<id>
'''GET''' /cgsnapshots/<id>
'''GET''' /cgsnapshots/<id>/members
'''DELETE''' /cgsnapshots/<id>

Admin Actions
'''POST''' /consistency-groups/<id>/action # os-reset-status and os-force-delete
Body:
{"os-reset_status": { "status": "available"}}
'''POST''' /cgsnapshots/<id>/action # os-reset-status and os-force-delete
Body:
{"os-force_delete": null}

Updates to Share resource
Adds consistency_group_id to POST/shares
Adds source_cgsnapshot_member_id
Add consistency_group_id query filter to /shares

==Policies==
policy.json - All consistency group policies should default to the default policy
"consistency_group:create" : "",
"consistency_group:delete": "",
"consistency_group:update": "",
"consistency_group:get": "",
"consistency_group:get_all": "",

"consistency_group:create_cgsnapshot" : "",
"consistency_group:delete_cgsnapshot": "",
"consistency_group:get_cgsnapshot": "",
"consistency_group:get_all_cgsnapshots": "",

==Driver API==

update_share_stats should now return 'consistency_group_support' pool/host/None.

def create_consistency_group(self, context, cg_dict, share_server=None):
"""Create a consistency group.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_consistency_group_from_cgsnapshot(self, context, cg_dict,
cgsnapshot_dict, share_server=None):
"""Create a consistency group from a cgsnapshot.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'shares': [<models.Share>], # The new shares being created
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:param cgsnapshot_dict:
EXAMPLE:
{'status': 'available',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': '0',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'consistency_group_id': '4b04fdc3-00b9-4909-ba1a-06e9b3f88b67',
'cgsnapshot_members': [
{'status': 'available',
'share_type_id': '1a9ed31e-ee70-483d-93ba-89690e028d7f',
'share_id': 'e14b5174-e534-4f35-bc4f-fe81c1575d6f',
'user_id': 'a0314a441ca842019b0952224aa39192',
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share': <models.Share>,
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share_proto': 'NFS',
'project_id': '13c0be6290934bd98596cfa004650049',
'cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'deleted_at': None,
'id': '6813e06b-a8f5-4784-b17d-f3e91afa370e',
'size': 1
}
],
'deleted_at': None,
'id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.

share_update_list - a list of dictionaries containing dicts for
every share created in the CG. Any share dicts should at a minimum
contain the 'id' key and 'export_locations'. Export locations
should be in the same format as returned by a share_create. This
list may be empty or None.
EXAMPLE:
[
{'id': 'uuid', 'export_locations': ['export_path']}
]
"""
raise NotImplementedError()

def delete_consistency_group(self, context, cg_dict, share_server=None):
"""Delete a consistency group

:param context:
:param cg_dict:
:return: cg_model_update
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_cgsnapshot(self, context, snap_dict, share_server=None):
"""Create a consistency group snapshot.

:param context:
:param snap_dict:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.

member_update_list - a list of dictionaries containing for every
member of the cgsnapshot. Each dict should contains values to be
updated for teh CGSnapshotMember in the database. This list may be
empty or None.
"""
raise NotImplementedError()

def delete_cgsnapshot(self, context, snap_dict, share_server=None):
"""Delete a consistency group snapshot

:param context:
:param snap_dict:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.
"""
raise NotImplementedError()

== Manila Client ==
* Do we want to alias the commands so that there are Cinder consistent commands and the improved names? (I.E cg-create = consisgroup-create) Perhaps we alias the cinderclient commands to be more like this?
cg-snapshot-create Creates a cgsnapshot.
cg-snapshot-delete Removes one or more cgsnapshots.
cg-snapshot-list Lists all cgsnapshots.
cg-snapshot-show Shows cgsnapshot details.

cg-create Creates a consistency group (--cgsnapshot to create from existing cg snapshot)
cg-delete Removes one or more consistency groups.
cg-list Lists all consistencygroups.
cg-show Shows details of a consistency group.
cg-update Updates a consistencygroup.

create --consistency-group Creates a share and puts it into consistency group

= Outstanding Questions / FAQ =
# How do we avoid confusion if no backends in a deployment support CGs?
#* It appears that Cinder has the policies default to 'nobody' but this seems odd, is there a better way?
#** (ameade) I can't think of a way to have devstack set the policies automatically either.
#** (ameade) we have chosen to go with an os-consistency-groups extension for now
# Do we need a way to fake CGs with the generic driver?
#* <strike> NO, but it will not be a core feature if it cannot be supported in the generic driver.</strike>
#* Ben says that we should fake it in the generic driver so it can be tested by the gate.
# Do we need a safeguard for deleting shares that are in a CG? (I.E. protected shares?) Are shares in a CG more valuable than normal shares?
# Should we overload the Snapshots table to include snapshots in a CGSnapshot to avoid adding a new db table? Or continue with having a new table?
# Why not have the cg delete all of the shares in it when deleting the cg?
#* This could cause odd state to occur if the backend fails when deleting shares and it will not be obvious where it failed. This is also extra complexity in the implementation that could instead be hidden by a client.

= Notes on Cinder Impl =

* A consistency group may be specified on volume create
* When a consistency group is created it is scheduled to a pool and all volume created in the CG will go to that pol

Cinder cli:
cgsnapshot-create Creates a cgsnapshot and snapshots of every volume in the CG that show up in snapshot-list.
cgsnapshot-delete Removes one or more cgsnapshots.
cgsnapshot-list Lists all cgsnapshots.
cgsnapshot-show Shows cgsnapshot details. <-- /cgsnapshots/detail?all_tenants=1&name=blah
consisgroup-create Creates a consistency group.
consisgroup-create-from-src Creates a consistency group from a cgsnapshot filled with volumes from the snapshots in the cg.
consisgroup-delete Removes one or more consistency groups.
consisgroup-list Lists all consistencygroups.
consisgroup-show Shows details of a consistency group. <-- /consistencygroups/detail?all_tenants=1&name=blah
consisgroup-update Updates a consistencygroup.

create --consisgroup-id

Cinder API:
'''GET''' /consistencygroups/detail
'''POST''' /consistencygroups
{'consistencygroup':{
'name': <string>,
'description': <string>,
'volume_types': <string>,
'availability_zone': <string>,
}}
'''POST''' /consistencygroups/create_from_src
{'consistencygroup-from-src':{
'name': <string>,
'description': <string>,
'cgsnapshot_id': <string>,
}}
'''GET''' /consistencygroups/<id>
'''POST''' consistencygroups/<id>/delete
{"consistencygroup": {"force": false}}
'''PUT''' /consistencygroups/<id>
{'consistencygroup':{
'name': <string>,
'description': <string>,
'add_volumes': <string>,
'remove_volumes': <string>,
}}

'''GET''' /cgsnapshots/detail
'''GET''' /cgsnapshots/<id>
'''POST''' /cgsnapshots
{'cgsnapshot':{
'consistencygroup_id': <string>,
}}
'''DELETE''' /cgsnapshots/<id>

Adds consistencygroup_id to POST/volumes

Xings commit message:
1) Create a CG, specifying all volume types that can be supported by this
CG. The scheduler chooses a backend that supports all specified volume types.
The CG will be empty when it is first created. Backend needs to report
consistencygroup_support = True. Volume type can have the following in
extra specs: {'capabilities:consistencygroup_support': '<is> True'}.
If consistencygroup_support is not in volume type extra specs, it will be
added to filter_properties by the scheduler to make sure that the scheduler
will select the backend which reports consistency group support capability.

Create CG CLI:
cinder consisgroup-create --volume-type type1,type2 mycg1

This will add a CG entry in the new consistencygroups table.

2) After the CG is created, create a new volume and add to the CG.
Repeat until all volumes are created for the CG.

Create volume CLI (with CG):
cinder create --volume-type type1 --consisgroup-id <CG uuid> 10

This will add a consistencygroup_id foreign key in the new volume
entry in the db.

3) Create a snapshot of the CG (cgsnapshot).

Create cgsnapshot CLI:
cinder cgsnapshot-create <CG uuid>

This will add a cgsnapshot entry in the new cgsnapshots table, create
snapshot for each volume in the CG, and add a cgsnapshot_id foreign key
in each newly created snapshot entry in the db.

===Questions===
# Why does policy.json default to 'nobody' being able to perform CG commands?
# Why are volume types specified when creating a CG?
#* This is because all volumes within a CG need to be on the same backend in order for the backend to honor the CG. This is important to note in the docs so that conflicting volume-types are not specified.
# What occurs when a volume on a different backend is added to a CG via the API?
#* Currently the API ensures that the volume volume_type matches a supported volume_type of the CG and that the CG and volume are on the same host (but can be on different pools).
# Is a volume created with a CG going to end up on the same host?
#* Yes, if a volume has a CG it will be cast to the same pool as the CG.
#* What if the pool is full but there are other available pools on the same backend? Should this fail?
# What happens if a driver does not support create_from_src?
#* The CG goes to ERROR a and volumes are created from the snapshot and also in ERROR status
#* A volume in a consistency group cannot be deleted and a CG with a volume cannot be deleted, therefore a volume must be removed from the CG before it can be deleted. But what if you don't support modifying a CG?
# Why do we need a force delete for CGs?
# What do I do if I do not want to see Snapshots in a CG when doing snapshot-list?
#* I suppose API filtering could allow for someone to not see all of the snapshots if they do not want to.

=== Cinder Bugs to file===
# What happens when you force delete a CG?
#* The volumes remain but still think they are in a CG, which means you still cannot delete the volume
# consisgroup-create-from-src says cgsnapshot is and optional param in cinderclient
# Creating a volume in a CG may not end up in the cg. The only check that is performed is that the volume-type of the volume is supported by the CG but volume-types are not 1-to-1 with backends. There could be backend A and B that both match with volume type 1. A cg is created with volume type 1 and ends up on backend A. Then a volume is created with the CG specified but ends up on backend B

=== Main impl ===
299b99437c2de84222fd585f06c2d05ca213225b ConsistencyGroup: Return 400 instead of 500 for invalid body
adb4c80be82caacad83f1366a4b34e5653fd5dab Create Consistency Group from CG Snapshot API
1a62a6e60fda73bf31256fbf684dc03bb6cf0038 Modify Consistency Group API
fc7a4384be2ccf8ff24f6c0f72c681ad9133801a Add tests for consistency groups DB migration
9082273305b0b9c117eb677a0e34c2ffde4e66f0 Volume types need to be specified when creating CG
764c5ec749821d36bb0215dc6002d3caea95d3b1 Delete consistency group failed
cf961f83ac323dfad1fa5e227d1e502a17529ecc Consistency Groups

=== Ex driver impl ===
8d5b795b37ed021c2639066689645f8aa0b1012f PureISCSIDriver consistency group updates.
39c1a8e08dc854e22ada315a46c20c99df2facf8 Add support to PureISCSIDriver for Consistency Groups
92a817708a3938b1b734d2caaa206b310996d8d0 EMC VMAX driver Kilo update

ThirdPartySystems/NetApp FC CI

2015-08-14T16:57:21Z

Alex Meade:

Manila/design/manila-liberty-consistency-groups

2015-08-13T19:06:52Z

Alex Meade: /* API */

= Consistency Groups =
This document describes the design choices and steps for the consistency group feature in core Manila.

Consistency groups are a mechanism to allow multiple filesystem shares to be guaranteed snapshots are able to be created at the exact same point in time. For example, a database may have its tables, logs, and configuration on separate volumes. If we wanted to restore the database from a previous point in time, it would only make sense to restore the logs, tables, and configuration together from the exact same point in time.

= Manila Core Changes =
The core manila changes are very similar to what has been done in Cinder. At a high level, changes in the DB, share service, drivers, scheduler service and api service will need to be changed.

Phase 1:
* Create/Delete CGs
* Create a share within a CG
* Delete a share that is in a CG
* Snapshot entire CGs
* Create CG from a CGSnapshot

Potential future features:
* Add/remove shares in a CG
* CG migration
* Replication of a CG

==Manila vs Cinder==
* Different CLI syntax
* Restful api tweaks
** /os-consistency-groups and /os-cgsnapshots are implemented as extensions and therefore entirely optional APIs
** Change /create_from_src to just be a cgsnapshot_id in the body on POST /os-consistency-groups
** Deletes to use the DELETE HTTP verb, os-force_delete is supported via the admin actions extension (/action)
* At the Driver API, cgsnapshot objects and cg objects (with all the information about what volumes are in the CG) need to be passed to the driver instead of just the cg/cgsnapshot name. This is important for backends that aren't managing a CG construct such as NetApp cDOT as they need to do things such as loop through the snapshots in a cg to delete them.
* The default policy in policy.json for consistency group operations is the default policy versus in Cinder is it 'nobody'
* Unlike Cinder, snapshots in a CGsnapshot are not the same as a normal snapshot. A CGsnapshot is treated as a single unit instead of a collection of snapshots.
* consistency_group_support scheduler capability has multiple values instead of boolean. This is to accommodate backends that require all volumes in a CG to be on the same pool vs. just the same host. It also allows for adding additional values in the future if a backend can support some other paradigm like a CG spanning multiple backends.

== User Workflows ==
* Snapshot multiple shares
*# Create a Consistency Group
*# Create share(s) with a CG specified
*# Create a CGsnapshot

* Create a CG
*# POST to /os-consistency-group
*#* specifiy share_types or the default share type will be used

* Copy a consistency group
*# Create a CGsnapshot
*# Create a CG with the CGSnapshot id

* Delete a CG
*# Delete all CG snapshots of the CG
*# Delete all shares in the CG
*# Delete the CG (Only empty CGs can be deleted)

* List shares in a given CG
*# GET /shares?consistencygroup_id=<cg_id>

* List the shares that were captured in a given CGSnapshot
*# GET /cgsnapshots/<id>/members

== System Workflows ==
* Creating a CG
*# Create the CG in the DB
*# Cast the consistency group creation to the scheduler
*## Choose a host for the CG
*# Cast the creation to the share service
*## Get the CG info from DB
*## Call the driver
*## Update database with info returned from driver (ex: CG status)

* Creating a CG from a cgsnapshot
*# Create a new CG in the DB
*# Create new shares from CG snapshot members in the DB
*# Schedule the CG create to the scheduler with the orig CG host/pool+share_type (depending on capability)
*# Call the driver with all info about source shares, new shares, cg, and cgsnapshot info.
*# Update CGsnapshot status and all shares status' to available

* Creating a cg snapshot
*# Create cgsnapshot entity in DB
*# For all shares in CG, create cgsnapshot_member in DB
*# cast create_cgsnapshot to the driver

* Adding a share to a CG
** On share creation:
**# Cast to scheduler with the host of the CG
**#* if perpool capable: Goes to the same pool as CG
**#* if per backend: goes to a pool on the same host that matches a CG share type

* Deleting a share (in a CG)
** Not allowed if the CG has snapshots of the share ( check if the share_id is in the CGSnapshotMembers tables)

== Scheduler ==
manila/scheduler/filter_scheduler.py should now look for consistency_group_support as a capability.

consistency_group_support
default: None 
Values:
* None - No support for CGs
* host - shares in a CG must be on pool(s) on the same host that also match the CG share type
* pool - shares in a CG must live in the same pool as the CG

==DB==
* New Tables:
class ConsistencyGroup(BASE, ManilaBase):
"""Represents a consistencygroup."""
__tablename__ = 'consistency_groups'
id = Column(String(36), primary_key=True)

user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)
deleted = Column(String(36), default='False')

host = Column(String(255))
name = Column(String(255))
description = Column(String(255))
status = Column(String(255))
source_cgsnapshot_id = Column(String(36))
share_network_id = Column(String(36), ForeignKey('share_networks.id'),
nullable=True)

class ConsistencyGroupShareTypeMapping(BASE, ManilaBase):
"""Represents the share types in a consistency group"""
__tablename__ = 'consistency_group_share_type_mappings'
id = Column(String(36), primary_key=True)
deleted = Column(String(36), default='False')
consistency_group_id = Column(String(36),
ForeignKey('consistency_groups.id'),
nullable=False)
share_type_id = Column(String(36),
ForeignKey('share_types.id'),
nullable=False)
consistency_group = orm.relationship(
ConsistencyGroup,
backref="share_types",
foreign_keys=consistency_group_id,
primaryjoin=('and_('
'ConsistencyGroupShareTypeMapping.consistency_group_id '
'== ConsistencyGroup.id,'
'ConsistencyGroupShareTypeMapping.deleted == "False")')
)

class CGSnapshot(BASE, ManilaBase):
"""Represents a cgsnapshot."""
__tablename__ = 'cgsnapshots'
id = Column(String(36), primary_key=True)

consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'))
user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)
deleted = Column(String(36), default='False')

name = Column(String(255))
description = Column(String(255))
status = Column(String(255))

consistency_group = orm.relationship(
ConsistencyGroup,
backref="cgsnapshots",
foreign_keys=consistency_group_id,
primaryjoin=('and_('
'CGSnapshot.consistency_group_id == ConsistencyGroup.id,'
'CGSnapshot.deleted == "False")')
)

class CGSnapshotMembers(BASE, ManilaBase):
__tablename__ = 'cgsnapshot_members'
id = Column(String(36), primary_key=True)
cgsnapshot_id = Column(String(36), ForeignKey('cgsnapshots.id'))
share_id = Column(String(36), ForeignKey('shares.id'))
size = Column(Integer)
status = Column(String(255))
share_proto = Column(String(255))
share_type_id = Column(String(36), ForeignKey('share_types.id'),
nullable=True)
user_id = Column(String(255))
project_id = Column(String(255))
deleted = Column(String(36), default='False')

cgsnapshot = relationship(
ConsistencyGroup,
backref="cgsnapshot_members",
foreign_keys=cgsnapshot_id,
primaryjoin='CGSnapshotMembers.cgsnapshot_id == CGSnapshot.id')

share = orm.relationship(Share, backref="cgsnapshot_members",
foreign_keys=share_id,
primaryjoin='and_('
'CGSnapshotMember.share_id == Share.id,'
'CGSnapshotMember.deleted == "False")')

* New fields in Share
consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'), nullable=True)
consistency_group = relationship(
ConsistencyGroup,
backref="shares",
foreign_keys=consistency_group_id,
primaryjoin='Share.consistency_group_id == ConsistencyGroup.id')

source_cgsnapshot_member_id = Column(String(36), nullable=True)

==API==
Schemas: https://wiki.openstack.org/wiki/Manila/design/manila-liberty-consistency-groups/api-schema

'''GET''' /os-consistency-groups/
'''GET''' /os-consistency-groups/detail
'''POST''' /os-consistency-groups
'''GET''' /os-consistency-groups/<id>
'''DELETE''' os-consistency-groups/<id>
'''PUT''' /os-consistency-groups/<id>

'''GET''' /os-cgsnapshots/
'''GET''' /os-cgsnapshots/detail
'''POST''' /os-cgsnapshots
'''PUT''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>/members
'''DELETE''' /os-cgsnapshots/<id>

Admin Actions
'''POST''' /os-consistency-groups/<id>/action # os-reset-status and os-force-delete
Body:
{"os-reset_status": { "status": "available"}}
'''POST''' /os-cgsnapshots/<id>/action # os-reset-status and os-force-delete
Body:
{"os-force_delete": null}

Updates to Share resource
Adds consistency_group_id to POST/shares
Adds source_cgsnapshot_member_id
Add consistency_group_id query filter to /shares

==Policies==
policy.json - All consistency group policies should default to the default policy
"consistency_group:create" : "",
"consistency_group:delete": "",
"consistency_group:update": "",
"consistency_group:get": "",
"consistency_group:get_all": "",

"consistency_group:create_cgsnapshot" : "",
"consistency_group:delete_cgsnapshot": "",
"consistency_group:get_cgsnapshot": "",
"consistency_group:get_all_cgsnapshots": "",

==Driver API==

update_share_stats should now return 'consistency_group_support' pool/host/None.

def create_consistency_group(self, context, cg_dict, share_server=None):
"""Create a consistency group.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_consistency_group_from_cgsnapshot(self, context, cg_dict,
cgsnapshot_dict, share_server=None):
"""Create a consistency group from a cgsnapshot.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'shares': [<models.Share>], # The new shares being created
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:param cgsnapshot_dict:
EXAMPLE:
{'status': 'available',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': '0',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'consistency_group_id': '4b04fdc3-00b9-4909-ba1a-06e9b3f88b67',
'cgsnapshot_members': [
{'status': 'available',
'share_type_id': '1a9ed31e-ee70-483d-93ba-89690e028d7f',
'share_id': 'e14b5174-e534-4f35-bc4f-fe81c1575d6f',
'user_id': 'a0314a441ca842019b0952224aa39192',
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share': <models.Share>,
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share_proto': 'NFS',
'project_id': '13c0be6290934bd98596cfa004650049',
'cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'deleted_at': None,
'id': '6813e06b-a8f5-4784-b17d-f3e91afa370e',
'size': 1
}
],
'deleted_at': None,
'id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.

share_update_list - a list of dictionaries containing dicts for
every share created in the CG. Any share dicts should at a minimum
contain the 'id' key and 'export_locations'. Export locations
should be in the same format as returned by a share_create. This
list may be empty or None.
EXAMPLE:
[
{'id': 'uuid', 'export_locations': ['export_path']}
]
"""
raise NotImplementedError()

def delete_consistency_group(self, context, cg_dict, share_server=None):
"""Delete a consistency group

:param context:
:param cg_dict:
:return: cg_model_update
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_cgsnapshot(self, context, snap_dict, share_server=None):
"""Create a consistency group snapshot.

:param context:
:param snap_dict:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.

member_update_list - a list of dictionaries containing for every
member of the cgsnapshot. Each dict should contains values to be
updated for teh CGSnapshotMember in the database. This list may be
empty or None.
"""
raise NotImplementedError()

def delete_cgsnapshot(self, context, snap_dict, share_server=None):
"""Delete a consistency group snapshot

:param context:
:param snap_dict:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.
"""
raise NotImplementedError()

== Manila Client ==
* Do we want to alias the commands so that there are Cinder consistent commands and the improved names? (I.E cg-create = consisgroup-create) Perhaps we alias the cinderclient commands to be more like this?
cg-snapshot-create Creates a cgsnapshot.
cg-snapshot-delete Removes one or more cgsnapshots.
cg-snapshot-list Lists all cgsnapshots.
cg-snapshot-show Shows cgsnapshot details.

cg-create Creates a consistency group (--cgsnapshot to create from existing cg snapshot)
cg-delete Removes one or more consistency groups.
cg-list Lists all consistencygroups.
cg-show Shows details of a consistency group.
cg-update Updates a consistencygroup.

create --consistency-group Creates a share and puts it into consistency group

= Outstanding Questions / FAQ =
# How do we avoid confusion if no backends in a deployment support CGs?
#* It appears that Cinder has the policies default to 'nobody' but this seems odd, is there a better way?
#** (ameade) I can't think of a way to have devstack set the policies automatically either.
#** (ameade) we have chosen to go with an os-consistency-groups extension for now
# Do we need a way to fake CGs with the generic driver?
#* <strike> NO, but it will not be a core feature if it cannot be supported in the generic driver.</strike>
#* Ben says that we should fake it in the generic driver so it can be tested by the gate.
# Do we need a safeguard for deleting shares that are in a CG? (I.E. protected shares?) Are shares in a CG more valuable than normal shares?
# Should we overload the Snapshots table to include snapshots in a CGSnapshot to avoid adding a new db table? Or continue with having a new table?
# Why not have the cg delete all of the shares in it when deleting the cg?
#* This could cause odd state to occur if the backend fails when deleting shares and it will not be obvious where it failed. This is also extra complexity in the implementation that could instead be hidden by a client.

= Notes on Cinder Impl =

* A consistency group may be specified on volume create
* When a consistency group is created it is scheduled to a pool and all volume created in the CG will go to that pol

Cinder cli:
cgsnapshot-create Creates a cgsnapshot and snapshots of every volume in the CG that show up in snapshot-list.
cgsnapshot-delete Removes one or more cgsnapshots.
cgsnapshot-list Lists all cgsnapshots.
cgsnapshot-show Shows cgsnapshot details. <-- /cgsnapshots/detail?all_tenants=1&name=blah
consisgroup-create Creates a consistency group.
consisgroup-create-from-src Creates a consistency group from a cgsnapshot filled with volumes from the snapshots in the cg.
consisgroup-delete Removes one or more consistency groups.
consisgroup-list Lists all consistencygroups.
consisgroup-show Shows details of a consistency group. <-- /consistencygroups/detail?all_tenants=1&name=blah
consisgroup-update Updates a consistencygroup.

create --consisgroup-id

Cinder API:
'''GET''' /consistencygroups/detail
'''POST''' /consistencygroups
{'consistencygroup':{
'name': <string>,
'description': <string>,
'volume_types': <string>,
'availability_zone': <string>,
}}
'''POST''' /consistencygroups/create_from_src
{'consistencygroup-from-src':{
'name': <string>,
'description': <string>,
'cgsnapshot_id': <string>,
}}
'''GET''' /consistencygroups/<id>
'''POST''' consistencygroups/<id>/delete
{"consistencygroup": {"force": false}}
'''PUT''' /consistencygroups/<id>
{'consistencygroup':{
'name': <string>,
'description': <string>,
'add_volumes': <string>,
'remove_volumes': <string>,
}}

'''GET''' /cgsnapshots/detail
'''GET''' /cgsnapshots/<id>
'''POST''' /cgsnapshots
{'cgsnapshot':{
'consistencygroup_id': <string>,
}}
'''DELETE''' /cgsnapshots/<id>

Adds consistencygroup_id to POST/volumes

Xings commit message:
1) Create a CG, specifying all volume types that can be supported by this
CG. The scheduler chooses a backend that supports all specified volume types.
The CG will be empty when it is first created. Backend needs to report
consistencygroup_support = True. Volume type can have the following in
extra specs: {'capabilities:consistencygroup_support': '<is> True'}.
If consistencygroup_support is not in volume type extra specs, it will be
added to filter_properties by the scheduler to make sure that the scheduler
will select the backend which reports consistency group support capability.

Create CG CLI:
cinder consisgroup-create --volume-type type1,type2 mycg1

This will add a CG entry in the new consistencygroups table.

2) After the CG is created, create a new volume and add to the CG.
Repeat until all volumes are created for the CG.

Create volume CLI (with CG):
cinder create --volume-type type1 --consisgroup-id <CG uuid> 10

This will add a consistencygroup_id foreign key in the new volume
entry in the db.

3) Create a snapshot of the CG (cgsnapshot).

Create cgsnapshot CLI:
cinder cgsnapshot-create <CG uuid>

This will add a cgsnapshot entry in the new cgsnapshots table, create
snapshot for each volume in the CG, and add a cgsnapshot_id foreign key
in each newly created snapshot entry in the db.

===Questions===
# Why does policy.json default to 'nobody' being able to perform CG commands?
# Why are volume types specified when creating a CG?
#* This is because all volumes within a CG need to be on the same backend in order for the backend to honor the CG. This is important to note in the docs so that conflicting volume-types are not specified.
# What occurs when a volume on a different backend is added to a CG via the API?
#* Currently the API ensures that the volume volume_type matches a supported volume_type of the CG and that the CG and volume are on the same host (but can be on different pools).
# Is a volume created with a CG going to end up on the same host?
#* Yes, if a volume has a CG it will be cast to the same pool as the CG.
#* What if the pool is full but there are other available pools on the same backend? Should this fail?
# What happens if a driver does not support create_from_src?
#* The CG goes to ERROR a and volumes are created from the snapshot and also in ERROR status
#* A volume in a consistency group cannot be deleted and a CG with a volume cannot be deleted, therefore a volume must be removed from the CG before it can be deleted. But what if you don't support modifying a CG?
# Why do we need a force delete for CGs?
# What do I do if I do not want to see Snapshots in a CG when doing snapshot-list?
#* I suppose API filtering could allow for someone to not see all of the snapshots if they do not want to.

=== Cinder Bugs to file===
# What happens when you force delete a CG?
#* The volumes remain but still think they are in a CG, which means you still cannot delete the volume
# consisgroup-create-from-src says cgsnapshot is and optional param in cinderclient
# Creating a volume in a CG may not end up in the cg. The only check that is performed is that the volume-type of the volume is supported by the CG but volume-types are not 1-to-1 with backends. There could be backend A and B that both match with volume type 1. A cg is created with volume type 1 and ends up on backend A. Then a volume is created with the CG specified but ends up on backend B

=== Main impl ===
299b99437c2de84222fd585f06c2d05ca213225b ConsistencyGroup: Return 400 instead of 500 for invalid body
adb4c80be82caacad83f1366a4b34e5653fd5dab Create Consistency Group from CG Snapshot API
1a62a6e60fda73bf31256fbf684dc03bb6cf0038 Modify Consistency Group API
fc7a4384be2ccf8ff24f6c0f72c681ad9133801a Add tests for consistency groups DB migration
9082273305b0b9c117eb677a0e34c2ffde4e66f0 Volume types need to be specified when creating CG
764c5ec749821d36bb0215dc6002d3caea95d3b1 Delete consistency group failed
cf961f83ac323dfad1fa5e227d1e502a17529ecc Consistency Groups

=== Ex driver impl ===
8d5b795b37ed021c2639066689645f8aa0b1012f PureISCSIDriver consistency group updates.
39c1a8e08dc854e22ada315a46c20c99df2facf8 Add support to PureISCSIDriver for Consistency Groups
92a817708a3938b1b734d2caaa206b310996d8d0 EMC VMAX driver Kilo update

Manila/design/manila-liberty-consistency-groups

2015-08-13T14:22:14Z

Alex Meade: /* Driver API */

= Consistency Groups =
This document describes the design choices and steps for the consistency group feature in core Manila.

Consistency groups are a mechanism to allow multiple filesystem shares to be guaranteed snapshots are able to be created at the exact same point in time. For example, a database may have its tables, logs, and configuration on separate volumes. If we wanted to restore the database from a previous point in time, it would only make sense to restore the logs, tables, and configuration together from the exact same point in time.

= Manila Core Changes =
The core manila changes are very similar to what has been done in Cinder. At a high level, changes in the DB, share service, drivers, scheduler service and api service will need to be changed.

Phase 1:
* Create/Delete CGs
* Create a share within a CG
* Delete a share that is in a CG
* Snapshot entire CGs
* Create CG from a CGSnapshot

Potential future features:
* Add/remove shares in a CG
* CG migration
* Replication of a CG

==Manila vs Cinder==
* Different CLI syntax
* Restful api tweaks
** /os-consistency-groups and /os-cgsnapshots are implemented as extensions and therefore entirely optional APIs
** Change /create_from_src to just be a cgsnapshot_id in the body on POST /os-consistency-groups
** Deletes to use the DELETE HTTP verb, os-force_delete is supported via the admin actions extension (/action)
* At the Driver API, cgsnapshot objects and cg objects (with all the information about what volumes are in the CG) need to be passed to the driver instead of just the cg/cgsnapshot name. This is important for backends that aren't managing a CG construct such as NetApp cDOT as they need to do things such as loop through the snapshots in a cg to delete them.
* The default policy in policy.json for consistency group operations is the default policy versus in Cinder is it 'nobody'
* Unlike Cinder, snapshots in a CGsnapshot are not the same as a normal snapshot. A CGsnapshot is treated as a single unit instead of a collection of snapshots.
* consistency_group_support scheduler capability has multiple values instead of boolean. This is to accommodate backends that require all volumes in a CG to be on the same pool vs. just the same host. It also allows for adding additional values in the future if a backend can support some other paradigm like a CG spanning multiple backends.

== User Workflows ==
* Snapshot multiple shares
*# Create a Consistency Group
*# Create share(s) with a CG specified
*# Create a CGsnapshot

* Create a CG
*# POST to /os-consistency-group
*#* specifiy share_types or the default share type will be used

* Copy a consistency group
*# Create a CGsnapshot
*# Create a CG with the CGSnapshot id

* Delete a CG
*# Delete all CG snapshots of the CG
*# Delete all shares in the CG
*# Delete the CG (Only empty CGs can be deleted)

* List shares in a given CG
*# GET /shares?consistencygroup_id=<cg_id>

* List the shares that were captured in a given CGSnapshot
*# GET /cgsnapshots/<id>/members

== System Workflows ==
* Creating a CG
*# Create the CG in the DB
*# Cast the consistency group creation to the scheduler
*## Choose a host for the CG
*# Cast the creation to the share service
*## Get the CG info from DB
*## Call the driver
*## Update database with info returned from driver (ex: CG status)

* Creating a CG from a cgsnapshot
*# Create a new CG in the DB
*# Create new shares from CG snapshot members in the DB
*# Schedule the CG create to the scheduler with the orig CG host/pool+share_type (depending on capability)
*# Call the driver with all info about source shares, new shares, cg, and cgsnapshot info.
*# Update CGsnapshot status and all shares status' to available

* Creating a cg snapshot
*# Create cgsnapshot entity in DB
*# For all shares in CG, create cgsnapshot_member in DB
*# cast create_cgsnapshot to the driver

* Adding a share to a CG
** On share creation:
**# Cast to scheduler with the host of the CG
**#* if perpool capable: Goes to the same pool as CG
**#* if per backend: goes to a pool on the same host that matches a CG share type

* Deleting a share (in a CG)
** Not allowed if the CG has snapshots of the share ( check if the share_id is in the CGSnapshotMembers tables)

== Scheduler ==
manila/scheduler/filter_scheduler.py should now look for consistency_group_support as a capability.

consistency_group_support
default: None 
Values:
* None - No support for CGs
* host - shares in a CG must be on pool(s) on the same host that also match the CG share type
* pool - shares in a CG must live in the same pool as the CG

==DB==
* New Tables:
class ConsistencyGroup(BASE, ManilaBase):
"""Represents a consistencygroup."""
__tablename__ = 'consistency_groups'
id = Column(String(36), primary_key=True)

user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)
deleted = Column(String(36), default='False')

host = Column(String(255))
name = Column(String(255))
description = Column(String(255))
status = Column(String(255))
source_cgsnapshot_id = Column(String(36))
share_network_id = Column(String(36), ForeignKey('share_networks.id'),
nullable=True)

class ConsistencyGroupShareTypeMapping(BASE, ManilaBase):
"""Represents the share types in a consistency group"""
__tablename__ = 'consistency_group_share_type_mappings'
id = Column(String(36), primary_key=True)
deleted = Column(String(36), default='False')
consistency_group_id = Column(String(36),
ForeignKey('consistency_groups.id'),
nullable=False)
share_type_id = Column(String(36),
ForeignKey('share_types.id'),
nullable=False)
consistency_group = orm.relationship(
ConsistencyGroup,
backref="share_types",
foreign_keys=consistency_group_id,
primaryjoin=('and_('
'ConsistencyGroupShareTypeMapping.consistency_group_id '
'== ConsistencyGroup.id,'
'ConsistencyGroupShareTypeMapping.deleted == "False")')
)

class CGSnapshot(BASE, ManilaBase):
"""Represents a cgsnapshot."""
__tablename__ = 'cgsnapshots'
id = Column(String(36), primary_key=True)

consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'))
user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)
deleted = Column(String(36), default='False')

name = Column(String(255))
description = Column(String(255))
status = Column(String(255))

consistency_group = orm.relationship(
ConsistencyGroup,
backref="cgsnapshots",
foreign_keys=consistency_group_id,
primaryjoin=('and_('
'CGSnapshot.consistency_group_id == ConsistencyGroup.id,'
'CGSnapshot.deleted == "False")')
)

class CGSnapshotMembers(BASE, ManilaBase):
__tablename__ = 'cgsnapshot_members'
id = Column(String(36), primary_key=True)
cgsnapshot_id = Column(String(36), ForeignKey('cgsnapshots.id'))
share_id = Column(String(36), ForeignKey('shares.id'))
size = Column(Integer)
status = Column(String(255))
share_proto = Column(String(255))
share_type_id = Column(String(36), ForeignKey('share_types.id'),
nullable=True)
user_id = Column(String(255))
project_id = Column(String(255))
deleted = Column(String(36), default='False')

cgsnapshot = relationship(
ConsistencyGroup,
backref="cgsnapshot_members",
foreign_keys=cgsnapshot_id,
primaryjoin='CGSnapshotMembers.cgsnapshot_id == CGSnapshot.id')

share = orm.relationship(Share, backref="cgsnapshot_members",
foreign_keys=share_id,
primaryjoin='and_('
'CGSnapshotMember.share_id == Share.id,'
'CGSnapshotMember.deleted == "False")')

* New fields in Share
consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'), nullable=True)
consistency_group = relationship(
ConsistencyGroup,
backref="shares",
foreign_keys=consistency_group_id,
primaryjoin='Share.consistency_group_id == ConsistencyGroup.id')

source_cgsnapshot_member_id = Column(String(36), nullable=True)

==API==
Schemas: https://wiki.openstack.org/wiki/Manila/design/manila-liberty-consistency-groups/api-schema

'''GET''' /os-consistency-groups/
'''GET''' /os-consistency-groups/detail
'''POST''' /os-consistency-groups
'''GET''' /os-consistency-groups/<id>
'''DELETE''' os-consistency-groups/<id>
'''PUT''' /os-consistency-groups/<id>

'''GET''' /os-cgsnapshots/
'''GET''' /os-cgsnapshots/detail
'''POST''' /os-cgsnapshots
'''PUT''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>/members
'''DELETE''' /os-cgsnapshots/<id>

Admin Actions
'''POST''' /os-consistency-groups/<id>/action # os-reset-status and os-force-delete
'''POST''' /os-cgsnapshots/<id>/action # os-reset-status and os-force-delete

Updates to Share resource
Adds consistency_group_id to POST/shares
Adds source_cgsnapshot_member_id
Add consistency_group_id query filter to /shares

==Policies==
policy.json - All consistency group policies should default to the default policy
"consistency_group:create" : "",
"consistency_group:delete": "",
"consistency_group:update": "",
"consistency_group:get": "",
"consistency_group:get_all": "",

"consistency_group:create_cgsnapshot" : "",
"consistency_group:delete_cgsnapshot": "",
"consistency_group:get_cgsnapshot": "",
"consistency_group:get_all_cgsnapshots": "",

==Driver API==

update_share_stats should now return 'consistency_group_support' pool/host/None.

def create_consistency_group(self, context, cg_dict, share_server=None):
"""Create a consistency group.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_consistency_group_from_cgsnapshot(self, context, cg_dict,
cgsnapshot_dict, share_server=None):
"""Create a consistency group from a cgsnapshot.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'shares': [<models.Share>], # The new shares being created
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:param cgsnapshot_dict:
EXAMPLE:
{'status': 'available',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': '0',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'consistency_group_id': '4b04fdc3-00b9-4909-ba1a-06e9b3f88b67',
'cgsnapshot_members': [
{'status': 'available',
'share_type_id': '1a9ed31e-ee70-483d-93ba-89690e028d7f',
'share_id': 'e14b5174-e534-4f35-bc4f-fe81c1575d6f',
'user_id': 'a0314a441ca842019b0952224aa39192',
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share': <models.Share>,
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share_proto': 'NFS',
'project_id': '13c0be6290934bd98596cfa004650049',
'cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'deleted_at': None,
'id': '6813e06b-a8f5-4784-b17d-f3e91afa370e',
'size': 1
}
],
'deleted_at': None,
'id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.

share_update_list - a list of dictionaries containing dicts for
every share created in the CG. Any share dicts should at a minimum
contain the 'id' key and 'export_locations'. Export locations
should be in the same format as returned by a share_create. This
list may be empty or None.
EXAMPLE:
[
{'id': 'uuid', 'export_locations': ['export_path']}
]
"""
raise NotImplementedError()

def delete_consistency_group(self, context, cg_dict, share_server=None):
"""Delete a consistency group

:param context:
:param cg_dict:
:return: cg_model_update
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_cgsnapshot(self, context, snap_dict, share_server=None):
"""Create a consistency group snapshot.

:param context:
:param snap_dict:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.

member_update_list - a list of dictionaries containing for every
member of the cgsnapshot. Each dict should contains values to be
updated for teh CGSnapshotMember in the database. This list may be
empty or None.
"""
raise NotImplementedError()

def delete_cgsnapshot(self, context, snap_dict, share_server=None):
"""Delete a consistency group snapshot

:param context:
:param snap_dict:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.
"""
raise NotImplementedError()

== Manila Client ==
* Do we want to alias the commands so that there are Cinder consistent commands and the improved names? (I.E cg-create = consisgroup-create) Perhaps we alias the cinderclient commands to be more like this?
cg-snapshot-create Creates a cgsnapshot.
cg-snapshot-delete Removes one or more cgsnapshots.
cg-snapshot-list Lists all cgsnapshots.
cg-snapshot-show Shows cgsnapshot details.

cg-create Creates a consistency group (--cgsnapshot to create from existing cg snapshot)
cg-delete Removes one or more consistency groups.
cg-list Lists all consistencygroups.
cg-show Shows details of a consistency group.
cg-update Updates a consistencygroup.

create --consistency-group Creates a share and puts it into consistency group

= Outstanding Questions / FAQ =
# How do we avoid confusion if no backends in a deployment support CGs?
#* It appears that Cinder has the policies default to 'nobody' but this seems odd, is there a better way?
#** (ameade) I can't think of a way to have devstack set the policies automatically either.
#** (ameade) we have chosen to go with an os-consistency-groups extension for now
# Do we need a way to fake CGs with the generic driver?
#* <strike> NO, but it will not be a core feature if it cannot be supported in the generic driver.</strike>
#* Ben says that we should fake it in the generic driver so it can be tested by the gate.
# Do we need a safeguard for deleting shares that are in a CG? (I.E. protected shares?) Are shares in a CG more valuable than normal shares?
# Should we overload the Snapshots table to include snapshots in a CGSnapshot to avoid adding a new db table? Or continue with having a new table?
# Why not have the cg delete all of the shares in it when deleting the cg?
#* This could cause odd state to occur if the backend fails when deleting shares and it will not be obvious where it failed. This is also extra complexity in the implementation that could instead be hidden by a client.

= Notes on Cinder Impl =

* A consistency group may be specified on volume create
* When a consistency group is created it is scheduled to a pool and all volume created in the CG will go to that pol

Cinder cli:
cgsnapshot-create Creates a cgsnapshot and snapshots of every volume in the CG that show up in snapshot-list.
cgsnapshot-delete Removes one or more cgsnapshots.
cgsnapshot-list Lists all cgsnapshots.
cgsnapshot-show Shows cgsnapshot details. <-- /cgsnapshots/detail?all_tenants=1&name=blah
consisgroup-create Creates a consistency group.
consisgroup-create-from-src Creates a consistency group from a cgsnapshot filled with volumes from the snapshots in the cg.
consisgroup-delete Removes one or more consistency groups.
consisgroup-list Lists all consistencygroups.
consisgroup-show Shows details of a consistency group. <-- /consistencygroups/detail?all_tenants=1&name=blah
consisgroup-update Updates a consistencygroup.

create --consisgroup-id

Cinder API:
'''GET''' /consistencygroups/detail
'''POST''' /consistencygroups
{'consistencygroup':{
'name': <string>,
'description': <string>,
'volume_types': <string>,
'availability_zone': <string>,
}}
'''POST''' /consistencygroups/create_from_src
{'consistencygroup-from-src':{
'name': <string>,
'description': <string>,
'cgsnapshot_id': <string>,
}}
'''GET''' /consistencygroups/<id>
'''POST''' consistencygroups/<id>/delete
{"consistencygroup": {"force": false}}
'''PUT''' /consistencygroups/<id>
{'consistencygroup':{
'name': <string>,
'description': <string>,
'add_volumes': <string>,
'remove_volumes': <string>,
}}

'''GET''' /cgsnapshots/detail
'''GET''' /cgsnapshots/<id>
'''POST''' /cgsnapshots
{'cgsnapshot':{
'consistencygroup_id': <string>,
}}
'''DELETE''' /cgsnapshots/<id>

Adds consistencygroup_id to POST/volumes

Xings commit message:
1) Create a CG, specifying all volume types that can be supported by this
CG. The scheduler chooses a backend that supports all specified volume types.
The CG will be empty when it is first created. Backend needs to report
consistencygroup_support = True. Volume type can have the following in
extra specs: {'capabilities:consistencygroup_support': '<is> True'}.
If consistencygroup_support is not in volume type extra specs, it will be
added to filter_properties by the scheduler to make sure that the scheduler
will select the backend which reports consistency group support capability.

Create CG CLI:
cinder consisgroup-create --volume-type type1,type2 mycg1

This will add a CG entry in the new consistencygroups table.

2) After the CG is created, create a new volume and add to the CG.
Repeat until all volumes are created for the CG.

Create volume CLI (with CG):
cinder create --volume-type type1 --consisgroup-id <CG uuid> 10

This will add a consistencygroup_id foreign key in the new volume
entry in the db.

3) Create a snapshot of the CG (cgsnapshot).

Create cgsnapshot CLI:
cinder cgsnapshot-create <CG uuid>

This will add a cgsnapshot entry in the new cgsnapshots table, create
snapshot for each volume in the CG, and add a cgsnapshot_id foreign key
in each newly created snapshot entry in the db.

===Questions===
# Why does policy.json default to 'nobody' being able to perform CG commands?
# Why are volume types specified when creating a CG?
#* This is because all volumes within a CG need to be on the same backend in order for the backend to honor the CG. This is important to note in the docs so that conflicting volume-types are not specified.
# What occurs when a volume on a different backend is added to a CG via the API?
#* Currently the API ensures that the volume volume_type matches a supported volume_type of the CG and that the CG and volume are on the same host (but can be on different pools).
# Is a volume created with a CG going to end up on the same host?
#* Yes, if a volume has a CG it will be cast to the same pool as the CG.
#* What if the pool is full but there are other available pools on the same backend? Should this fail?
# What happens if a driver does not support create_from_src?
#* The CG goes to ERROR a and volumes are created from the snapshot and also in ERROR status
#* A volume in a consistency group cannot be deleted and a CG with a volume cannot be deleted, therefore a volume must be removed from the CG before it can be deleted. But what if you don't support modifying a CG?
# Why do we need a force delete for CGs?
# What do I do if I do not want to see Snapshots in a CG when doing snapshot-list?
#* I suppose API filtering could allow for someone to not see all of the snapshots if they do not want to.

=== Cinder Bugs to file===
# What happens when you force delete a CG?
#* The volumes remain but still think they are in a CG, which means you still cannot delete the volume
# consisgroup-create-from-src says cgsnapshot is and optional param in cinderclient
# Creating a volume in a CG may not end up in the cg. The only check that is performed is that the volume-type of the volume is supported by the CG but volume-types are not 1-to-1 with backends. There could be backend A and B that both match with volume type 1. A cg is created with volume type 1 and ends up on backend A. Then a volume is created with the CG specified but ends up on backend B

=== Main impl ===
299b99437c2de84222fd585f06c2d05ca213225b ConsistencyGroup: Return 400 instead of 500 for invalid body
adb4c80be82caacad83f1366a4b34e5653fd5dab Create Consistency Group from CG Snapshot API
1a62a6e60fda73bf31256fbf684dc03bb6cf0038 Modify Consistency Group API
fc7a4384be2ccf8ff24f6c0f72c681ad9133801a Add tests for consistency groups DB migration
9082273305b0b9c117eb677a0e34c2ffde4e66f0 Volume types need to be specified when creating CG
764c5ec749821d36bb0215dc6002d3caea95d3b1 Delete consistency group failed
cf961f83ac323dfad1fa5e227d1e502a17529ecc Consistency Groups

=== Ex driver impl ===
8d5b795b37ed021c2639066689645f8aa0b1012f PureISCSIDriver consistency group updates.
39c1a8e08dc854e22ada315a46c20c99df2facf8 Add support to PureISCSIDriver for Consistency Groups
92a817708a3938b1b734d2caaa206b310996d8d0 EMC VMAX driver Kilo update

Manila/design/manila-liberty-consistency-groups

2015-08-13T13:35:26Z

Alex Meade: /* DB */

= Consistency Groups =
This document describes the design choices and steps for the consistency group feature in core Manila.

Consistency groups are a mechanism to allow multiple filesystem shares to be guaranteed snapshots are able to be created at the exact same point in time. For example, a database may have its tables, logs, and configuration on separate volumes. If we wanted to restore the database from a previous point in time, it would only make sense to restore the logs, tables, and configuration together from the exact same point in time.

= Manila Core Changes =
The core manila changes are very similar to what has been done in Cinder. At a high level, changes in the DB, share service, drivers, scheduler service and api service will need to be changed.

Phase 1:
* Create/Delete CGs
* Create a share within a CG
* Delete a share that is in a CG
* Snapshot entire CGs
* Create CG from a CGSnapshot

Potential future features:
* Add/remove shares in a CG
* CG migration
* Replication of a CG

==Manila vs Cinder==
* Different CLI syntax
* Restful api tweaks
** /os-consistency-groups and /os-cgsnapshots are implemented as extensions and therefore entirely optional APIs
** Change /create_from_src to just be a cgsnapshot_id in the body on POST /os-consistency-groups
** Deletes to use the DELETE HTTP verb, os-force_delete is supported via the admin actions extension (/action)
* At the Driver API, cgsnapshot objects and cg objects (with all the information about what volumes are in the CG) need to be passed to the driver instead of just the cg/cgsnapshot name. This is important for backends that aren't managing a CG construct such as NetApp cDOT as they need to do things such as loop through the snapshots in a cg to delete them.
* The default policy in policy.json for consistency group operations is the default policy versus in Cinder is it 'nobody'
* Unlike Cinder, snapshots in a CGsnapshot are not the same as a normal snapshot. A CGsnapshot is treated as a single unit instead of a collection of snapshots.
* consistency_group_support scheduler capability has multiple values instead of boolean. This is to accommodate backends that require all volumes in a CG to be on the same pool vs. just the same host. It also allows for adding additional values in the future if a backend can support some other paradigm like a CG spanning multiple backends.

== User Workflows ==
* Snapshot multiple shares
*# Create a Consistency Group
*# Create share(s) with a CG specified
*# Create a CGsnapshot

* Create a CG
*# POST to /os-consistency-group
*#* specifiy share_types or the default share type will be used

* Copy a consistency group
*# Create a CGsnapshot
*# Create a CG with the CGSnapshot id

* Delete a CG
*# Delete all CG snapshots of the CG
*# Delete all shares in the CG
*# Delete the CG (Only empty CGs can be deleted)

* List shares in a given CG
*# GET /shares?consistencygroup_id=<cg_id>

* List the shares that were captured in a given CGSnapshot
*# GET /cgsnapshots/<id>/members

== System Workflows ==
* Creating a CG
*# Create the CG in the DB
*# Cast the consistency group creation to the scheduler
*## Choose a host for the CG
*# Cast the creation to the share service
*## Get the CG info from DB
*## Call the driver
*## Update database with info returned from driver (ex: CG status)

* Creating a CG from a cgsnapshot
*# Create a new CG in the DB
*# Create new shares from CG snapshot members in the DB
*# Schedule the CG create to the scheduler with the orig CG host/pool+share_type (depending on capability)
*# Call the driver with all info about source shares, new shares, cg, and cgsnapshot info.
*# Update CGsnapshot status and all shares status' to available

* Creating a cg snapshot
*# Create cgsnapshot entity in DB
*# For all shares in CG, create cgsnapshot_member in DB
*# cast create_cgsnapshot to the driver

* Adding a share to a CG
** On share creation:
**# Cast to scheduler with the host of the CG
**#* if perpool capable: Goes to the same pool as CG
**#* if per backend: goes to a pool on the same host that matches a CG share type

* Deleting a share (in a CG)
** Not allowed if the CG has snapshots of the share ( check if the share_id is in the CGSnapshotMembers tables)

== Scheduler ==
manila/scheduler/filter_scheduler.py should now look for consistency_group_support as a capability.

consistency_group_support
default: None 
Values:
* None - No support for CGs
* host - shares in a CG must be on pool(s) on the same host that also match the CG share type
* pool - shares in a CG must live in the same pool as the CG

==DB==
* New Tables:
class ConsistencyGroup(BASE, ManilaBase):
"""Represents a consistencygroup."""
__tablename__ = 'consistency_groups'
id = Column(String(36), primary_key=True)

user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)
deleted = Column(String(36), default='False')

host = Column(String(255))
name = Column(String(255))
description = Column(String(255))
status = Column(String(255))
source_cgsnapshot_id = Column(String(36))
share_network_id = Column(String(36), ForeignKey('share_networks.id'),
nullable=True)

class ConsistencyGroupShareTypeMapping(BASE, ManilaBase):
"""Represents the share types in a consistency group"""
__tablename__ = 'consistency_group_share_type_mappings'
id = Column(String(36), primary_key=True)
deleted = Column(String(36), default='False')
consistency_group_id = Column(String(36),
ForeignKey('consistency_groups.id'),
nullable=False)
share_type_id = Column(String(36),
ForeignKey('share_types.id'),
nullable=False)
consistency_group = orm.relationship(
ConsistencyGroup,
backref="share_types",
foreign_keys=consistency_group_id,
primaryjoin=('and_('
'ConsistencyGroupShareTypeMapping.consistency_group_id '
'== ConsistencyGroup.id,'
'ConsistencyGroupShareTypeMapping.deleted == "False")')
)

class CGSnapshot(BASE, ManilaBase):
"""Represents a cgsnapshot."""
__tablename__ = 'cgsnapshots'
id = Column(String(36), primary_key=True)

consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'))
user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)
deleted = Column(String(36), default='False')

name = Column(String(255))
description = Column(String(255))
status = Column(String(255))

consistency_group = orm.relationship(
ConsistencyGroup,
backref="cgsnapshots",
foreign_keys=consistency_group_id,
primaryjoin=('and_('
'CGSnapshot.consistency_group_id == ConsistencyGroup.id,'
'CGSnapshot.deleted == "False")')
)

class CGSnapshotMembers(BASE, ManilaBase):
__tablename__ = 'cgsnapshot_members'
id = Column(String(36), primary_key=True)
cgsnapshot_id = Column(String(36), ForeignKey('cgsnapshots.id'))
share_id = Column(String(36), ForeignKey('shares.id'))
size = Column(Integer)
status = Column(String(255))
share_proto = Column(String(255))
share_type_id = Column(String(36), ForeignKey('share_types.id'),
nullable=True)
user_id = Column(String(255))
project_id = Column(String(255))
deleted = Column(String(36), default='False')

cgsnapshot = relationship(
ConsistencyGroup,
backref="cgsnapshot_members",
foreign_keys=cgsnapshot_id,
primaryjoin='CGSnapshotMembers.cgsnapshot_id == CGSnapshot.id')

share = orm.relationship(Share, backref="cgsnapshot_members",
foreign_keys=share_id,
primaryjoin='and_('
'CGSnapshotMember.share_id == Share.id,'
'CGSnapshotMember.deleted == "False")')

* New fields in Share
consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'), nullable=True)
consistency_group = relationship(
ConsistencyGroup,
backref="shares",
foreign_keys=consistency_group_id,
primaryjoin='Share.consistency_group_id == ConsistencyGroup.id')

source_cgsnapshot_member_id = Column(String(36), nullable=True)

==API==
Schemas: https://wiki.openstack.org/wiki/Manila/design/manila-liberty-consistency-groups/api-schema

'''GET''' /os-consistency-groups/
'''GET''' /os-consistency-groups/detail
'''POST''' /os-consistency-groups
'''GET''' /os-consistency-groups/<id>
'''DELETE''' os-consistency-groups/<id>
'''PUT''' /os-consistency-groups/<id>

'''GET''' /os-cgsnapshots/
'''GET''' /os-cgsnapshots/detail
'''POST''' /os-cgsnapshots
'''PUT''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>/members
'''DELETE''' /os-cgsnapshots/<id>

Admin Actions
'''POST''' /os-consistency-groups/<id>/action # os-reset-status and os-force-delete
'''POST''' /os-cgsnapshots/<id>/action # os-reset-status and os-force-delete

Updates to Share resource
Adds consistency_group_id to POST/shares
Adds source_cgsnapshot_member_id
Add consistency_group_id query filter to /shares

==Policies==
policy.json - All consistency group policies should default to the default policy
"consistency_group:create" : "",
"consistency_group:delete": "",
"consistency_group:update": "",
"consistency_group:get": "",
"consistency_group:get_all": "",

"consistency_group:create_cgsnapshot" : "",
"consistency_group:delete_cgsnapshot": "",
"consistency_group:get_cgsnapshot": "",
"consistency_group:get_all_cgsnapshots": "",

==Driver API==

update_share_stats should now return 'consistency_group_support' pool/host/None.

def create_consistency_group(self, context, cg_dict, share_server=None):
"""Create a consistency group.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_consistency_group_from_cgsnapshot(self, context, cg_dict,
cgsnapshot_dict, share_server=None):
"""Create a consistency group from a cgsnapshot.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'shares': [<models.Share>], # The new shares being created
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:param cgsnapshot_dict:
EXAMPLE:
{'status': 'available',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': '0',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'consistency_group_id': '4b04fdc3-00b9-4909-ba1a-06e9b3f88b67',
'cgsnapshot_members': [
{'status': 'available',
'share_type_id': '1a9ed31e-ee70-483d-93ba-89690e028d7f',
'share_id': 'e14b5174-e534-4f35-bc4f-fe81c1575d6f',
'user_id': 'a0314a441ca842019b0952224aa39192',
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share': <models.Share>,
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share_proto': 'NFS',
'project_id': '13c0be6290934bd98596cfa004650049',
'cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'deleted_at': None,
'id': '6813e06b-a8f5-4784-b17d-f3e91afa370e',
'size': 1
}
],
'deleted_at': None,
'id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.

share_update_list - a list of dictionaries containing dicts for
every share created in the CG. Any share dicts should at a minimum
contain the 'id' key and 'export_locations'. Export locations
should be in the same format as returned by a share_create. This
list may be empty or None.
EXAMPLE:
[
{'id': 'uuid', 'export_locations': ['export_path']}
]
"""
raise NotImplementedError()

def delete_consistency_group(self, context, cg_dict, share_server=None):
"""Delete a consistency group

:param context:
:param cg_dict:
EXAMPLE:
:return: cg_model_update
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_cgsnapshot(self, context, snap_dict, share_server=None):
"""Create a consistency group snapshot.

:param context:
:param snap_dict:
EXAMPLE:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.

member_update_list - a list of dictionaries containing for every
member of the cgsnapshot. Each dict should contains values to be
updated for teh CGSnapshotMember in the database. This list may be
empty or None.
"""
raise NotImplementedError()

def delete_cgsnapshot(self, context, snap_dict, share_server=None):
"""Delete a consistency group snapshot

:param context:
:param snap_dict:
EXAMPLE:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.
"""
raise NotImplementedError()

== Manila Client ==
* Do we want to alias the commands so that there are Cinder consistent commands and the improved names? (I.E cg-create = consisgroup-create) Perhaps we alias the cinderclient commands to be more like this?
cg-snapshot-create Creates a cgsnapshot.
cg-snapshot-delete Removes one or more cgsnapshots.
cg-snapshot-list Lists all cgsnapshots.
cg-snapshot-show Shows cgsnapshot details.

cg-create Creates a consistency group (--cgsnapshot to create from existing cg snapshot)
cg-delete Removes one or more consistency groups.
cg-list Lists all consistencygroups.
cg-show Shows details of a consistency group.
cg-update Updates a consistencygroup.

create --consistency-group Creates a share and puts it into consistency group

= Outstanding Questions / FAQ =
# How do we avoid confusion if no backends in a deployment support CGs?
#* It appears that Cinder has the policies default to 'nobody' but this seems odd, is there a better way?
#** (ameade) I can't think of a way to have devstack set the policies automatically either.
#** (ameade) we have chosen to go with an os-consistency-groups extension for now
# Do we need a way to fake CGs with the generic driver?
#* <strike> NO, but it will not be a core feature if it cannot be supported in the generic driver.</strike>
#* Ben says that we should fake it in the generic driver so it can be tested by the gate.
# Do we need a safeguard for deleting shares that are in a CG? (I.E. protected shares?) Are shares in a CG more valuable than normal shares?
# Should we overload the Snapshots table to include snapshots in a CGSnapshot to avoid adding a new db table? Or continue with having a new table?
# Why not have the cg delete all of the shares in it when deleting the cg?
#* This could cause odd state to occur if the backend fails when deleting shares and it will not be obvious where it failed. This is also extra complexity in the implementation that could instead be hidden by a client.

= Notes on Cinder Impl =

* A consistency group may be specified on volume create
* When a consistency group is created it is scheduled to a pool and all volume created in the CG will go to that pol

Cinder cli:
cgsnapshot-create Creates a cgsnapshot and snapshots of every volume in the CG that show up in snapshot-list.
cgsnapshot-delete Removes one or more cgsnapshots.
cgsnapshot-list Lists all cgsnapshots.
cgsnapshot-show Shows cgsnapshot details. <-- /cgsnapshots/detail?all_tenants=1&name=blah
consisgroup-create Creates a consistency group.
consisgroup-create-from-src Creates a consistency group from a cgsnapshot filled with volumes from the snapshots in the cg.
consisgroup-delete Removes one or more consistency groups.
consisgroup-list Lists all consistencygroups.
consisgroup-show Shows details of a consistency group. <-- /consistencygroups/detail?all_tenants=1&name=blah
consisgroup-update Updates a consistencygroup.

create --consisgroup-id

Cinder API:
'''GET''' /consistencygroups/detail
'''POST''' /consistencygroups
{'consistencygroup':{
'name': <string>,
'description': <string>,
'volume_types': <string>,
'availability_zone': <string>,
}}
'''POST''' /consistencygroups/create_from_src
{'consistencygroup-from-src':{
'name': <string>,
'description': <string>,
'cgsnapshot_id': <string>,
}}
'''GET''' /consistencygroups/<id>
'''POST''' consistencygroups/<id>/delete
{"consistencygroup": {"force": false}}
'''PUT''' /consistencygroups/<id>
{'consistencygroup':{
'name': <string>,
'description': <string>,
'add_volumes': <string>,
'remove_volumes': <string>,
}}

'''GET''' /cgsnapshots/detail
'''GET''' /cgsnapshots/<id>
'''POST''' /cgsnapshots
{'cgsnapshot':{
'consistencygroup_id': <string>,
}}
'''DELETE''' /cgsnapshots/<id>

Adds consistencygroup_id to POST/volumes

Xings commit message:
1) Create a CG, specifying all volume types that can be supported by this
CG. The scheduler chooses a backend that supports all specified volume types.
The CG will be empty when it is first created. Backend needs to report
consistencygroup_support = True. Volume type can have the following in
extra specs: {'capabilities:consistencygroup_support': '<is> True'}.
If consistencygroup_support is not in volume type extra specs, it will be
added to filter_properties by the scheduler to make sure that the scheduler
will select the backend which reports consistency group support capability.

Create CG CLI:
cinder consisgroup-create --volume-type type1,type2 mycg1

This will add a CG entry in the new consistencygroups table.

2) After the CG is created, create a new volume and add to the CG.
Repeat until all volumes are created for the CG.

Create volume CLI (with CG):
cinder create --volume-type type1 --consisgroup-id <CG uuid> 10

This will add a consistencygroup_id foreign key in the new volume
entry in the db.

3) Create a snapshot of the CG (cgsnapshot).

Create cgsnapshot CLI:
cinder cgsnapshot-create <CG uuid>

This will add a cgsnapshot entry in the new cgsnapshots table, create
snapshot for each volume in the CG, and add a cgsnapshot_id foreign key
in each newly created snapshot entry in the db.

===Questions===
# Why does policy.json default to 'nobody' being able to perform CG commands?
# Why are volume types specified when creating a CG?
#* This is because all volumes within a CG need to be on the same backend in order for the backend to honor the CG. This is important to note in the docs so that conflicting volume-types are not specified.
# What occurs when a volume on a different backend is added to a CG via the API?
#* Currently the API ensures that the volume volume_type matches a supported volume_type of the CG and that the CG and volume are on the same host (but can be on different pools).
# Is a volume created with a CG going to end up on the same host?
#* Yes, if a volume has a CG it will be cast to the same pool as the CG.
#* What if the pool is full but there are other available pools on the same backend? Should this fail?
# What happens if a driver does not support create_from_src?
#* The CG goes to ERROR a and volumes are created from the snapshot and also in ERROR status
#* A volume in a consistency group cannot be deleted and a CG with a volume cannot be deleted, therefore a volume must be removed from the CG before it can be deleted. But what if you don't support modifying a CG?
# Why do we need a force delete for CGs?
# What do I do if I do not want to see Snapshots in a CG when doing snapshot-list?
#* I suppose API filtering could allow for someone to not see all of the snapshots if they do not want to.

=== Cinder Bugs to file===
# What happens when you force delete a CG?
#* The volumes remain but still think they are in a CG, which means you still cannot delete the volume
# consisgroup-create-from-src says cgsnapshot is and optional param in cinderclient
# Creating a volume in a CG may not end up in the cg. The only check that is performed is that the volume-type of the volume is supported by the CG but volume-types are not 1-to-1 with backends. There could be backend A and B that both match with volume type 1. A cg is created with volume type 1 and ends up on backend A. Then a volume is created with the CG specified but ends up on backend B

=== Main impl ===
299b99437c2de84222fd585f06c2d05ca213225b ConsistencyGroup: Return 400 instead of 500 for invalid body
adb4c80be82caacad83f1366a4b34e5653fd5dab Create Consistency Group from CG Snapshot API
1a62a6e60fda73bf31256fbf684dc03bb6cf0038 Modify Consistency Group API
fc7a4384be2ccf8ff24f6c0f72c681ad9133801a Add tests for consistency groups DB migration
9082273305b0b9c117eb677a0e34c2ffde4e66f0 Volume types need to be specified when creating CG
764c5ec749821d36bb0215dc6002d3caea95d3b1 Delete consistency group failed
cf961f83ac323dfad1fa5e227d1e502a17529ecc Consistency Groups

=== Ex driver impl ===
8d5b795b37ed021c2639066689645f8aa0b1012f PureISCSIDriver consistency group updates.
39c1a8e08dc854e22ada315a46c20c99df2facf8 Add support to PureISCSIDriver for Consistency Groups
92a817708a3938b1b734d2caaa206b310996d8d0 EMC VMAX driver Kilo update

Manila/design/manila-liberty-consistency-groups

2015-08-13T13:34:56Z

Alex Meade:

= Consistency Groups =
This document describes the design choices and steps for the consistency group feature in core Manila.

Consistency groups are a mechanism to allow multiple filesystem shares to be guaranteed snapshots are able to be created at the exact same point in time. For example, a database may have its tables, logs, and configuration on separate volumes. If we wanted to restore the database from a previous point in time, it would only make sense to restore the logs, tables, and configuration together from the exact same point in time.

= Manila Core Changes =
The core manila changes are very similar to what has been done in Cinder. At a high level, changes in the DB, share service, drivers, scheduler service and api service will need to be changed.

Phase 1:
* Create/Delete CGs
* Create a share within a CG
* Delete a share that is in a CG
* Snapshot entire CGs
* Create CG from a CGSnapshot

Potential future features:
* Add/remove shares in a CG
* CG migration
* Replication of a CG

==Manila vs Cinder==
* Different CLI syntax
* Restful api tweaks
** /os-consistency-groups and /os-cgsnapshots are implemented as extensions and therefore entirely optional APIs
** Change /create_from_src to just be a cgsnapshot_id in the body on POST /os-consistency-groups
** Deletes to use the DELETE HTTP verb, os-force_delete is supported via the admin actions extension (/action)
* At the Driver API, cgsnapshot objects and cg objects (with all the information about what volumes are in the CG) need to be passed to the driver instead of just the cg/cgsnapshot name. This is important for backends that aren't managing a CG construct such as NetApp cDOT as they need to do things such as loop through the snapshots in a cg to delete them.
* The default policy in policy.json for consistency group operations is the default policy versus in Cinder is it 'nobody'
* Unlike Cinder, snapshots in a CGsnapshot are not the same as a normal snapshot. A CGsnapshot is treated as a single unit instead of a collection of snapshots.
* consistency_group_support scheduler capability has multiple values instead of boolean. This is to accommodate backends that require all volumes in a CG to be on the same pool vs. just the same host. It also allows for adding additional values in the future if a backend can support some other paradigm like a CG spanning multiple backends.

== User Workflows ==
* Snapshot multiple shares
*# Create a Consistency Group
*# Create share(s) with a CG specified
*# Create a CGsnapshot

* Create a CG
*# POST to /os-consistency-group
*#* specifiy share_types or the default share type will be used

* Copy a consistency group
*# Create a CGsnapshot
*# Create a CG with the CGSnapshot id

* Delete a CG
*# Delete all CG snapshots of the CG
*# Delete all shares in the CG
*# Delete the CG (Only empty CGs can be deleted)

* List shares in a given CG
*# GET /shares?consistencygroup_id=<cg_id>

* List the shares that were captured in a given CGSnapshot
*# GET /cgsnapshots/<id>/members

== System Workflows ==
* Creating a CG
*# Create the CG in the DB
*# Cast the consistency group creation to the scheduler
*## Choose a host for the CG
*# Cast the creation to the share service
*## Get the CG info from DB
*## Call the driver
*## Update database with info returned from driver (ex: CG status)

* Creating a CG from a cgsnapshot
*# Create a new CG in the DB
*# Create new shares from CG snapshot members in the DB
*# Schedule the CG create to the scheduler with the orig CG host/pool+share_type (depending on capability)
*# Call the driver with all info about source shares, new shares, cg, and cgsnapshot info.
*# Update CGsnapshot status and all shares status' to available

* Creating a cg snapshot
*# Create cgsnapshot entity in DB
*# For all shares in CG, create cgsnapshot_member in DB
*# cast create_cgsnapshot to the driver

* Adding a share to a CG
** On share creation:
**# Cast to scheduler with the host of the CG
**#* if perpool capable: Goes to the same pool as CG
**#* if per backend: goes to a pool on the same host that matches a CG share type

* Deleting a share (in a CG)
** Not allowed if the CG has snapshots of the share ( check if the share_id is in the CGSnapshotMembers tables)

== Scheduler ==
manila/scheduler/filter_scheduler.py should now look for consistency_group_support as a capability.

consistency_group_support
default: None 
Values:
* None - No support for CGs
* host - shares in a CG must be on pool(s) on the same host that also match the CG share type
* pool - shares in a CG must live in the same pool as the CG

==DB==
* New Tables:
class ConsistencyGroup(BASE, ManilaBase):
"""Represents a consistencygroup."""
__tablename__ = 'consistency_groups'
id = Column(String(36), primary_key=True)

user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)
deleted = Column(String(36), default='False')

host = Column(String(255))
name = Column(String(255))
description = Column(String(255))
status = Column(String(255))
source_cgsnapshot_id = Column(String(36))
share_network_id = Column(String(36), ForeignKey('share_networks.id'),
nullable=True)

class ConsistencyGroupShareTypeMapping(BASE, ManilaBase):
"""Represents the share types in a consistency group"""
__tablename__ = 'consistency_group_share_type_mappings'
id = Column(String(36), primary_key=True)
deleted = Column(String(36), default='False')
consistency_group_id = Column(String(36),
ForeignKey('consistency_groups.id'),
nullable=False)
share_type_id = Column(String(36),
ForeignKey('share_types.id'),
nullable=False)
consistency_group = orm.relationship(
ConsistencyGroup,
backref="share_types",
foreign_keys=consistency_group_id,
primaryjoin=('and_('
'ConsistencyGroupShareTypeMapping.consistency_group_id '
'== ConsistencyGroup.id,'
'ConsistencyGroupShareTypeMapping.deleted == "False")')
)

class CGSnapshot(BASE, ManilaBase):
"""Represents a cgsnapshot."""
__tablename__ = 'cgsnapshots'
id = Column(String(36), primary_key=True)

consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'))
user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)
deleted = Column(String(36), default='False')

name = Column(String(255))
description = Column(String(255))
status = Column(String(255))

consistency_group = orm.relationship(
ConsistencyGroup,
backref="cgsnapshots",
foreign_keys=consistency_group_id,
primaryjoin=('and_('
'CGSnapshot.consistency_group_id == ConsistencyGroup.id,'
'CGSnapshot.deleted == "False")')
)

class CGSnapshotMembers(BASE, ManilaBase):
__tablename__ = 'cgsnapshot_members'
id = Column(String(36), primary_key=True)
cgsnapshot_id = Column(String(36), ForeignKey('cgsnapshots.id'))
share_id = Column(String(36), ForeignKey('shares.id'))
size = Column(Integer)
status = Column(String(255))
share_proto = Column(String(255))
share_type_id = Column(String(36), ForeignKey('share_types.id'),
nullable=True)
user_id = Column(String(255))
project_id = Column(String(255))
deleted = Column(String(36), default='False')

cgsnapshot = relationship(
ConsistencyGroup,
backref="cgsnapshot_members",
foreign_keys=cgsnapshot_id,
primaryjoin='CGSnapshotMembers.cgsnapshot_id == CGSnapshot.id')

share = orm.relationship(Share, backref="cgsnapshot_members",
foreign_keys=share_id,
primaryjoin='and_('
'CGSnapshotMember.share_id == Share.id,'
'CGSnapshotMember.deleted == "False")')

* New fields in Share
consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'), nullable=True)
consistency_group = relationship(
ConsistencyGroup,
backref="shares",
foreign_keys=consistency_group_id,
primaryjoin='Share.consistency_group_id == ConsistencyGroup.id')

source_cgsnapshot_member_id = Column(String(36), nullable=True)

==API==
Schemas: https://wiki.openstack.org/wiki/Manila/design/manila-liberty-consistency-groups/api-schema

'''GET''' /os-consistency-groups/
'''GET''' /os-consistency-groups/detail
'''POST''' /os-consistency-groups
'''GET''' /os-consistency-groups/<id>
'''DELETE''' os-consistency-groups/<id>
'''PUT''' /os-consistency-groups/<id>

'''GET''' /os-cgsnapshots/
'''GET''' /os-cgsnapshots/detail
'''POST''' /os-cgsnapshots
'''PUT''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>/members
'''DELETE''' /os-cgsnapshots/<id>

Admin Actions
'''POST''' /os-consistency-groups/<id>/action # os-reset-status and os-force-delete
'''POST''' /os-cgsnapshots/<id>/action # os-reset-status and os-force-delete

Updates to Share resource
Adds consistency_group_id to POST/shares
Adds source_cgsnapshot_member_id
Add consistency_group_id query filter to /shares

==Policies==
policy.json - All consistency group policies should default to the default policy
"consistency_group:create" : "",
"consistency_group:delete": "",
"consistency_group:update": "",
"consistency_group:get": "",
"consistency_group:get_all": "",

"consistency_group:create_cgsnapshot" : "",
"consistency_group:delete_cgsnapshot": "",
"consistency_group:get_cgsnapshot": "",
"consistency_group:get_all_cgsnapshots": "",

==Driver API==

update_share_stats should now return 'consistency_group_support' pool/host/None.

def create_consistency_group(self, context, cg_dict, share_server=None):
"""Create a consistency group.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_consistency_group_from_cgsnapshot(self, context, cg_dict,
cgsnapshot_dict, share_server=None):
"""Create a consistency group from a cgsnapshot.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'shares': [<models.Share>], # The new shares being created
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:param cgsnapshot_dict:
EXAMPLE:
{'status': 'available',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': '0',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'consistency_group_id': '4b04fdc3-00b9-4909-ba1a-06e9b3f88b67',
'cgsnapshot_members': [
{'status': 'available',
'share_type_id': '1a9ed31e-ee70-483d-93ba-89690e028d7f',
'share_id': 'e14b5174-e534-4f35-bc4f-fe81c1575d6f',
'user_id': 'a0314a441ca842019b0952224aa39192',
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share': <models.Share>,
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share_proto': 'NFS',
'project_id': '13c0be6290934bd98596cfa004650049',
'cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'deleted_at': None,
'id': '6813e06b-a8f5-4784-b17d-f3e91afa370e',
'size': 1
}
],
'deleted_at': None,
'id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.

share_update_list - a list of dictionaries containing dicts for
every share created in the CG. Any share dicts should at a minimum
contain the 'id' key and 'export_locations'. Export locations
should be in the same format as returned by a share_create. This
list may be empty or None.
EXAMPLE:
[
{'id': 'uuid', 'export_locations': ['export_path']}
]
"""
raise NotImplementedError()

def delete_consistency_group(self, context, cg_dict, share_server=None):
"""Delete a consistency group

:param context:
:param cg_dict:
EXAMPLE:
:return: cg_model_update
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_cgsnapshot(self, context, snap_dict, share_server=None):
"""Create a consistency group snapshot.

:param context:
:param snap_dict:
EXAMPLE:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.

member_update_list - a list of dictionaries containing for every
member of the cgsnapshot. Each dict should contains values to be
updated for teh CGSnapshotMember in the database. This list may be
empty or None.
"""
raise NotImplementedError()

def delete_cgsnapshot(self, context, snap_dict, share_server=None):
"""Delete a consistency group snapshot

:param context:
:param snap_dict:
EXAMPLE:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.
"""
raise NotImplementedError()

== Manila Client ==
* Do we want to alias the commands so that there are Cinder consistent commands and the improved names? (I.E cg-create = consisgroup-create) Perhaps we alias the cinderclient commands to be more like this?
cg-snapshot-create Creates a cgsnapshot.
cg-snapshot-delete Removes one or more cgsnapshots.
cg-snapshot-list Lists all cgsnapshots.
cg-snapshot-show Shows cgsnapshot details.

cg-create Creates a consistency group (--cgsnapshot to create from existing cg snapshot)
cg-delete Removes one or more consistency groups.
cg-list Lists all consistencygroups.
cg-show Shows details of a consistency group.
cg-update Updates a consistencygroup.

create --consistency-group Creates a share and puts it into consistency group

= Outstanding Questions / FAQ =
# How do we avoid confusion if no backends in a deployment support CGs?
#* It appears that Cinder has the policies default to 'nobody' but this seems odd, is there a better way?
#** (ameade) I can't think of a way to have devstack set the policies automatically either.
#** (ameade) we have chosen to go with an os-consistency-groups extension for now
# Do we need a way to fake CGs with the generic driver?
#* <strike> NO, but it will not be a core feature if it cannot be supported in the generic driver.</strike>
#* Ben says that we should fake it in the generic driver so it can be tested by the gate.
# Do we need a safeguard for deleting shares that are in a CG? (I.E. protected shares?) Are shares in a CG more valuable than normal shares?
# Should we overload the Snapshots table to include snapshots in a CGSnapshot to avoid adding a new db table? Or continue with having a new table?
# Why not have the cg delete all of the shares in it when deleting the cg?
#* This could cause odd state to occur if the backend fails when deleting shares and it will not be obvious where it failed. This is also extra complexity in the implementation that could instead be hidden by a client.

= Notes on Cinder Impl =

* A consistency group may be specified on volume create
* When a consistency group is created it is scheduled to a pool and all volume created in the CG will go to that pol

Cinder cli:
cgsnapshot-create Creates a cgsnapshot and snapshots of every volume in the CG that show up in snapshot-list.
cgsnapshot-delete Removes one or more cgsnapshots.
cgsnapshot-list Lists all cgsnapshots.
cgsnapshot-show Shows cgsnapshot details. <-- /cgsnapshots/detail?all_tenants=1&name=blah
consisgroup-create Creates a consistency group.
consisgroup-create-from-src Creates a consistency group from a cgsnapshot filled with volumes from the snapshots in the cg.
consisgroup-delete Removes one or more consistency groups.
consisgroup-list Lists all consistencygroups.
consisgroup-show Shows details of a consistency group. <-- /consistencygroups/detail?all_tenants=1&name=blah
consisgroup-update Updates a consistencygroup.

create --consisgroup-id

Cinder API:
'''GET''' /consistencygroups/detail
'''POST''' /consistencygroups
{'consistencygroup':{
'name': <string>,
'description': <string>,
'volume_types': <string>,
'availability_zone': <string>,
}}
'''POST''' /consistencygroups/create_from_src
{'consistencygroup-from-src':{
'name': <string>,
'description': <string>,
'cgsnapshot_id': <string>,
}}
'''GET''' /consistencygroups/<id>
'''POST''' consistencygroups/<id>/delete
{"consistencygroup": {"force": false}}
'''PUT''' /consistencygroups/<id>
{'consistencygroup':{
'name': <string>,
'description': <string>,
'add_volumes': <string>,
'remove_volumes': <string>,
}}

'''GET''' /cgsnapshots/detail
'''GET''' /cgsnapshots/<id>
'''POST''' /cgsnapshots
{'cgsnapshot':{
'consistencygroup_id': <string>,
}}
'''DELETE''' /cgsnapshots/<id>

Adds consistencygroup_id to POST/volumes

Xings commit message:
1) Create a CG, specifying all volume types that can be supported by this
CG. The scheduler chooses a backend that supports all specified volume types.
The CG will be empty when it is first created. Backend needs to report
consistencygroup_support = True. Volume type can have the following in
extra specs: {'capabilities:consistencygroup_support': '<is> True'}.
If consistencygroup_support is not in volume type extra specs, it will be
added to filter_properties by the scheduler to make sure that the scheduler
will select the backend which reports consistency group support capability.

Create CG CLI:
cinder consisgroup-create --volume-type type1,type2 mycg1

This will add a CG entry in the new consistencygroups table.

2) After the CG is created, create a new volume and add to the CG.
Repeat until all volumes are created for the CG.

Create volume CLI (with CG):
cinder create --volume-type type1 --consisgroup-id <CG uuid> 10

This will add a consistencygroup_id foreign key in the new volume
entry in the db.

3) Create a snapshot of the CG (cgsnapshot).

Create cgsnapshot CLI:
cinder cgsnapshot-create <CG uuid>

This will add a cgsnapshot entry in the new cgsnapshots table, create
snapshot for each volume in the CG, and add a cgsnapshot_id foreign key
in each newly created snapshot entry in the db.

===Questions===
# Why does policy.json default to 'nobody' being able to perform CG commands?
# Why are volume types specified when creating a CG?
#* This is because all volumes within a CG need to be on the same backend in order for the backend to honor the CG. This is important to note in the docs so that conflicting volume-types are not specified.
# What occurs when a volume on a different backend is added to a CG via the API?
#* Currently the API ensures that the volume volume_type matches a supported volume_type of the CG and that the CG and volume are on the same host (but can be on different pools).
# Is a volume created with a CG going to end up on the same host?
#* Yes, if a volume has a CG it will be cast to the same pool as the CG.
#* What if the pool is full but there are other available pools on the same backend? Should this fail?
# What happens if a driver does not support create_from_src?
#* The CG goes to ERROR a and volumes are created from the snapshot and also in ERROR status
#* A volume in a consistency group cannot be deleted and a CG with a volume cannot be deleted, therefore a volume must be removed from the CG before it can be deleted. But what if you don't support modifying a CG?
# Why do we need a force delete for CGs?
# What do I do if I do not want to see Snapshots in a CG when doing snapshot-list?
#* I suppose API filtering could allow for someone to not see all of the snapshots if they do not want to.

=== Cinder Bugs to file===
# What happens when you force delete a CG?
#* The volumes remain but still think they are in a CG, which means you still cannot delete the volume
# consisgroup-create-from-src says cgsnapshot is and optional param in cinderclient
# Creating a volume in a CG may not end up in the cg. The only check that is performed is that the volume-type of the volume is supported by the CG but volume-types are not 1-to-1 with backends. There could be backend A and B that both match with volume type 1. A cg is created with volume type 1 and ends up on backend A. Then a volume is created with the CG specified but ends up on backend B

=== Main impl ===
299b99437c2de84222fd585f06c2d05ca213225b ConsistencyGroup: Return 400 instead of 500 for invalid body
adb4c80be82caacad83f1366a4b34e5653fd5dab Create Consistency Group from CG Snapshot API
1a62a6e60fda73bf31256fbf684dc03bb6cf0038 Modify Consistency Group API
fc7a4384be2ccf8ff24f6c0f72c681ad9133801a Add tests for consistency groups DB migration
9082273305b0b9c117eb677a0e34c2ffde4e66f0 Volume types need to be specified when creating CG
764c5ec749821d36bb0215dc6002d3caea95d3b1 Delete consistency group failed
cf961f83ac323dfad1fa5e227d1e502a17529ecc Consistency Groups

=== Ex driver impl ===
8d5b795b37ed021c2639066689645f8aa0b1012f PureISCSIDriver consistency group updates.
39c1a8e08dc854e22ada315a46c20c99df2facf8 Add support to PureISCSIDriver for Consistency Groups
92a817708a3938b1b734d2caaa206b310996d8d0 EMC VMAX driver Kilo update

Manila/design/manila-liberty-consistency-groups

2015-08-13T13:31:20Z

Alex Meade: /* System Workflows */

= Consistency Groups =
This document describes the design choices and steps for the consistency group feature in core Manila.

Consistency groups are a mechanism to allow multiple filesystem shares to be guaranteed snapshots are able to be created at the exact same point in time. For example, a database may have its tables, logs, and configuration on separate volumes. If we wanted to restore the database from a previous point in time, it would only make sense to restore the logs, tables, and configuration together from the exact same point in time.

= Manila Core Changes =
The core manila changes are very similar to what has been done in Cinder. At a high level, changes in the DB, share service, drivers, scheduler service and api service will need to be changed.

Phase 1:
* Create/Delete CGs
* Create a share within a CG
* Delete a share that is in a CG
* Snapshot entire CGs
* Create CG from a CGSnapshot

Potential future features:
* Add/remove shares in a CG
* CG migration
* Replication of a CG

==Manila vs Cinder==
* Different CLI syntax
* Restful api tweaks
** /os-consistency-groups and /os-cgsnapshots are implemented as extensions and therefore entirely optional APIs
** Change /create_from_src to just be a cgsnapshot_id in the body on POST /os-consistency-groups
** Deletes to use the DELETE HTTP verb, os-force_delete is supported via the admin actions extension (/action)
* At the Driver API, cgsnapshot objects and cg objects (with all the information about what volumes are in the CG) need to be passed to the driver instead of just the cg/cgsnapshot name. This is important for backends that aren't managing a CG construct such as NetApp cDOT as they need to do things such as loop through the snapshots in a cg to delete them.
* The default policy in policy.json for consistency group operations is the default policy versus in Cinder is it 'nobody'
* Unlike Cinder, snapshots in a CGsnapshot are not the same as a normal snapshot. A CGsnapshot is treated as a single unit instead of a collection of snapshots.
* consistency_group_support scheduler capability has multiple values instead of boolean. This is to accommodate backends that require all volumes in a CG to be on the same pool vs. just the same host. It also allows for adding additional values in the future if a backend can support some other paradigm like a CG spanning multiple backends.

== User Workflows ==
* Snapshot multiple shares
*# Create a Consistency Group
*# Create share(s) with a CG specified
*# Create a CGsnapshot

* Create a CG
*# POST to /os-consistency-group
*#* specifiy share_types or the default share type will be used

* Copy a consistency group
*# Create a CGsnapshot
*# Create a CG with the CGSnapshot id

* Delete a CG
*# Delete all CG snapshots of the CG
*# Delete all shares in the CG
*# Delete the CG (Only empty CGs can be deleted)

* List shares in a given CG
*# GET /shares?consistencygroup_id=<cg_id>

* List the shares that were captured in a given CGSnapshot
*# GET /cgsnapshots/<id>/members

== System Workflows ==
* Creating a CG
*# Create the CG in the DB
*# Cast the consistency group creation to the scheduler
*## Choose a host for the CG
*# Cast the creation to the share service
*## Get the CG info from DB
*## Call the driver
*## Update database with info returned from driver (ex: CG status)

* Creating a CG from a cgsnapshot
*# Create a new CG in the DB
*# Create new shares from CG snapshot members in the DB
*# Schedule the CG create to the scheduler with the orig CG host/pool+share_type (depending on capability)
*# Call the driver with all info about source shares, new shares, cg, and cgsnapshot info.
*# Update CGsnapshot status and all shares status' to available

* Creating a cg snapshot
*# Create cgsnapshot entity in DB
*# For all shares in CG, create cgsnapshot_member in DB
*# cast create_cgsnapshot to the driver

* Adding a share to a CG
** On share creation:
**# Cast to scheduler with the host of the CG
**#* if perpool capable: Goes to the same pool as CG
**#* if per backend: goes to a pool on the same host that matches a CG share type

* Deleting a share (in a CG)
** Not allowed if the CG has snapshots of the share ( check if the share_id is in the CGSnapshotMembers tables)

== Scheduler ==
manila/scheduler/filter_scheduler.py should now look for consistency_group_support as a capability.

consistency_group_support
default: None 
Values:
* None - No support for CGs
* host - shares in a CG must be on pool(s) on the same host that also match the CG share type
* pool - shares in a CG must live in the same pool as the CG

==DB==
* New Tables:
class ConsistencyGroup(BASE, ManilaBase):
"""Represents a consistencygroup."""
__tablename__ = 'consistency_groups'
id = Column(String(36), primary_key=True)

user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)
deleted = Column(String(36), default='False')

host = Column(String(255))
name = Column(String(255))
description = Column(String(255))
status = Column(String(255))
source_cgsnapshot_id = Column(String(36))

class ConsistencyGroupShareTypeMapping(BASE, ManilaBase):
"""Represents the share types in a consistency group"""
__tablename__ = 'consistency_group_share_type_mappings'
id = Column(Integer, primary_key=True)
deleted = Column(String(36), default='False')
consistency_group_id = Column(String(36),
ForeignKey('consistency_groups.id'),
nullable=False)
share_type_id = Column(String(36),
ForeignKey('share_types.id'),
nullable=False)
consistency_group = orm.relationship(
ConsistencyGroup,
backref="share_types",
foreign_keys=consistency_group_id,
primaryjoin=('and_('
'ConsistencyGroupShareTypeMapping.consistency_group_id == '
'ConsistencyGroups.id,'
'ConsistencyGroupShareTypeMapping.deleted == 0)')
)

class CGSnapshot(BASE, ManilaBase):
"""Represents a cgsnapshot."""
__tablename__ = 'cgsnapshots'
id = Column(String(36), primary_key=True)

consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'))
user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)

name = Column(String(255))
description = Column(String(255))
status = Column(String(255))

class CGSnapshotMembers(BASE, ManilaBase):
__tablename__ = 'cgsnapshot_members'
id = Column(String(36), primary_key=True)
cgsnapshot_id = Column(String(36), ForeignKey('cgsnapshots.id'))
share_id = Column(String(36), ForeignKey('shares.id'))
size = Column(Integer)
status = Column(String(255))
share_proto = Column(String(255))
share_type_id = Column(String(36), ForeignKey('share_types.id'),
nullable=True)
user_id = Column(String(255))
project_id = Column(String(255))
deleted = Column(String(36), default='False')

cgsnapshot = relationship(
ConsistencyGroup,
backref="cgsnapshot_members",
foreign_keys=cgsnapshot_id,
primaryjoin='CGSnapshotMembers.cgsnapshot_id == CGSnapshot.id')

* New fields in Share
consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'), nullable=True)
consistency_group = relationship(
ConsistencyGroup,
backref="shares",
foreign_keys=consistency_group_id,
primaryjoin='Share.consistency_group_id == ConsistencyGroup.id')

source_cgsnapshot_member_id = Column(String(36), nullable=True)

==API==
Schemas: https://wiki.openstack.org/wiki/Manila/design/manila-liberty-consistency-groups/api-schema

'''GET''' /os-consistency-groups/
'''GET''' /os-consistency-groups/detail
'''POST''' /os-consistency-groups
'''GET''' /os-consistency-groups/<id>
'''DELETE''' os-consistency-groups/<id>
'''PUT''' /os-consistency-groups/<id>

'''GET''' /os-cgsnapshots/
'''GET''' /os-cgsnapshots/detail
'''POST''' /os-cgsnapshots
'''PUT''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>/members
'''DELETE''' /os-cgsnapshots/<id>

Admin Actions
'''POST''' /os-consistency-groups/<id>/action # os-reset-status and os-force-delete
'''POST''' /os-cgsnapshots/<id>/action # os-reset-status and os-force-delete

Updates to Share resource
Adds consistency_group_id to POST/shares
Adds source_cgsnapshot_member_id
Add consistency_group_id query filter to /shares

==Policies==
policy.json - All consistency group policies should default to the default policy
"consistency_group:create" : "",
"consistency_group:delete": "",
"consistency_group:update": "",
"consistency_group:get": "",
"consistency_group:get_all": "",

"consistency_group:create_cgsnapshot" : "",
"consistency_group:delete_cgsnapshot": "",
"consistency_group:get_cgsnapshot": "",
"consistency_group:get_all_cgsnapshots": "",

==Driver API==

update_share_stats should now return 'consistency_group_support' pool/host/None.

def create_consistency_group(self, context, cg_dict, share_server=None):
"""Create a consistency group.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_consistency_group_from_cgsnapshot(self, context, cg_dict,
cgsnapshot_dict, share_server=None):
"""Create a consistency group from a cgsnapshot.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'shares': [<models.Share>], # The new shares being created
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:param cgsnapshot_dict:
EXAMPLE:
{'status': 'available',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': '0',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'consistency_group_id': '4b04fdc3-00b9-4909-ba1a-06e9b3f88b67',
'cgsnapshot_members': [
{'status': 'available',
'share_type_id': '1a9ed31e-ee70-483d-93ba-89690e028d7f',
'share_id': 'e14b5174-e534-4f35-bc4f-fe81c1575d6f',
'user_id': 'a0314a441ca842019b0952224aa39192',
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share': <models.Share>,
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share_proto': 'NFS',
'project_id': '13c0be6290934bd98596cfa004650049',
'cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'deleted_at': None,
'id': '6813e06b-a8f5-4784-b17d-f3e91afa370e',
'size': 1
}
],
'deleted_at': None,
'id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.

share_update_list - a list of dictionaries containing dicts for
every share created in the CG. Any share dicts should at a minimum
contain the 'id' key and 'export_locations'. Export locations
should be in the same format as returned by a share_create. This
list may be empty or None.
EXAMPLE:
[
{'id': 'uuid', 'export_locations': ['export_path']}
]
"""
raise NotImplementedError()

def delete_consistency_group(self, context, cg_dict, share_server=None):
"""Delete a consistency group

:param context:
:param cg_dict:
EXAMPLE:
:return: cg_model_update
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_cgsnapshot(self, context, snap_dict, share_server=None):
"""Create a consistency group snapshot.

:param context:
:param snap_dict:
EXAMPLE:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.

member_update_list - a list of dictionaries containing for every
member of the cgsnapshot. Each dict should contains values to be
updated for teh CGSnapshotMember in the database. This list may be
empty or None.
"""
raise NotImplementedError()

def delete_cgsnapshot(self, context, snap_dict, share_server=None):
"""Delete a consistency group snapshot

:param context:
:param snap_dict:
EXAMPLE:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.
"""
raise NotImplementedError()

== Manila Client ==
* Do we want to alias the commands so that there are Cinder consistent commands and the improved names? (I.E cg-create = consisgroup-create) Perhaps we alias the cinderclient commands to be more like this?
cg-snapshot-create Creates a cgsnapshot.
cg-snapshot-delete Removes one or more cgsnapshots.
cg-snapshot-list Lists all cgsnapshots.
cg-snapshot-show Shows cgsnapshot details.

cg-create Creates a consistency group (--cgsnapshot to create from existing cg snapshot)
cg-delete Removes one or more consistency groups.
cg-list Lists all consistencygroups.
cg-show Shows details of a consistency group.
cg-update Updates a consistencygroup.

create --consistency-group Creates a share and puts it into consistency group

= Outstanding Questions / FAQ =
# How do we avoid confusion if no backends in a deployment support CGs?
#* It appears that Cinder has the policies default to 'nobody' but this seems odd, is there a better way?
#** (ameade) I can't think of a way to have devstack set the policies automatically either.
#** (ameade) we have chosen to go with an os-consistency-groups extension for now
# Do we need a way to fake CGs with the generic driver?
#* <strike> NO, but it will not be a core feature if it cannot be supported in the generic driver.</strike>
#* Ben says that we should fake it in the generic driver so it can be tested by the gate.
# Do we need a safeguard for deleting shares that are in a CG? (I.E. protected shares?) Are shares in a CG more valuable than normal shares?
# Should we overload the Snapshots table to include snapshots in a CGSnapshot to avoid adding a new db table? Or continue with having a new table?
# Why not have the cg delete all of the shares in it when deleting the cg?
#* This could cause odd state to occur if the backend fails when deleting shares and it will not be obvious where it failed. This is also extra complexity in the implementation that could instead be hidden by a client.

= Notes on Cinder Impl =

* A consistency group may be specified on volume create
* When a consistency group is created it is scheduled to a pool and all volume created in the CG will go to that pol

Cinder cli:
cgsnapshot-create Creates a cgsnapshot and snapshots of every volume in the CG that show up in snapshot-list.
cgsnapshot-delete Removes one or more cgsnapshots.
cgsnapshot-list Lists all cgsnapshots.
cgsnapshot-show Shows cgsnapshot details. <-- /cgsnapshots/detail?all_tenants=1&name=blah
consisgroup-create Creates a consistency group.
consisgroup-create-from-src Creates a consistency group from a cgsnapshot filled with volumes from the snapshots in the cg.
consisgroup-delete Removes one or more consistency groups.
consisgroup-list Lists all consistencygroups.
consisgroup-show Shows details of a consistency group. <-- /consistencygroups/detail?all_tenants=1&name=blah
consisgroup-update Updates a consistencygroup.

create --consisgroup-id

Cinder API:
'''GET''' /consistencygroups/detail
'''POST''' /consistencygroups
{'consistencygroup':{
'name': <string>,
'description': <string>,
'volume_types': <string>,
'availability_zone': <string>,
}}
'''POST''' /consistencygroups/create_from_src
{'consistencygroup-from-src':{
'name': <string>,
'description': <string>,
'cgsnapshot_id': <string>,
}}
'''GET''' /consistencygroups/<id>
'''POST''' consistencygroups/<id>/delete
{"consistencygroup": {"force": false}}
'''PUT''' /consistencygroups/<id>
{'consistencygroup':{
'name': <string>,
'description': <string>,
'add_volumes': <string>,
'remove_volumes': <string>,
}}

'''GET''' /cgsnapshots/detail
'''GET''' /cgsnapshots/<id>
'''POST''' /cgsnapshots
{'cgsnapshot':{
'consistencygroup_id': <string>,
}}
'''DELETE''' /cgsnapshots/<id>

Adds consistencygroup_id to POST/volumes

Xings commit message:
1) Create a CG, specifying all volume types that can be supported by this
CG. The scheduler chooses a backend that supports all specified volume types.
The CG will be empty when it is first created. Backend needs to report
consistencygroup_support = True. Volume type can have the following in
extra specs: {'capabilities:consistencygroup_support': '<is> True'}.
If consistencygroup_support is not in volume type extra specs, it will be
added to filter_properties by the scheduler to make sure that the scheduler
will select the backend which reports consistency group support capability.

Create CG CLI:
cinder consisgroup-create --volume-type type1,type2 mycg1

This will add a CG entry in the new consistencygroups table.

2) After the CG is created, create a new volume and add to the CG.
Repeat until all volumes are created for the CG.

Create volume CLI (with CG):
cinder create --volume-type type1 --consisgroup-id <CG uuid> 10

This will add a consistencygroup_id foreign key in the new volume
entry in the db.

3) Create a snapshot of the CG (cgsnapshot).

Create cgsnapshot CLI:
cinder cgsnapshot-create <CG uuid>

This will add a cgsnapshot entry in the new cgsnapshots table, create
snapshot for each volume in the CG, and add a cgsnapshot_id foreign key
in each newly created snapshot entry in the db.

===Questions===
# Why does policy.json default to 'nobody' being able to perform CG commands?
# Why are volume types specified when creating a CG?
#* This is because all volumes within a CG need to be on the same backend in order for the backend to honor the CG. This is important to note in the docs so that conflicting volume-types are not specified.
# What occurs when a volume on a different backend is added to a CG via the API?
#* Currently the API ensures that the volume volume_type matches a supported volume_type of the CG and that the CG and volume are on the same host (but can be on different pools).
# Is a volume created with a CG going to end up on the same host?
#* Yes, if a volume has a CG it will be cast to the same pool as the CG.
#* What if the pool is full but there are other available pools on the same backend? Should this fail?
# What happens if a driver does not support create_from_src?
#* The CG goes to ERROR a and volumes are created from the snapshot and also in ERROR status
#* A volume in a consistency group cannot be deleted and a CG with a volume cannot be deleted, therefore a volume must be removed from the CG before it can be deleted. But what if you don't support modifying a CG?
# Why do we need a force delete for CGs?
# What do I do if I do not want to see Snapshots in a CG when doing snapshot-list?
#* I suppose API filtering could allow for someone to not see all of the snapshots if they do not want to.

=== Cinder Bugs to file===
# What happens when you force delete a CG?
#* The volumes remain but still think they are in a CG, which means you still cannot delete the volume
# consisgroup-create-from-src says cgsnapshot is and optional param in cinderclient
# Creating a volume in a CG may not end up in the cg. The only check that is performed is that the volume-type of the volume is supported by the CG but volume-types are not 1-to-1 with backends. There could be backend A and B that both match with volume type 1. A cg is created with volume type 1 and ends up on backend A. Then a volume is created with the CG specified but ends up on backend B

=== Main impl ===
299b99437c2de84222fd585f06c2d05ca213225b ConsistencyGroup: Return 400 instead of 500 for invalid body
adb4c80be82caacad83f1366a4b34e5653fd5dab Create Consistency Group from CG Snapshot API
1a62a6e60fda73bf31256fbf684dc03bb6cf0038 Modify Consistency Group API
fc7a4384be2ccf8ff24f6c0f72c681ad9133801a Add tests for consistency groups DB migration
9082273305b0b9c117eb677a0e34c2ffde4e66f0 Volume types need to be specified when creating CG
764c5ec749821d36bb0215dc6002d3caea95d3b1 Delete consistency group failed
cf961f83ac323dfad1fa5e227d1e502a17529ecc Consistency Groups

=== Ex driver impl ===
8d5b795b37ed021c2639066689645f8aa0b1012f PureISCSIDriver consistency group updates.
39c1a8e08dc854e22ada315a46c20c99df2facf8 Add support to PureISCSIDriver for Consistency Groups
92a817708a3938b1b734d2caaa206b310996d8d0 EMC VMAX driver Kilo update

Manila/design/manila-liberty-consistency-groups

2015-08-13T13:28:11Z

Alex Meade: /* User Workflows */

= Consistency Groups =
This document describes the design choices and steps for the consistency group feature in core Manila.

Consistency groups are a mechanism to allow multiple filesystem shares to be guaranteed snapshots are able to be created at the exact same point in time. For example, a database may have its tables, logs, and configuration on separate volumes. If we wanted to restore the database from a previous point in time, it would only make sense to restore the logs, tables, and configuration together from the exact same point in time.

= Manila Core Changes =
The core manila changes are very similar to what has been done in Cinder. At a high level, changes in the DB, share service, drivers, scheduler service and api service will need to be changed.

Phase 1:
* Create/Delete CGs
* Create a share within a CG
* Delete a share that is in a CG
* Snapshot entire CGs
* Create CG from a CGSnapshot

Potential future features:
* Add/remove shares in a CG
* CG migration
* Replication of a CG

==Manila vs Cinder==
* Different CLI syntax
* Restful api tweaks
** /os-consistency-groups and /os-cgsnapshots are implemented as extensions and therefore entirely optional APIs
** Change /create_from_src to just be a cgsnapshot_id in the body on POST /os-consistency-groups
** Deletes to use the DELETE HTTP verb, os-force_delete is supported via the admin actions extension (/action)
* At the Driver API, cgsnapshot objects and cg objects (with all the information about what volumes are in the CG) need to be passed to the driver instead of just the cg/cgsnapshot name. This is important for backends that aren't managing a CG construct such as NetApp cDOT as they need to do things such as loop through the snapshots in a cg to delete them.
* The default policy in policy.json for consistency group operations is the default policy versus in Cinder is it 'nobody'
* Unlike Cinder, snapshots in a CGsnapshot are not the same as a normal snapshot. A CGsnapshot is treated as a single unit instead of a collection of snapshots.
* consistency_group_support scheduler capability has multiple values instead of boolean. This is to accommodate backends that require all volumes in a CG to be on the same pool vs. just the same host. It also allows for adding additional values in the future if a backend can support some other paradigm like a CG spanning multiple backends.

== User Workflows ==
* Snapshot multiple shares
*# Create a Consistency Group
*# Create share(s) with a CG specified
*# Create a CGsnapshot

* Create a CG
*# POST to /os-consistency-group
*#* specifiy share_types or the default share type will be used

* Copy a consistency group
*# Create a CGsnapshot
*# Create a CG with the CGSnapshot id

* Delete a CG
*# Delete all CG snapshots of the CG
*# Delete all shares in the CG
*# Delete the CG (Only empty CGs can be deleted)

* List shares in a given CG
*# GET /shares?consistencygroup_id=<cg_id>

* List the shares that were captured in a given CGSnapshot
*# GET /cgsnapshots/<id>/members

== System Workflows ==
* Creating a CG
*# Create the CG in the DB
*# Cast the consistency group creation to the scheduler
*## Choose a host for the CG
*# Cast the creation to the share service
*## Get the CG info from DB
*## Call the driver
*## Update database with info returned from driver (ex: CG status)

* Creating a CG from a cgsnapshot
*# Create a new CG in the DB
*# Schedule the CG create to the scheduler with the orig CG host/pool+share_type (depending on capability)
*# For each member of the snapshot, schedule a create share on the same host/pool as CG

* Creating a cg snapshot
*# Create cgsnapshot entity in DB
*# For all shares in CG, create cgsnapshot_member in DB
*# cast create_cgsnapshot to the driver

* Adding a share to a CG
** On share creation:
**# Cast to scheduler with the host of the CG
**#* if perpool capable: Goes to the same pool as CG
**#* if per backend: goes to a pool on the same host that matches a CG share type

* Deleting a share (in a CG)
** Not allowed if the CG has snapshots of the share ( check if the share_id is in the CGSnapshotMembers tables)

== Scheduler ==
manila/scheduler/filter_scheduler.py should now look for consistency_group_support as a capability.

consistency_group_support
default: None 
Values:
* None - No support for CGs
* host - shares in a CG must be on pool(s) on the same host that also match the CG share type
* pool - shares in a CG must live in the same pool as the CG

==DB==
* New Tables:
class ConsistencyGroup(BASE, ManilaBase):
"""Represents a consistencygroup."""
__tablename__ = 'consistency_groups'
id = Column(String(36), primary_key=True)

user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)
deleted = Column(String(36), default='False')

host = Column(String(255))
name = Column(String(255))
description = Column(String(255))
status = Column(String(255))
source_cgsnapshot_id = Column(String(36))

class ConsistencyGroupShareTypeMapping(BASE, ManilaBase):
"""Represents the share types in a consistency group"""
__tablename__ = 'consistency_group_share_type_mappings'
id = Column(Integer, primary_key=True)
deleted = Column(String(36), default='False')
consistency_group_id = Column(String(36),
ForeignKey('consistency_groups.id'),
nullable=False)
share_type_id = Column(String(36),
ForeignKey('share_types.id'),
nullable=False)
consistency_group = orm.relationship(
ConsistencyGroup,
backref="share_types",
foreign_keys=consistency_group_id,
primaryjoin=('and_('
'ConsistencyGroupShareTypeMapping.consistency_group_id == '
'ConsistencyGroups.id,'
'ConsistencyGroupShareTypeMapping.deleted == 0)')
)

class CGSnapshot(BASE, ManilaBase):
"""Represents a cgsnapshot."""
__tablename__ = 'cgsnapshots'
id = Column(String(36), primary_key=True)

consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'))
user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)

name = Column(String(255))
description = Column(String(255))
status = Column(String(255))

class CGSnapshotMembers(BASE, ManilaBase):
__tablename__ = 'cgsnapshot_members'
id = Column(String(36), primary_key=True)
cgsnapshot_id = Column(String(36), ForeignKey('cgsnapshots.id'))
share_id = Column(String(36), ForeignKey('shares.id'))
size = Column(Integer)
status = Column(String(255))
share_proto = Column(String(255))
share_type_id = Column(String(36), ForeignKey('share_types.id'),
nullable=True)
user_id = Column(String(255))
project_id = Column(String(255))
deleted = Column(String(36), default='False')

cgsnapshot = relationship(
ConsistencyGroup,
backref="cgsnapshot_members",
foreign_keys=cgsnapshot_id,
primaryjoin='CGSnapshotMembers.cgsnapshot_id == CGSnapshot.id')

* New fields in Share
consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'), nullable=True)
consistency_group = relationship(
ConsistencyGroup,
backref="shares",
foreign_keys=consistency_group_id,
primaryjoin='Share.consistency_group_id == ConsistencyGroup.id')

source_cgsnapshot_member_id = Column(String(36), nullable=True)

==API==
Schemas: https://wiki.openstack.org/wiki/Manila/design/manila-liberty-consistency-groups/api-schema

'''GET''' /os-consistency-groups/
'''GET''' /os-consistency-groups/detail
'''POST''' /os-consistency-groups
'''GET''' /os-consistency-groups/<id>
'''DELETE''' os-consistency-groups/<id>
'''PUT''' /os-consistency-groups/<id>

'''GET''' /os-cgsnapshots/
'''GET''' /os-cgsnapshots/detail
'''POST''' /os-cgsnapshots
'''PUT''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>/members
'''DELETE''' /os-cgsnapshots/<id>

Admin Actions
'''POST''' /os-consistency-groups/<id>/action # os-reset-status and os-force-delete
'''POST''' /os-cgsnapshots/<id>/action # os-reset-status and os-force-delete

Updates to Share resource
Adds consistency_group_id to POST/shares
Adds source_cgsnapshot_member_id
Add consistency_group_id query filter to /shares

==Policies==
policy.json - All consistency group policies should default to the default policy
"consistency_group:create" : "",
"consistency_group:delete": "",
"consistency_group:update": "",
"consistency_group:get": "",
"consistency_group:get_all": "",

"consistency_group:create_cgsnapshot" : "",
"consistency_group:delete_cgsnapshot": "",
"consistency_group:get_cgsnapshot": "",
"consistency_group:get_all_cgsnapshots": "",

==Driver API==

update_share_stats should now return 'consistency_group_support' pool/host/None.

def create_consistency_group(self, context, cg_dict, share_server=None):
"""Create a consistency group.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_consistency_group_from_cgsnapshot(self, context, cg_dict,
cgsnapshot_dict, share_server=None):
"""Create a consistency group from a cgsnapshot.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'shares': [<models.Share>], # The new shares being created
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:param cgsnapshot_dict:
EXAMPLE:
{'status': 'available',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': '0',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'consistency_group_id': '4b04fdc3-00b9-4909-ba1a-06e9b3f88b67',
'cgsnapshot_members': [
{'status': 'available',
'share_type_id': '1a9ed31e-ee70-483d-93ba-89690e028d7f',
'share_id': 'e14b5174-e534-4f35-bc4f-fe81c1575d6f',
'user_id': 'a0314a441ca842019b0952224aa39192',
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share': <models.Share>,
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share_proto': 'NFS',
'project_id': '13c0be6290934bd98596cfa004650049',
'cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'deleted_at': None,
'id': '6813e06b-a8f5-4784-b17d-f3e91afa370e',
'size': 1
}
],
'deleted_at': None,
'id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.

share_update_list - a list of dictionaries containing dicts for
every share created in the CG. Any share dicts should at a minimum
contain the 'id' key and 'export_locations'. Export locations
should be in the same format as returned by a share_create. This
list may be empty or None.
EXAMPLE:
[
{'id': 'uuid', 'export_locations': ['export_path']}
]
"""
raise NotImplementedError()

def delete_consistency_group(self, context, cg_dict, share_server=None):
"""Delete a consistency group

:param context:
:param cg_dict:
EXAMPLE:
:return: cg_model_update
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_cgsnapshot(self, context, snap_dict, share_server=None):
"""Create a consistency group snapshot.

:param context:
:param snap_dict:
EXAMPLE:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.

member_update_list - a list of dictionaries containing for every
member of the cgsnapshot. Each dict should contains values to be
updated for teh CGSnapshotMember in the database. This list may be
empty or None.
"""
raise NotImplementedError()

def delete_cgsnapshot(self, context, snap_dict, share_server=None):
"""Delete a consistency group snapshot

:param context:
:param snap_dict:
EXAMPLE:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.
"""
raise NotImplementedError()

== Manila Client ==
* Do we want to alias the commands so that there are Cinder consistent commands and the improved names? (I.E cg-create = consisgroup-create) Perhaps we alias the cinderclient commands to be more like this?
cg-snapshot-create Creates a cgsnapshot.
cg-snapshot-delete Removes one or more cgsnapshots.
cg-snapshot-list Lists all cgsnapshots.
cg-snapshot-show Shows cgsnapshot details.

cg-create Creates a consistency group (--cgsnapshot to create from existing cg snapshot)
cg-delete Removes one or more consistency groups.
cg-list Lists all consistencygroups.
cg-show Shows details of a consistency group.
cg-update Updates a consistencygroup.

create --consistency-group Creates a share and puts it into consistency group

= Outstanding Questions / FAQ =
# How do we avoid confusion if no backends in a deployment support CGs?
#* It appears that Cinder has the policies default to 'nobody' but this seems odd, is there a better way?
#** (ameade) I can't think of a way to have devstack set the policies automatically either.
#** (ameade) we have chosen to go with an os-consistency-groups extension for now
# Do we need a way to fake CGs with the generic driver?
#* <strike> NO, but it will not be a core feature if it cannot be supported in the generic driver.</strike>
#* Ben says that we should fake it in the generic driver so it can be tested by the gate.
# Do we need a safeguard for deleting shares that are in a CG? (I.E. protected shares?) Are shares in a CG more valuable than normal shares?
# Should we overload the Snapshots table to include snapshots in a CGSnapshot to avoid adding a new db table? Or continue with having a new table?
# Why not have the cg delete all of the shares in it when deleting the cg?
#* This could cause odd state to occur if the backend fails when deleting shares and it will not be obvious where it failed. This is also extra complexity in the implementation that could instead be hidden by a client.

= Notes on Cinder Impl =

* A consistency group may be specified on volume create
* When a consistency group is created it is scheduled to a pool and all volume created in the CG will go to that pol

Cinder cli:
cgsnapshot-create Creates a cgsnapshot and snapshots of every volume in the CG that show up in snapshot-list.
cgsnapshot-delete Removes one or more cgsnapshots.
cgsnapshot-list Lists all cgsnapshots.
cgsnapshot-show Shows cgsnapshot details. <-- /cgsnapshots/detail?all_tenants=1&name=blah
consisgroup-create Creates a consistency group.
consisgroup-create-from-src Creates a consistency group from a cgsnapshot filled with volumes from the snapshots in the cg.
consisgroup-delete Removes one or more consistency groups.
consisgroup-list Lists all consistencygroups.
consisgroup-show Shows details of a consistency group. <-- /consistencygroups/detail?all_tenants=1&name=blah
consisgroup-update Updates a consistencygroup.

create --consisgroup-id

Cinder API:
'''GET''' /consistencygroups/detail
'''POST''' /consistencygroups
{'consistencygroup':{
'name': <string>,
'description': <string>,
'volume_types': <string>,
'availability_zone': <string>,
}}
'''POST''' /consistencygroups/create_from_src
{'consistencygroup-from-src':{
'name': <string>,
'description': <string>,
'cgsnapshot_id': <string>,
}}
'''GET''' /consistencygroups/<id>
'''POST''' consistencygroups/<id>/delete
{"consistencygroup": {"force": false}}
'''PUT''' /consistencygroups/<id>
{'consistencygroup':{
'name': <string>,
'description': <string>,
'add_volumes': <string>,
'remove_volumes': <string>,
}}

'''GET''' /cgsnapshots/detail
'''GET''' /cgsnapshots/<id>
'''POST''' /cgsnapshots
{'cgsnapshot':{
'consistencygroup_id': <string>,
}}
'''DELETE''' /cgsnapshots/<id>

Adds consistencygroup_id to POST/volumes

Xings commit message:
1) Create a CG, specifying all volume types that can be supported by this
CG. The scheduler chooses a backend that supports all specified volume types.
The CG will be empty when it is first created. Backend needs to report
consistencygroup_support = True. Volume type can have the following in
extra specs: {'capabilities:consistencygroup_support': '<is> True'}.
If consistencygroup_support is not in volume type extra specs, it will be
added to filter_properties by the scheduler to make sure that the scheduler
will select the backend which reports consistency group support capability.

Create CG CLI:
cinder consisgroup-create --volume-type type1,type2 mycg1

This will add a CG entry in the new consistencygroups table.

2) After the CG is created, create a new volume and add to the CG.
Repeat until all volumes are created for the CG.

Create volume CLI (with CG):
cinder create --volume-type type1 --consisgroup-id <CG uuid> 10

This will add a consistencygroup_id foreign key in the new volume
entry in the db.

3) Create a snapshot of the CG (cgsnapshot).

Create cgsnapshot CLI:
cinder cgsnapshot-create <CG uuid>

This will add a cgsnapshot entry in the new cgsnapshots table, create
snapshot for each volume in the CG, and add a cgsnapshot_id foreign key
in each newly created snapshot entry in the db.

===Questions===
# Why does policy.json default to 'nobody' being able to perform CG commands?
# Why are volume types specified when creating a CG?
#* This is because all volumes within a CG need to be on the same backend in order for the backend to honor the CG. This is important to note in the docs so that conflicting volume-types are not specified.
# What occurs when a volume on a different backend is added to a CG via the API?
#* Currently the API ensures that the volume volume_type matches a supported volume_type of the CG and that the CG and volume are on the same host (but can be on different pools).
# Is a volume created with a CG going to end up on the same host?
#* Yes, if a volume has a CG it will be cast to the same pool as the CG.
#* What if the pool is full but there are other available pools on the same backend? Should this fail?
# What happens if a driver does not support create_from_src?
#* The CG goes to ERROR a and volumes are created from the snapshot and also in ERROR status
#* A volume in a consistency group cannot be deleted and a CG with a volume cannot be deleted, therefore a volume must be removed from the CG before it can be deleted. But what if you don't support modifying a CG?
# Why do we need a force delete for CGs?
# What do I do if I do not want to see Snapshots in a CG when doing snapshot-list?
#* I suppose API filtering could allow for someone to not see all of the snapshots if they do not want to.

=== Cinder Bugs to file===
# What happens when you force delete a CG?
#* The volumes remain but still think they are in a CG, which means you still cannot delete the volume
# consisgroup-create-from-src says cgsnapshot is and optional param in cinderclient
# Creating a volume in a CG may not end up in the cg. The only check that is performed is that the volume-type of the volume is supported by the CG but volume-types are not 1-to-1 with backends. There could be backend A and B that both match with volume type 1. A cg is created with volume type 1 and ends up on backend A. Then a volume is created with the CG specified but ends up on backend B

=== Main impl ===
299b99437c2de84222fd585f06c2d05ca213225b ConsistencyGroup: Return 400 instead of 500 for invalid body
adb4c80be82caacad83f1366a4b34e5653fd5dab Create Consistency Group from CG Snapshot API
1a62a6e60fda73bf31256fbf684dc03bb6cf0038 Modify Consistency Group API
fc7a4384be2ccf8ff24f6c0f72c681ad9133801a Add tests for consistency groups DB migration
9082273305b0b9c117eb677a0e34c2ffde4e66f0 Volume types need to be specified when creating CG
764c5ec749821d36bb0215dc6002d3caea95d3b1 Delete consistency group failed
cf961f83ac323dfad1fa5e227d1e502a17529ecc Consistency Groups

=== Ex driver impl ===
8d5b795b37ed021c2639066689645f8aa0b1012f PureISCSIDriver consistency group updates.
39c1a8e08dc854e22ada315a46c20c99df2facf8 Add support to PureISCSIDriver for Consistency Groups
92a817708a3938b1b734d2caaa206b310996d8d0 EMC VMAX driver Kilo update

Manila/design/manila-liberty-consistency-groups

2015-08-13T13:27:15Z

Alex Meade: /* Manila vs Cinder */

= Consistency Groups =
This document describes the design choices and steps for the consistency group feature in core Manila.

Consistency groups are a mechanism to allow multiple filesystem shares to be guaranteed snapshots are able to be created at the exact same point in time. For example, a database may have its tables, logs, and configuration on separate volumes. If we wanted to restore the database from a previous point in time, it would only make sense to restore the logs, tables, and configuration together from the exact same point in time.

= Manila Core Changes =
The core manila changes are very similar to what has been done in Cinder. At a high level, changes in the DB, share service, drivers, scheduler service and api service will need to be changed.

Phase 1:
* Create/Delete CGs
* Create a share within a CG
* Delete a share that is in a CG
* Snapshot entire CGs
* Create CG from a CGSnapshot

Potential future features:
* Add/remove shares in a CG
* CG migration
* Replication of a CG

==Manila vs Cinder==
* Different CLI syntax
* Restful api tweaks
** /os-consistency-groups and /os-cgsnapshots are implemented as extensions and therefore entirely optional APIs
** Change /create_from_src to just be a cgsnapshot_id in the body on POST /os-consistency-groups
** Deletes to use the DELETE HTTP verb, os-force_delete is supported via the admin actions extension (/action)
* At the Driver API, cgsnapshot objects and cg objects (with all the information about what volumes are in the CG) need to be passed to the driver instead of just the cg/cgsnapshot name. This is important for backends that aren't managing a CG construct such as NetApp cDOT as they need to do things such as loop through the snapshots in a cg to delete them.
* The default policy in policy.json for consistency group operations is the default policy versus in Cinder is it 'nobody'
* Unlike Cinder, snapshots in a CGsnapshot are not the same as a normal snapshot. A CGsnapshot is treated as a single unit instead of a collection of snapshots.
* consistency_group_support scheduler capability has multiple values instead of boolean. This is to accommodate backends that require all volumes in a CG to be on the same pool vs. just the same host. It also allows for adding additional values in the future if a backend can support some other paradigm like a CG spanning multiple backends.

== User Workflows ==
* Snapshot multiple shares
*# Create a Consistency Group
*# Create share(s) with a CG specified
*# Create a CGsnapshot

* Create a CG
*# POST to /os-consistency-group
*#* specifiy share_types or the default share type will be used

* Copy a consistency group
*# Create a CGsnapshot
*# Create a CG with the CGSnapshot id

* Delete a CG (and all of its shares)
*# Delete all CG snapshots of the CG
*# Delete all shares in the CG
*# Delete the CG (Only empty CGs can be deleted)

* List shares in a given CG
*# GET /shares?consistencygroup_id=<cg_id>

* List the shares that were captured in a given CGSnapshot
*# GET /cgsnapshots/<id>/members

== System Workflows ==
* Creating a CG
*# Create the CG in the DB
*# Cast the consistency group creation to the scheduler
*## Choose a host for the CG
*# Cast the creation to the share service
*## Get the CG info from DB
*## Call the driver
*## Update database with info returned from driver (ex: CG status)

* Creating a CG from a cgsnapshot
*# Create a new CG in the DB
*# Schedule the CG create to the scheduler with the orig CG host/pool+share_type (depending on capability)
*# For each member of the snapshot, schedule a create share on the same host/pool as CG

* Creating a cg snapshot
*# Create cgsnapshot entity in DB
*# For all shares in CG, create cgsnapshot_member in DB
*# cast create_cgsnapshot to the driver

* Adding a share to a CG
** On share creation:
**# Cast to scheduler with the host of the CG
**#* if perpool capable: Goes to the same pool as CG
**#* if per backend: goes to a pool on the same host that matches a CG share type

* Deleting a share (in a CG)
** Not allowed if the CG has snapshots of the share ( check if the share_id is in the CGSnapshotMembers tables)

== Scheduler ==
manila/scheduler/filter_scheduler.py should now look for consistency_group_support as a capability.

consistency_group_support
default: None 
Values:
* None - No support for CGs
* host - shares in a CG must be on pool(s) on the same host that also match the CG share type
* pool - shares in a CG must live in the same pool as the CG

==DB==
* New Tables:
class ConsistencyGroup(BASE, ManilaBase):
"""Represents a consistencygroup."""
__tablename__ = 'consistency_groups'
id = Column(String(36), primary_key=True)

user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)
deleted = Column(String(36), default='False')

host = Column(String(255))
name = Column(String(255))
description = Column(String(255))
status = Column(String(255))
source_cgsnapshot_id = Column(String(36))

class ConsistencyGroupShareTypeMapping(BASE, ManilaBase):
"""Represents the share types in a consistency group"""
__tablename__ = 'consistency_group_share_type_mappings'
id = Column(Integer, primary_key=True)
deleted = Column(String(36), default='False')
consistency_group_id = Column(String(36),
ForeignKey('consistency_groups.id'),
nullable=False)
share_type_id = Column(String(36),
ForeignKey('share_types.id'),
nullable=False)
consistency_group = orm.relationship(
ConsistencyGroup,
backref="share_types",
foreign_keys=consistency_group_id,
primaryjoin=('and_('
'ConsistencyGroupShareTypeMapping.consistency_group_id == '
'ConsistencyGroups.id,'
'ConsistencyGroupShareTypeMapping.deleted == 0)')
)

class CGSnapshot(BASE, ManilaBase):
"""Represents a cgsnapshot."""
__tablename__ = 'cgsnapshots'
id = Column(String(36), primary_key=True)

consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'))
user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)

name = Column(String(255))
description = Column(String(255))
status = Column(String(255))

class CGSnapshotMembers(BASE, ManilaBase):
__tablename__ = 'cgsnapshot_members'
id = Column(String(36), primary_key=True)
cgsnapshot_id = Column(String(36), ForeignKey('cgsnapshots.id'))
share_id = Column(String(36), ForeignKey('shares.id'))
size = Column(Integer)
status = Column(String(255))
share_proto = Column(String(255))
share_type_id = Column(String(36), ForeignKey('share_types.id'),
nullable=True)
user_id = Column(String(255))
project_id = Column(String(255))
deleted = Column(String(36), default='False')

cgsnapshot = relationship(
ConsistencyGroup,
backref="cgsnapshot_members",
foreign_keys=cgsnapshot_id,
primaryjoin='CGSnapshotMembers.cgsnapshot_id == CGSnapshot.id')

* New fields in Share
consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'), nullable=True)
consistency_group = relationship(
ConsistencyGroup,
backref="shares",
foreign_keys=consistency_group_id,
primaryjoin='Share.consistency_group_id == ConsistencyGroup.id')

source_cgsnapshot_member_id = Column(String(36), nullable=True)

==API==
Schemas: https://wiki.openstack.org/wiki/Manila/design/manila-liberty-consistency-groups/api-schema

'''GET''' /os-consistency-groups/
'''GET''' /os-consistency-groups/detail
'''POST''' /os-consistency-groups
'''GET''' /os-consistency-groups/<id>
'''DELETE''' os-consistency-groups/<id>
'''PUT''' /os-consistency-groups/<id>

'''GET''' /os-cgsnapshots/
'''GET''' /os-cgsnapshots/detail
'''POST''' /os-cgsnapshots
'''PUT''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>/members
'''DELETE''' /os-cgsnapshots/<id>

Admin Actions
'''POST''' /os-consistency-groups/<id>/action # os-reset-status and os-force-delete
'''POST''' /os-cgsnapshots/<id>/action # os-reset-status and os-force-delete

Updates to Share resource
Adds consistency_group_id to POST/shares
Adds source_cgsnapshot_member_id
Add consistency_group_id query filter to /shares

==Policies==
policy.json - All consistency group policies should default to the default policy
"consistency_group:create" : "",
"consistency_group:delete": "",
"consistency_group:update": "",
"consistency_group:get": "",
"consistency_group:get_all": "",

"consistency_group:create_cgsnapshot" : "",
"consistency_group:delete_cgsnapshot": "",
"consistency_group:get_cgsnapshot": "",
"consistency_group:get_all_cgsnapshots": "",

==Driver API==

update_share_stats should now return 'consistency_group_support' pool/host/None.

def create_consistency_group(self, context, cg_dict, share_server=None):
"""Create a consistency group.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_consistency_group_from_cgsnapshot(self, context, cg_dict,
cgsnapshot_dict, share_server=None):
"""Create a consistency group from a cgsnapshot.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'shares': [<models.Share>], # The new shares being created
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:param cgsnapshot_dict:
EXAMPLE:
{'status': 'available',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': '0',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'consistency_group_id': '4b04fdc3-00b9-4909-ba1a-06e9b3f88b67',
'cgsnapshot_members': [
{'status': 'available',
'share_type_id': '1a9ed31e-ee70-483d-93ba-89690e028d7f',
'share_id': 'e14b5174-e534-4f35-bc4f-fe81c1575d6f',
'user_id': 'a0314a441ca842019b0952224aa39192',
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share': <models.Share>,
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share_proto': 'NFS',
'project_id': '13c0be6290934bd98596cfa004650049',
'cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'deleted_at': None,
'id': '6813e06b-a8f5-4784-b17d-f3e91afa370e',
'size': 1
}
],
'deleted_at': None,
'id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.

share_update_list - a list of dictionaries containing dicts for
every share created in the CG. Any share dicts should at a minimum
contain the 'id' key and 'export_locations'. Export locations
should be in the same format as returned by a share_create. This
list may be empty or None.
EXAMPLE:
[
{'id': 'uuid', 'export_locations': ['export_path']}
]
"""
raise NotImplementedError()

def delete_consistency_group(self, context, cg_dict, share_server=None):
"""Delete a consistency group

:param context:
:param cg_dict:
EXAMPLE:
:return: cg_model_update
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_cgsnapshot(self, context, snap_dict, share_server=None):
"""Create a consistency group snapshot.

:param context:
:param snap_dict:
EXAMPLE:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.

member_update_list - a list of dictionaries containing for every
member of the cgsnapshot. Each dict should contains values to be
updated for teh CGSnapshotMember in the database. This list may be
empty or None.
"""
raise NotImplementedError()

def delete_cgsnapshot(self, context, snap_dict, share_server=None):
"""Delete a consistency group snapshot

:param context:
:param snap_dict:
EXAMPLE:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.
"""
raise NotImplementedError()

== Manila Client ==
* Do we want to alias the commands so that there are Cinder consistent commands and the improved names? (I.E cg-create = consisgroup-create) Perhaps we alias the cinderclient commands to be more like this?
cg-snapshot-create Creates a cgsnapshot.
cg-snapshot-delete Removes one or more cgsnapshots.
cg-snapshot-list Lists all cgsnapshots.
cg-snapshot-show Shows cgsnapshot details.

cg-create Creates a consistency group (--cgsnapshot to create from existing cg snapshot)
cg-delete Removes one or more consistency groups.
cg-list Lists all consistencygroups.
cg-show Shows details of a consistency group.
cg-update Updates a consistencygroup.

create --consistency-group Creates a share and puts it into consistency group

= Outstanding Questions / FAQ =
# How do we avoid confusion if no backends in a deployment support CGs?
#* It appears that Cinder has the policies default to 'nobody' but this seems odd, is there a better way?
#** (ameade) I can't think of a way to have devstack set the policies automatically either.
#** (ameade) we have chosen to go with an os-consistency-groups extension for now
# Do we need a way to fake CGs with the generic driver?
#* <strike> NO, but it will not be a core feature if it cannot be supported in the generic driver.</strike>
#* Ben says that we should fake it in the generic driver so it can be tested by the gate.
# Do we need a safeguard for deleting shares that are in a CG? (I.E. protected shares?) Are shares in a CG more valuable than normal shares?
# Should we overload the Snapshots table to include snapshots in a CGSnapshot to avoid adding a new db table? Or continue with having a new table?
# Why not have the cg delete all of the shares in it when deleting the cg?
#* This could cause odd state to occur if the backend fails when deleting shares and it will not be obvious where it failed. This is also extra complexity in the implementation that could instead be hidden by a client.

= Notes on Cinder Impl =

* A consistency group may be specified on volume create
* When a consistency group is created it is scheduled to a pool and all volume created in the CG will go to that pol

Cinder cli:
cgsnapshot-create Creates a cgsnapshot and snapshots of every volume in the CG that show up in snapshot-list.
cgsnapshot-delete Removes one or more cgsnapshots.
cgsnapshot-list Lists all cgsnapshots.
cgsnapshot-show Shows cgsnapshot details. <-- /cgsnapshots/detail?all_tenants=1&name=blah
consisgroup-create Creates a consistency group.
consisgroup-create-from-src Creates a consistency group from a cgsnapshot filled with volumes from the snapshots in the cg.
consisgroup-delete Removes one or more consistency groups.
consisgroup-list Lists all consistencygroups.
consisgroup-show Shows details of a consistency group. <-- /consistencygroups/detail?all_tenants=1&name=blah
consisgroup-update Updates a consistencygroup.

create --consisgroup-id

Cinder API:
'''GET''' /consistencygroups/detail
'''POST''' /consistencygroups
{'consistencygroup':{
'name': <string>,
'description': <string>,
'volume_types': <string>,
'availability_zone': <string>,
}}
'''POST''' /consistencygroups/create_from_src
{'consistencygroup-from-src':{
'name': <string>,
'description': <string>,
'cgsnapshot_id': <string>,
}}
'''GET''' /consistencygroups/<id>
'''POST''' consistencygroups/<id>/delete
{"consistencygroup": {"force": false}}
'''PUT''' /consistencygroups/<id>
{'consistencygroup':{
'name': <string>,
'description': <string>,
'add_volumes': <string>,
'remove_volumes': <string>,
}}

'''GET''' /cgsnapshots/detail
'''GET''' /cgsnapshots/<id>
'''POST''' /cgsnapshots
{'cgsnapshot':{
'consistencygroup_id': <string>,
}}
'''DELETE''' /cgsnapshots/<id>

Adds consistencygroup_id to POST/volumes

Xings commit message:
1) Create a CG, specifying all volume types that can be supported by this
CG. The scheduler chooses a backend that supports all specified volume types.
The CG will be empty when it is first created. Backend needs to report
consistencygroup_support = True. Volume type can have the following in
extra specs: {'capabilities:consistencygroup_support': '<is> True'}.
If consistencygroup_support is not in volume type extra specs, it will be
added to filter_properties by the scheduler to make sure that the scheduler
will select the backend which reports consistency group support capability.

Create CG CLI:
cinder consisgroup-create --volume-type type1,type2 mycg1

This will add a CG entry in the new consistencygroups table.

2) After the CG is created, create a new volume and add to the CG.
Repeat until all volumes are created for the CG.

Create volume CLI (with CG):
cinder create --volume-type type1 --consisgroup-id <CG uuid> 10

This will add a consistencygroup_id foreign key in the new volume
entry in the db.

3) Create a snapshot of the CG (cgsnapshot).

Create cgsnapshot CLI:
cinder cgsnapshot-create <CG uuid>

This will add a cgsnapshot entry in the new cgsnapshots table, create
snapshot for each volume in the CG, and add a cgsnapshot_id foreign key
in each newly created snapshot entry in the db.

===Questions===
# Why does policy.json default to 'nobody' being able to perform CG commands?
# Why are volume types specified when creating a CG?
#* This is because all volumes within a CG need to be on the same backend in order for the backend to honor the CG. This is important to note in the docs so that conflicting volume-types are not specified.
# What occurs when a volume on a different backend is added to a CG via the API?
#* Currently the API ensures that the volume volume_type matches a supported volume_type of the CG and that the CG and volume are on the same host (but can be on different pools).
# Is a volume created with a CG going to end up on the same host?
#* Yes, if a volume has a CG it will be cast to the same pool as the CG.
#* What if the pool is full but there are other available pools on the same backend? Should this fail?
# What happens if a driver does not support create_from_src?
#* The CG goes to ERROR a and volumes are created from the snapshot and also in ERROR status
#* A volume in a consistency group cannot be deleted and a CG with a volume cannot be deleted, therefore a volume must be removed from the CG before it can be deleted. But what if you don't support modifying a CG?
# Why do we need a force delete for CGs?
# What do I do if I do not want to see Snapshots in a CG when doing snapshot-list?
#* I suppose API filtering could allow for someone to not see all of the snapshots if they do not want to.

=== Cinder Bugs to file===
# What happens when you force delete a CG?
#* The volumes remain but still think they are in a CG, which means you still cannot delete the volume
# consisgroup-create-from-src says cgsnapshot is and optional param in cinderclient
# Creating a volume in a CG may not end up in the cg. The only check that is performed is that the volume-type of the volume is supported by the CG but volume-types are not 1-to-1 with backends. There could be backend A and B that both match with volume type 1. A cg is created with volume type 1 and ends up on backend A. Then a volume is created with the CG specified but ends up on backend B

=== Main impl ===
299b99437c2de84222fd585f06c2d05ca213225b ConsistencyGroup: Return 400 instead of 500 for invalid body
adb4c80be82caacad83f1366a4b34e5653fd5dab Create Consistency Group from CG Snapshot API
1a62a6e60fda73bf31256fbf684dc03bb6cf0038 Modify Consistency Group API
fc7a4384be2ccf8ff24f6c0f72c681ad9133801a Add tests for consistency groups DB migration
9082273305b0b9c117eb677a0e34c2ffde4e66f0 Volume types need to be specified when creating CG
764c5ec749821d36bb0215dc6002d3caea95d3b1 Delete consistency group failed
cf961f83ac323dfad1fa5e227d1e502a17529ecc Consistency Groups

=== Ex driver impl ===
8d5b795b37ed021c2639066689645f8aa0b1012f PureISCSIDriver consistency group updates.
39c1a8e08dc854e22ada315a46c20c99df2facf8 Add support to PureISCSIDriver for Consistency Groups
92a817708a3938b1b734d2caaa206b310996d8d0 EMC VMAX driver Kilo update

Manila/design/manila-liberty-consistency-groups

2015-08-13T13:24:48Z

Alex Meade: /* API */

= Consistency Groups =
This document describes the design choices and steps for the consistency group feature in core Manila.

Consistency groups are a mechanism to allow multiple filesystem shares to be guaranteed snapshots are able to be created at the exact same point in time. For example, a database may have its tables, logs, and configuration on separate volumes. If we wanted to restore the database from a previous point in time, it would only make sense to restore the logs, tables, and configuration together from the exact same point in time.

= Manila Core Changes =
The core manila changes are very similar to what has been done in Cinder. At a high level, changes in the DB, share service, drivers, scheduler service and api service will need to be changed.

Phase 1:
* Create/Delete CGs
* Create a share within a CG
* Delete a share that is in a CG
* Snapshot entire CGs
* Create CG from a CGSnapshot

Potential future features:
* Add/remove shares in a CG
* CG migration
* Replication of a CG

==Manila vs Cinder==
* Different CLI syntax
* Restful api tweaks
** Change /create_from_src to just be a cgsnapshot_id in the body on POST /shares
** Deletes to use the DELETE HTTP verb, os-force_delete is only supported via the admin actions extension /action
* At the Driver API, cgsnapshot objects and cg objects (with all the information about what volumes are in the CG) need to be passed to the driver instead of just the cg/cgsnapshot name. This is important for backends that aren't managing a CG construct such as NetApp cDOT as they need to do things such as loop through the snapshots in a cg to delete them.
* The default policy in policy.json for consistency group operations is the default policy versus in Cinder is it 'nobody'
* Unlike Cinder, snapshots in a CGsnapshot are not the same as a normal snapshot. A CGsnapshot is treated as a single unit instead of a collection of snapshots.
* consistency_group_support scheduler capability has multiple values instead of boolean. This is to accommodate backends that require all volumes in a CG to be on the same pool vs. just the same host. It also allows for adding additional values in the future if a backend can support some other paradigm like a CG spanning multiple backends.

== User Workflows ==
* Snapshot multiple shares
*# Create a Consistency Group
*# Create share(s) with a CG specified
*# Create a CGsnapshot

* Create a CG
*# POST to /os-consistency-group
*#* specifiy share_types or the default share type will be used

* Copy a consistency group
*# Create a CGsnapshot
*# Create a CG with the CGSnapshot id

* Delete a CG (and all of its shares)
*# Delete all CG snapshots of the CG
*# Delete all shares in the CG
*# Delete the CG (Only empty CGs can be deleted)

* List shares in a given CG
*# GET /shares?consistencygroup_id=<cg_id>

* List the shares that were captured in a given CGSnapshot
*# GET /cgsnapshots/<id>/members

== System Workflows ==
* Creating a CG
*# Create the CG in the DB
*# Cast the consistency group creation to the scheduler
*## Choose a host for the CG
*# Cast the creation to the share service
*## Get the CG info from DB
*## Call the driver
*## Update database with info returned from driver (ex: CG status)

* Creating a CG from a cgsnapshot
*# Create a new CG in the DB
*# Schedule the CG create to the scheduler with the orig CG host/pool+share_type (depending on capability)
*# For each member of the snapshot, schedule a create share on the same host/pool as CG

* Creating a cg snapshot
*# Create cgsnapshot entity in DB
*# For all shares in CG, create cgsnapshot_member in DB
*# cast create_cgsnapshot to the driver

* Adding a share to a CG
** On share creation:
**# Cast to scheduler with the host of the CG
**#* if perpool capable: Goes to the same pool as CG
**#* if per backend: goes to a pool on the same host that matches a CG share type

* Deleting a share (in a CG)
** Not allowed if the CG has snapshots of the share ( check if the share_id is in the CGSnapshotMembers tables)

== Scheduler ==
manila/scheduler/filter_scheduler.py should now look for consistency_group_support as a capability.

consistency_group_support
default: None 
Values:
* None - No support for CGs
* host - shares in a CG must be on pool(s) on the same host that also match the CG share type
* pool - shares in a CG must live in the same pool as the CG

==DB==
* New Tables:
class ConsistencyGroup(BASE, ManilaBase):
"""Represents a consistencygroup."""
__tablename__ = 'consistency_groups'
id = Column(String(36), primary_key=True)

user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)
deleted = Column(String(36), default='False')

host = Column(String(255))
name = Column(String(255))
description = Column(String(255))
status = Column(String(255))
source_cgsnapshot_id = Column(String(36))

class ConsistencyGroupShareTypeMapping(BASE, ManilaBase):
"""Represents the share types in a consistency group"""
__tablename__ = 'consistency_group_share_type_mappings'
id = Column(Integer, primary_key=True)
deleted = Column(String(36), default='False')
consistency_group_id = Column(String(36),
ForeignKey('consistency_groups.id'),
nullable=False)
share_type_id = Column(String(36),
ForeignKey('share_types.id'),
nullable=False)
consistency_group = orm.relationship(
ConsistencyGroup,
backref="share_types",
foreign_keys=consistency_group_id,
primaryjoin=('and_('
'ConsistencyGroupShareTypeMapping.consistency_group_id == '
'ConsistencyGroups.id,'
'ConsistencyGroupShareTypeMapping.deleted == 0)')
)

class CGSnapshot(BASE, ManilaBase):
"""Represents a cgsnapshot."""
__tablename__ = 'cgsnapshots'
id = Column(String(36), primary_key=True)

consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'))
user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)

name = Column(String(255))
description = Column(String(255))
status = Column(String(255))

class CGSnapshotMembers(BASE, ManilaBase):
__tablename__ = 'cgsnapshot_members'
id = Column(String(36), primary_key=True)
cgsnapshot_id = Column(String(36), ForeignKey('cgsnapshots.id'))
share_id = Column(String(36), ForeignKey('shares.id'))
size = Column(Integer)
status = Column(String(255))
share_proto = Column(String(255))
share_type_id = Column(String(36), ForeignKey('share_types.id'),
nullable=True)
user_id = Column(String(255))
project_id = Column(String(255))
deleted = Column(String(36), default='False')

cgsnapshot = relationship(
ConsistencyGroup,
backref="cgsnapshot_members",
foreign_keys=cgsnapshot_id,
primaryjoin='CGSnapshotMembers.cgsnapshot_id == CGSnapshot.id')

* New fields in Share
consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'), nullable=True)
consistency_group = relationship(
ConsistencyGroup,
backref="shares",
foreign_keys=consistency_group_id,
primaryjoin='Share.consistency_group_id == ConsistencyGroup.id')

source_cgsnapshot_member_id = Column(String(36), nullable=True)

==API==
Schemas: https://wiki.openstack.org/wiki/Manila/design/manila-liberty-consistency-groups/api-schema

'''GET''' /os-consistency-groups/
'''GET''' /os-consistency-groups/detail
'''POST''' /os-consistency-groups
'''GET''' /os-consistency-groups/<id>
'''DELETE''' os-consistency-groups/<id>
'''PUT''' /os-consistency-groups/<id>

'''GET''' /os-cgsnapshots/
'''GET''' /os-cgsnapshots/detail
'''POST''' /os-cgsnapshots
'''PUT''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>/members
'''DELETE''' /os-cgsnapshots/<id>

Admin Actions
'''POST''' /os-consistency-groups/<id>/action # os-reset-status and os-force-delete
'''POST''' /os-cgsnapshots/<id>/action # os-reset-status and os-force-delete

Updates to Share resource
Adds consistency_group_id to POST/shares
Adds source_cgsnapshot_member_id
Add consistency_group_id query filter to /shares

==Policies==
policy.json - All consistency group policies should default to the default policy
"consistency_group:create" : "",
"consistency_group:delete": "",
"consistency_group:update": "",
"consistency_group:get": "",
"consistency_group:get_all": "",

"consistency_group:create_cgsnapshot" : "",
"consistency_group:delete_cgsnapshot": "",
"consistency_group:get_cgsnapshot": "",
"consistency_group:get_all_cgsnapshots": "",

==Driver API==

update_share_stats should now return 'consistency_group_support' pool/host/None.

def create_consistency_group(self, context, cg_dict, share_server=None):
"""Create a consistency group.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_consistency_group_from_cgsnapshot(self, context, cg_dict,
cgsnapshot_dict, share_server=None):
"""Create a consistency group from a cgsnapshot.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'shares': [<models.Share>], # The new shares being created
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:param cgsnapshot_dict:
EXAMPLE:
{'status': 'available',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': '0',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'consistency_group_id': '4b04fdc3-00b9-4909-ba1a-06e9b3f88b67',
'cgsnapshot_members': [
{'status': 'available',
'share_type_id': '1a9ed31e-ee70-483d-93ba-89690e028d7f',
'share_id': 'e14b5174-e534-4f35-bc4f-fe81c1575d6f',
'user_id': 'a0314a441ca842019b0952224aa39192',
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share': <models.Share>,
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share_proto': 'NFS',
'project_id': '13c0be6290934bd98596cfa004650049',
'cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'deleted_at': None,
'id': '6813e06b-a8f5-4784-b17d-f3e91afa370e',
'size': 1
}
],
'deleted_at': None,
'id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.

share_update_list - a list of dictionaries containing dicts for
every share created in the CG. Any share dicts should at a minimum
contain the 'id' key and 'export_locations'. Export locations
should be in the same format as returned by a share_create. This
list may be empty or None.
EXAMPLE:
[
{'id': 'uuid', 'export_locations': ['export_path']}
]
"""
raise NotImplementedError()

def delete_consistency_group(self, context, cg_dict, share_server=None):
"""Delete a consistency group

:param context:
:param cg_dict:
EXAMPLE:
:return: cg_model_update
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_cgsnapshot(self, context, snap_dict, share_server=None):
"""Create a consistency group snapshot.

:param context:
:param snap_dict:
EXAMPLE:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.

member_update_list - a list of dictionaries containing for every
member of the cgsnapshot. Each dict should contains values to be
updated for teh CGSnapshotMember in the database. This list may be
empty or None.
"""
raise NotImplementedError()

def delete_cgsnapshot(self, context, snap_dict, share_server=None):
"""Delete a consistency group snapshot

:param context:
:param snap_dict:
EXAMPLE:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.
"""
raise NotImplementedError()

== Manila Client ==
* Do we want to alias the commands so that there are Cinder consistent commands and the improved names? (I.E cg-create = consisgroup-create) Perhaps we alias the cinderclient commands to be more like this?
cg-snapshot-create Creates a cgsnapshot.
cg-snapshot-delete Removes one or more cgsnapshots.
cg-snapshot-list Lists all cgsnapshots.
cg-snapshot-show Shows cgsnapshot details.

cg-create Creates a consistency group (--cgsnapshot to create from existing cg snapshot)
cg-delete Removes one or more consistency groups.
cg-list Lists all consistencygroups.
cg-show Shows details of a consistency group.
cg-update Updates a consistencygroup.

create --consistency-group Creates a share and puts it into consistency group

= Outstanding Questions / FAQ =
# How do we avoid confusion if no backends in a deployment support CGs?
#* It appears that Cinder has the policies default to 'nobody' but this seems odd, is there a better way?
#** (ameade) I can't think of a way to have devstack set the policies automatically either.
#** (ameade) we have chosen to go with an os-consistency-groups extension for now
# Do we need a way to fake CGs with the generic driver?
#* <strike> NO, but it will not be a core feature if it cannot be supported in the generic driver.</strike>
#* Ben says that we should fake it in the generic driver so it can be tested by the gate.
# Do we need a safeguard for deleting shares that are in a CG? (I.E. protected shares?) Are shares in a CG more valuable than normal shares?
# Should we overload the Snapshots table to include snapshots in a CGSnapshot to avoid adding a new db table? Or continue with having a new table?
# Why not have the cg delete all of the shares in it when deleting the cg?
#* This could cause odd state to occur if the backend fails when deleting shares and it will not be obvious where it failed. This is also extra complexity in the implementation that could instead be hidden by a client.

= Notes on Cinder Impl =

* A consistency group may be specified on volume create
* When a consistency group is created it is scheduled to a pool and all volume created in the CG will go to that pol

Cinder cli:
cgsnapshot-create Creates a cgsnapshot and snapshots of every volume in the CG that show up in snapshot-list.
cgsnapshot-delete Removes one or more cgsnapshots.
cgsnapshot-list Lists all cgsnapshots.
cgsnapshot-show Shows cgsnapshot details. <-- /cgsnapshots/detail?all_tenants=1&name=blah
consisgroup-create Creates a consistency group.
consisgroup-create-from-src Creates a consistency group from a cgsnapshot filled with volumes from the snapshots in the cg.
consisgroup-delete Removes one or more consistency groups.
consisgroup-list Lists all consistencygroups.
consisgroup-show Shows details of a consistency group. <-- /consistencygroups/detail?all_tenants=1&name=blah
consisgroup-update Updates a consistencygroup.

create --consisgroup-id

Cinder API:
'''GET''' /consistencygroups/detail
'''POST''' /consistencygroups
{'consistencygroup':{
'name': <string>,
'description': <string>,
'volume_types': <string>,
'availability_zone': <string>,
}}
'''POST''' /consistencygroups/create_from_src
{'consistencygroup-from-src':{
'name': <string>,
'description': <string>,
'cgsnapshot_id': <string>,
}}
'''GET''' /consistencygroups/<id>
'''POST''' consistencygroups/<id>/delete
{"consistencygroup": {"force": false}}
'''PUT''' /consistencygroups/<id>
{'consistencygroup':{
'name': <string>,
'description': <string>,
'add_volumes': <string>,
'remove_volumes': <string>,
}}

'''GET''' /cgsnapshots/detail
'''GET''' /cgsnapshots/<id>
'''POST''' /cgsnapshots
{'cgsnapshot':{
'consistencygroup_id': <string>,
}}
'''DELETE''' /cgsnapshots/<id>

Adds consistencygroup_id to POST/volumes

Xings commit message:
1) Create a CG, specifying all volume types that can be supported by this
CG. The scheduler chooses a backend that supports all specified volume types.
The CG will be empty when it is first created. Backend needs to report
consistencygroup_support = True. Volume type can have the following in
extra specs: {'capabilities:consistencygroup_support': '<is> True'}.
If consistencygroup_support is not in volume type extra specs, it will be
added to filter_properties by the scheduler to make sure that the scheduler
will select the backend which reports consistency group support capability.

Create CG CLI:
cinder consisgroup-create --volume-type type1,type2 mycg1

This will add a CG entry in the new consistencygroups table.

2) After the CG is created, create a new volume and add to the CG.
Repeat until all volumes are created for the CG.

Create volume CLI (with CG):
cinder create --volume-type type1 --consisgroup-id <CG uuid> 10

This will add a consistencygroup_id foreign key in the new volume
entry in the db.

3) Create a snapshot of the CG (cgsnapshot).

Create cgsnapshot CLI:
cinder cgsnapshot-create <CG uuid>

This will add a cgsnapshot entry in the new cgsnapshots table, create
snapshot for each volume in the CG, and add a cgsnapshot_id foreign key
in each newly created snapshot entry in the db.

===Questions===
# Why does policy.json default to 'nobody' being able to perform CG commands?
# Why are volume types specified when creating a CG?
#* This is because all volumes within a CG need to be on the same backend in order for the backend to honor the CG. This is important to note in the docs so that conflicting volume-types are not specified.
# What occurs when a volume on a different backend is added to a CG via the API?
#* Currently the API ensures that the volume volume_type matches a supported volume_type of the CG and that the CG and volume are on the same host (but can be on different pools).
# Is a volume created with a CG going to end up on the same host?
#* Yes, if a volume has a CG it will be cast to the same pool as the CG.
#* What if the pool is full but there are other available pools on the same backend? Should this fail?
# What happens if a driver does not support create_from_src?
#* The CG goes to ERROR a and volumes are created from the snapshot and also in ERROR status
#* A volume in a consistency group cannot be deleted and a CG with a volume cannot be deleted, therefore a volume must be removed from the CG before it can be deleted. But what if you don't support modifying a CG?
# Why do we need a force delete for CGs?
# What do I do if I do not want to see Snapshots in a CG when doing snapshot-list?
#* I suppose API filtering could allow for someone to not see all of the snapshots if they do not want to.

=== Cinder Bugs to file===
# What happens when you force delete a CG?
#* The volumes remain but still think they are in a CG, which means you still cannot delete the volume
# consisgroup-create-from-src says cgsnapshot is and optional param in cinderclient
# Creating a volume in a CG may not end up in the cg. The only check that is performed is that the volume-type of the volume is supported by the CG but volume-types are not 1-to-1 with backends. There could be backend A and B that both match with volume type 1. A cg is created with volume type 1 and ends up on backend A. Then a volume is created with the CG specified but ends up on backend B

=== Main impl ===
299b99437c2de84222fd585f06c2d05ca213225b ConsistencyGroup: Return 400 instead of 500 for invalid body
adb4c80be82caacad83f1366a4b34e5653fd5dab Create Consistency Group from CG Snapshot API
1a62a6e60fda73bf31256fbf684dc03bb6cf0038 Modify Consistency Group API
fc7a4384be2ccf8ff24f6c0f72c681ad9133801a Add tests for consistency groups DB migration
9082273305b0b9c117eb677a0e34c2ffde4e66f0 Volume types need to be specified when creating CG
764c5ec749821d36bb0215dc6002d3caea95d3b1 Delete consistency group failed
cf961f83ac323dfad1fa5e227d1e502a17529ecc Consistency Groups

=== Ex driver impl ===
8d5b795b37ed021c2639066689645f8aa0b1012f PureISCSIDriver consistency group updates.
39c1a8e08dc854e22ada315a46c20c99df2facf8 Add support to PureISCSIDriver for Consistency Groups
92a817708a3938b1b734d2caaa206b310996d8d0 EMC VMAX driver Kilo update

Manila/design/manila-liberty-consistency-groups/api-schema

2015-08-13T13:23:00Z

Alex Meade: /* Resource schemas */

= Resource schemas=

== Simple CG ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer"
},
"name": {
"description": "A non-unique name for a CG",
"type": "string"
},
},
"required": ["id", "name"]
}

== Detailed CG ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "string",
'maxLength': 36,
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"share_network_id": {
"description": "The share network to provision the CG and all of it's shares into",
"type": "string",
'maxLength': 36,
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"created_at": {
"description": "The time at which the CG was created",
"type": "string"
},
"status": {
"description": "Status of the CG",
"type": "string",
"enum": ["creating", "available", "deleting", "error"]
},
"project_id": {
"description": "The owner of the CG",
"type": "string"
},
"host": {
"description": "The backend where the CG lives",
"type": "string"
},
"source_cgsnapshot_id": {
"description": "The uuid of the cgsnapshot from which this CG was created.",
"type": ['null', 'string'],
},
"share_types": {
"description": "The share types that shares in the CG are allowed to be a part of",
"type": "array",
"items": {
"type": "string"
},
},
},
"required": ["id", "name", "description", "created_at", "status", "project_id", "host"]
}

== Simple CG Snapshot ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CGsnapshot",
"type": "string"
},
},
"required": ["id", "name"]
}

== Detailed CG Snapshot ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CG snapshot",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG snapshot",
"type": "string"
},
"created_at": {
"description": "The time at which the CG snapshot was created",
"type": "string"
},
"status": {
"description": "Status of the CG snapshot",
"type": "string",
"enum": ["creating", "available", "deleting", "error"]
},
"project_id": {
"description": "The owner of the CG snapshot",
"type": "string"
},
"consistency_group_id": {
"description": "The uuid of the consistency group from which this CG snapshot was created.",
"type": ['null', 'string'],
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
},
"required": ["id", "name", "description", "created_at", "status", "project_id"]
}

== CG Snapshot member ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot Member",
"description": "A resource representing a member of a CGSnapshot",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"cgsnapshot_id": {
"description": "The unique identifier for the parent cg_snapshot",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"share_id": {
"description": "The unique identifier for the share this snapshot was created from.",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"share_type_id": {
"description": "The unique identifier for the share_type of the share this snapshot was created from.",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"size": {
"description": "The storage size of this snapshot",
"type": "integer",
},
"share_protocol": {
"description": "The share protocol of the parent share",
"type": "string",
},
},
}

= Request schemas =

== POST /os-consistency-groups ==
{ "consistency_group" {
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"source_cgsnapshot_id": {
"description": "The uuid of the cgsnapshot from which this CG was created. Cannot be supplied when 'share_types' is provided.",
"type": ['null', 'string'],
},
"share_types": {
"description": "The share types that shares in the CG are allowed to be a part of. Cannot be supplied when 'source_cgsnapshot_id' is provided.",
"type": "string"
},
},
"required": []
}
}

Response: {'consistency_group": detailed CG}

== GET /os-consistency-groups ==
Response:
{'consistency_groups": [Simple CG]}

== GET /os-consistency-groups/detail ==
Response:
{'consistency_groups": [Detailed CG]}

== GET /os-consistency-groups/<id> ==
Response: {'consistency_group": Detailed CG}

== PUT /os-consistency-groups/<id> ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"required": []
}

Response: {'consistency_group": Detailed CG}

== POST /os-cgsnapshots ==
{'cgsnapshot": {
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"consistency_group_id": {
"description": "The uuid of the CG from which to create this CG snapshot.",
"type": 'string',
},
},
"required": ['consistency_group_id']
}}
Response: {'cgsnapshot": detailed CGSnapshot}

== PUT /os-cgsnapshots/<id> ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"required": []
}

Response: {'cgsnapshot": Detailed CGSnapshot}

Manila/Meetings

2015-08-11T14:05:31Z

Alex Meade: /* Next meeting */

Manila/design/manila-liberty-consistency-groups

2015-08-10T23:15:43Z

Alex Meade: /* Manila vs Cinder */

= Consistency Groups =
This document describes the design choices and steps for the consistency group feature in core Manila.

Consistency groups are a mechanism to allow multiple filesystem shares to be guaranteed snapshots are able to be created at the exact same point in time. For example, a database may have its tables, logs, and configuration on separate volumes. If we wanted to restore the database from a previous point in time, it would only make sense to restore the logs, tables, and configuration together from the exact same point in time.

= Manila Core Changes =
The core manila changes are very similar to what has been done in Cinder. At a high level, changes in the DB, share service, drivers, scheduler service and api service will need to be changed.

Phase 1:
* Create/Delete CGs
* Create a share within a CG
* Delete a share that is in a CG
* Snapshot entire CGs
* Create CG from a CGSnapshot

Potential future features:
* Add/remove shares in a CG
* CG migration
* Replication of a CG

==Manila vs Cinder==
* Different CLI syntax
* Restful api tweaks
** Change /create_from_src to just be a cgsnapshot_id in the body on POST /shares
** Deletes to use the DELETE HTTP verb, os-force_delete is only supported via the admin actions extension /action
* At the Driver API, cgsnapshot objects and cg objects (with all the information about what volumes are in the CG) need to be passed to the driver instead of just the cg/cgsnapshot name. This is important for backends that aren't managing a CG construct such as NetApp cDOT as they need to do things such as loop through the snapshots in a cg to delete them.
* The default policy in policy.json for consistency group operations is the default policy versus in Cinder is it 'nobody'
* Unlike Cinder, snapshots in a CGsnapshot are not the same as a normal snapshot. A CGsnapshot is treated as a single unit instead of a collection of snapshots.
* consistency_group_support scheduler capability has multiple values instead of boolean. This is to accommodate backends that require all volumes in a CG to be on the same pool vs. just the same host. It also allows for adding additional values in the future if a backend can support some other paradigm like a CG spanning multiple backends.

== User Workflows ==
* Snapshot multiple shares
*# Create a Consistency Group
*# Create share(s) with a CG specified
*# Create a CGsnapshot

* Create a CG
*# POST to /os-consistency-group
*#* specifiy share_types or the default share type will be used

* Copy a consistency group
*# Create a CGsnapshot
*# Create a CG with the CGSnapshot id

* Delete a CG (and all of its shares)
*# Delete all CG snapshots of the CG
*# Delete all shares in the CG
*# Delete the CG (Only empty CGs can be deleted)

* List shares in a given CG
*# GET /shares?consistencygroup_id=<cg_id>

* List the shares that were captured in a given CGSnapshot
*# GET /cgsnapshots/<id>/members

== System Workflows ==
* Creating a CG
*# Create the CG in the DB
*# Cast the consistency group creation to the scheduler
*## Choose a host for the CG
*# Cast the creation to the share service
*## Get the CG info from DB
*## Call the driver
*## Update database with info returned from driver (ex: CG status)

* Creating a CG from a cgsnapshot
*# Create a new CG in the DB
*# Schedule the CG create to the scheduler with the orig CG host/pool+share_type (depending on capability)
*# For each member of the snapshot, schedule a create share on the same host/pool as CG

* Creating a cg snapshot
*# Create cgsnapshot entity in DB
*# For all shares in CG, create cgsnapshot_member in DB
*# cast create_cgsnapshot to the driver

* Adding a share to a CG
** On share creation:
**# Cast to scheduler with the host of the CG
**#* if perpool capable: Goes to the same pool as CG
**#* if per backend: goes to a pool on the same host that matches a CG share type

* Deleting a share (in a CG)
** Not allowed if the CG has snapshots of the share ( check if the share_id is in the CGSnapshotMembers tables)

== Scheduler ==
manila/scheduler/filter_scheduler.py should now look for consistency_group_support as a capability.

consistency_group_support
default: None 
Values:
* None - No support for CGs
* host - shares in a CG must be on pool(s) on the same host that also match the CG share type
* pool - shares in a CG must live in the same pool as the CG

==DB==
* New Tables:
class ConsistencyGroup(BASE, ManilaBase):
"""Represents a consistencygroup."""
__tablename__ = 'consistency_groups'
id = Column(String(36), primary_key=True)

user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)
deleted = Column(String(36), default='False')

host = Column(String(255))
name = Column(String(255))
description = Column(String(255))
status = Column(String(255))
source_cgsnapshot_id = Column(String(36))

class ConsistencyGroupShareTypeMapping(BASE, ManilaBase):
"""Represents the share types in a consistency group"""
__tablename__ = 'consistency_group_share_type_mappings'
id = Column(Integer, primary_key=True)
deleted = Column(String(36), default='False')
consistency_group_id = Column(String(36),
ForeignKey('consistency_groups.id'),
nullable=False)
share_type_id = Column(String(36),
ForeignKey('share_types.id'),
nullable=False)
consistency_group = orm.relationship(
ConsistencyGroup,
backref="share_types",
foreign_keys=consistency_group_id,
primaryjoin=('and_('
'ConsistencyGroupShareTypeMapping.consistency_group_id == '
'ConsistencyGroups.id,'
'ConsistencyGroupShareTypeMapping.deleted == 0)')
)

class CGSnapshot(BASE, ManilaBase):
"""Represents a cgsnapshot."""
__tablename__ = 'cgsnapshots'
id = Column(String(36), primary_key=True)

consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'))
user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)

name = Column(String(255))
description = Column(String(255))
status = Column(String(255))

class CGSnapshotMembers(BASE, ManilaBase):
__tablename__ = 'cgsnapshot_members'
id = Column(String(36), primary_key=True)
cgsnapshot_id = Column(String(36), ForeignKey('cgsnapshots.id'))
share_id = Column(String(36), ForeignKey('shares.id'))
size = Column(Integer)
status = Column(String(255))
share_proto = Column(String(255))
share_type_id = Column(String(36), ForeignKey('share_types.id'),
nullable=True)
user_id = Column(String(255))
project_id = Column(String(255))
deleted = Column(String(36), default='False')

cgsnapshot = relationship(
ConsistencyGroup,
backref="cgsnapshot_members",
foreign_keys=cgsnapshot_id,
primaryjoin='CGSnapshotMembers.cgsnapshot_id == CGSnapshot.id')

* New fields in Share
consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'), nullable=True)
consistency_group = relationship(
ConsistencyGroup,
backref="shares",
foreign_keys=consistency_group_id,
primaryjoin='Share.consistency_group_id == ConsistencyGroup.id')

source_cgsnapshot_member_id = Column(String(36), nullable=True)

==API==
Schemas: https://wiki.openstack.org/wiki/Manila/design/manila-liberty-consistency-groups/api-schema

'''GET''' /os-consistency-groups/
'''GET''' /os-consistency-groups/detail
'''POST''' /os-consistency-groups
'''GET''' /os-consistency-groups/<id>
'''DELETE''' os-consistency-groups/<id>
'''PUT''' /os-consistency-groups/<id>

'''GET''' /os-cgsnapshots/
'''GET''' /os-cgsnapshots/detail
'''POST''' /os-cgsnapshots
'''PUT''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>/members
'''DELETE''' /os-cgsnapshots/<id>

Admin Actions
'''POST''' /os-consistency-groups/<id>/action # os-reset-status and os-force-delete
'''POST''' /os-cgsnapshots/<id>/action # os-reset-status and os-force-delete

Adds consistency_group_id to POST/shares
Add consistency_group_id query filter to /shares

==Policies==
policy.json - All consistency group policies should default to the default policy
"consistency_group:create" : "",
"consistency_group:delete": "",
"consistency_group:update": "",
"consistency_group:get": "",
"consistency_group:get_all": "",

"consistency_group:create_cgsnapshot" : "",
"consistency_group:delete_cgsnapshot": "",
"consistency_group:get_cgsnapshot": "",
"consistency_group:get_all_cgsnapshots": "",

==Driver API==

update_share_stats should now return 'consistency_group_support' pool/host/None.

def create_consistency_group(self, context, cg_dict, share_server=None):
"""Create a consistency group.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_consistency_group_from_cgsnapshot(self, context, cg_dict,
cgsnapshot_dict, share_server=None):
"""Create a consistency group from a cgsnapshot.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'shares': [<models.Share>], # The new shares being created
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:param cgsnapshot_dict:
EXAMPLE:
{'status': 'available',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': '0',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'consistency_group_id': '4b04fdc3-00b9-4909-ba1a-06e9b3f88b67',
'cgsnapshot_members': [
{'status': 'available',
'share_type_id': '1a9ed31e-ee70-483d-93ba-89690e028d7f',
'share_id': 'e14b5174-e534-4f35-bc4f-fe81c1575d6f',
'user_id': 'a0314a441ca842019b0952224aa39192',
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share': <models.Share>,
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share_proto': 'NFS',
'project_id': '13c0be6290934bd98596cfa004650049',
'cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'deleted_at': None,
'id': '6813e06b-a8f5-4784-b17d-f3e91afa370e',
'size': 1
}
],
'deleted_at': None,
'id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.

share_update_list - a list of dictionaries containing dicts for
every share created in the CG. Any share dicts should at a minimum
contain the 'id' key and 'export_locations'. Export locations
should be in the same format as returned by a share_create. This
list may be empty or None.
EXAMPLE:
[
{'id': 'uuid', 'export_locations': ['export_path']}
]
"""
raise NotImplementedError()

def delete_consistency_group(self, context, cg_dict, share_server=None):
"""Delete a consistency group

:param context:
:param cg_dict:
EXAMPLE:
:return: cg_model_update
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_cgsnapshot(self, context, snap_dict, share_server=None):
"""Create a consistency group snapshot.

:param context:
:param snap_dict:
EXAMPLE:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.

member_update_list - a list of dictionaries containing for every
member of the cgsnapshot. Each dict should contains values to be
updated for teh CGSnapshotMember in the database. This list may be
empty or None.
"""
raise NotImplementedError()

def delete_cgsnapshot(self, context, snap_dict, share_server=None):
"""Delete a consistency group snapshot

:param context:
:param snap_dict:
EXAMPLE:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.
"""
raise NotImplementedError()

== Manila Client ==
* Do we want to alias the commands so that there are Cinder consistent commands and the improved names? (I.E cg-create = consisgroup-create) Perhaps we alias the cinderclient commands to be more like this?
cg-snapshot-create Creates a cgsnapshot.
cg-snapshot-delete Removes one or more cgsnapshots.
cg-snapshot-list Lists all cgsnapshots.
cg-snapshot-show Shows cgsnapshot details.

cg-create Creates a consistency group (--cgsnapshot to create from existing cg snapshot)
cg-delete Removes one or more consistency groups.
cg-list Lists all consistencygroups.
cg-show Shows details of a consistency group.
cg-update Updates a consistencygroup.

create --consistency-group Creates a share and puts it into consistency group

= Outstanding Questions / FAQ =
# How do we avoid confusion if no backends in a deployment support CGs?
#* It appears that Cinder has the policies default to 'nobody' but this seems odd, is there a better way?
#** (ameade) I can't think of a way to have devstack set the policies automatically either.
#** (ameade) we have chosen to go with an os-consistency-groups extension for now
# Do we need a way to fake CGs with the generic driver?
#* <strike> NO, but it will not be a core feature if it cannot be supported in the generic driver.</strike>
#* Ben says that we should fake it in the generic driver so it can be tested by the gate.
# Do we need a safeguard for deleting shares that are in a CG? (I.E. protected shares?) Are shares in a CG more valuable than normal shares?
# Should we overload the Snapshots table to include snapshots in a CGSnapshot to avoid adding a new db table? Or continue with having a new table?
# Why not have the cg delete all of the shares in it when deleting the cg?
#* This could cause odd state to occur if the backend fails when deleting shares and it will not be obvious where it failed. This is also extra complexity in the implementation that could instead be hidden by a client.

= Notes on Cinder Impl =

* A consistency group may be specified on volume create
* When a consistency group is created it is scheduled to a pool and all volume created in the CG will go to that pol

Cinder cli:
cgsnapshot-create Creates a cgsnapshot and snapshots of every volume in the CG that show up in snapshot-list.
cgsnapshot-delete Removes one or more cgsnapshots.
cgsnapshot-list Lists all cgsnapshots.
cgsnapshot-show Shows cgsnapshot details. <-- /cgsnapshots/detail?all_tenants=1&name=blah
consisgroup-create Creates a consistency group.
consisgroup-create-from-src Creates a consistency group from a cgsnapshot filled with volumes from the snapshots in the cg.
consisgroup-delete Removes one or more consistency groups.
consisgroup-list Lists all consistencygroups.
consisgroup-show Shows details of a consistency group. <-- /consistencygroups/detail?all_tenants=1&name=blah
consisgroup-update Updates a consistencygroup.

create --consisgroup-id

Cinder API:
'''GET''' /consistencygroups/detail
'''POST''' /consistencygroups
{'consistencygroup':{
'name': <string>,
'description': <string>,
'volume_types': <string>,
'availability_zone': <string>,
}}
'''POST''' /consistencygroups/create_from_src
{'consistencygroup-from-src':{
'name': <string>,
'description': <string>,
'cgsnapshot_id': <string>,
}}
'''GET''' /consistencygroups/<id>
'''POST''' consistencygroups/<id>/delete
{"consistencygroup": {"force": false}}
'''PUT''' /consistencygroups/<id>
{'consistencygroup':{
'name': <string>,
'description': <string>,
'add_volumes': <string>,
'remove_volumes': <string>,
}}

'''GET''' /cgsnapshots/detail
'''GET''' /cgsnapshots/<id>
'''POST''' /cgsnapshots
{'cgsnapshot':{
'consistencygroup_id': <string>,
}}
'''DELETE''' /cgsnapshots/<id>

Adds consistencygroup_id to POST/volumes

Xings commit message:
1) Create a CG, specifying all volume types that can be supported by this
CG. The scheduler chooses a backend that supports all specified volume types.
The CG will be empty when it is first created. Backend needs to report
consistencygroup_support = True. Volume type can have the following in
extra specs: {'capabilities:consistencygroup_support': '<is> True'}.
If consistencygroup_support is not in volume type extra specs, it will be
added to filter_properties by the scheduler to make sure that the scheduler
will select the backend which reports consistency group support capability.

Create CG CLI:
cinder consisgroup-create --volume-type type1,type2 mycg1

This will add a CG entry in the new consistencygroups table.

2) After the CG is created, create a new volume and add to the CG.
Repeat until all volumes are created for the CG.

Create volume CLI (with CG):
cinder create --volume-type type1 --consisgroup-id <CG uuid> 10

This will add a consistencygroup_id foreign key in the new volume
entry in the db.

3) Create a snapshot of the CG (cgsnapshot).

Create cgsnapshot CLI:
cinder cgsnapshot-create <CG uuid>

This will add a cgsnapshot entry in the new cgsnapshots table, create
snapshot for each volume in the CG, and add a cgsnapshot_id foreign key
in each newly created snapshot entry in the db.

===Questions===
# Why does policy.json default to 'nobody' being able to perform CG commands?
# Why are volume types specified when creating a CG?
#* This is because all volumes within a CG need to be on the same backend in order for the backend to honor the CG. This is important to note in the docs so that conflicting volume-types are not specified.
# What occurs when a volume on a different backend is added to a CG via the API?
#* Currently the API ensures that the volume volume_type matches a supported volume_type of the CG and that the CG and volume are on the same host (but can be on different pools).
# Is a volume created with a CG going to end up on the same host?
#* Yes, if a volume has a CG it will be cast to the same pool as the CG.
#* What if the pool is full but there are other available pools on the same backend? Should this fail?
# What happens if a driver does not support create_from_src?
#* The CG goes to ERROR a and volumes are created from the snapshot and also in ERROR status
#* A volume in a consistency group cannot be deleted and a CG with a volume cannot be deleted, therefore a volume must be removed from the CG before it can be deleted. But what if you don't support modifying a CG?
# Why do we need a force delete for CGs?
# What do I do if I do not want to see Snapshots in a CG when doing snapshot-list?
#* I suppose API filtering could allow for someone to not see all of the snapshots if they do not want to.

=== Cinder Bugs to file===
# What happens when you force delete a CG?
#* The volumes remain but still think they are in a CG, which means you still cannot delete the volume
# consisgroup-create-from-src says cgsnapshot is and optional param in cinderclient
# Creating a volume in a CG may not end up in the cg. The only check that is performed is that the volume-type of the volume is supported by the CG but volume-types are not 1-to-1 with backends. There could be backend A and B that both match with volume type 1. A cg is created with volume type 1 and ends up on backend A. Then a volume is created with the CG specified but ends up on backend B

=== Main impl ===
299b99437c2de84222fd585f06c2d05ca213225b ConsistencyGroup: Return 400 instead of 500 for invalid body
adb4c80be82caacad83f1366a4b34e5653fd5dab Create Consistency Group from CG Snapshot API
1a62a6e60fda73bf31256fbf684dc03bb6cf0038 Modify Consistency Group API
fc7a4384be2ccf8ff24f6c0f72c681ad9133801a Add tests for consistency groups DB migration
9082273305b0b9c117eb677a0e34c2ffde4e66f0 Volume types need to be specified when creating CG
764c5ec749821d36bb0215dc6002d3caea95d3b1 Delete consistency group failed
cf961f83ac323dfad1fa5e227d1e502a17529ecc Consistency Groups

=== Ex driver impl ===
8d5b795b37ed021c2639066689645f8aa0b1012f PureISCSIDriver consistency group updates.
39c1a8e08dc854e22ada315a46c20c99df2facf8 Add support to PureISCSIDriver for Consistency Groups
92a817708a3938b1b734d2caaa206b310996d8d0 EMC VMAX driver Kilo update

Manila/design/manila-liberty-consistency-groups

2015-08-10T23:09:56Z

Alex Meade: /* Outstanding Questions / FAQ */

= Consistency Groups =
This document describes the design choices and steps for the consistency group feature in core Manila.

Consistency groups are a mechanism to allow multiple filesystem shares to be guaranteed snapshots are able to be created at the exact same point in time. For example, a database may have its tables, logs, and configuration on separate volumes. If we wanted to restore the database from a previous point in time, it would only make sense to restore the logs, tables, and configuration together from the exact same point in time.

= Manila Core Changes =
The core manila changes are very similar to what has been done in Cinder. At a high level, changes in the DB, share service, drivers, scheduler service and api service will need to be changed.

Phase 1:
* Create/Delete CGs
* Create a share within a CG
* Delete a share that is in a CG
* Snapshot entire CGs
* Create CG from a CGSnapshot

Potential future features:
* Add/remove shares in a CG
* CG migration
* Replication of a CG

==Manila vs Cinder==
* Different CLI syntax
* Restful api tweaks
** Change /create_from_src to just be a cgsnapshot_id in the body on POST /shares
** Deletes to use the DELETE HTTP verb, force_delete not supported until we have a standard API guideline
* At the Driver API, cgsnapshot objects and cg objects (with all the information about what volumes are in the CG) need to be passed to the driver instead of just the cg/cgsnapshot name. This is important for backends that aren't managing a CG construct such as NetApp cDOT as they need to do things such as loop through the snapshots in a cg to delete them.
* The default policy in policy.json for consistency group operations is the default policy versus in Cinder is it 'nobody'
* Unlike Cinder, snapshots in a CGsnapshot are not the same as a normal snapshot. A CGsnapshot is treated as a single unit instead of a collection of snapshots.
* consistency_group_support scheduler capability has multiple values instead of boolean. This is to accommodate backends that require all volumes in a CG to be on the same pool vs. just the same host. It also allows for adding additional values in the future if a backend can support some other paradigm like a CG spanning multiple backends.

== User Workflows ==
* Snapshot multiple shares
*# Create a Consistency Group
*# Create share(s) with a CG specified
*# Create a CGsnapshot

* Create a CG
*# POST to /os-consistency-group
*#* specifiy share_types or the default share type will be used

* Copy a consistency group
*# Create a CGsnapshot
*# Create a CG with the CGSnapshot id

* Delete a CG (and all of its shares)
*# Delete all CG snapshots of the CG
*# Delete all shares in the CG
*# Delete the CG (Only empty CGs can be deleted)

* List shares in a given CG
*# GET /shares?consistencygroup_id=<cg_id>

* List the shares that were captured in a given CGSnapshot
*# GET /cgsnapshots/<id>/members

== System Workflows ==
* Creating a CG
*# Create the CG in the DB
*# Cast the consistency group creation to the scheduler
*## Choose a host for the CG
*# Cast the creation to the share service
*## Get the CG info from DB
*## Call the driver
*## Update database with info returned from driver (ex: CG status)

* Creating a CG from a cgsnapshot
*# Create a new CG in the DB
*# Schedule the CG create to the scheduler with the orig CG host/pool+share_type (depending on capability)
*# For each member of the snapshot, schedule a create share on the same host/pool as CG

* Creating a cg snapshot
*# Create cgsnapshot entity in DB
*# For all shares in CG, create cgsnapshot_member in DB
*# cast create_cgsnapshot to the driver

* Adding a share to a CG
** On share creation:
**# Cast to scheduler with the host of the CG
**#* if perpool capable: Goes to the same pool as CG
**#* if per backend: goes to a pool on the same host that matches a CG share type

* Deleting a share (in a CG)
** Not allowed if the CG has snapshots of the share ( check if the share_id is in the CGSnapshotMembers tables)

== Scheduler ==
manila/scheduler/filter_scheduler.py should now look for consistency_group_support as a capability.

consistency_group_support
default: None 
Values:
* None - No support for CGs
* host - shares in a CG must be on pool(s) on the same host that also match the CG share type
* pool - shares in a CG must live in the same pool as the CG

==DB==
* New Tables:
class ConsistencyGroup(BASE, ManilaBase):
"""Represents a consistencygroup."""
__tablename__ = 'consistency_groups'
id = Column(String(36), primary_key=True)

user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)
deleted = Column(String(36), default='False')

host = Column(String(255))
name = Column(String(255))
description = Column(String(255))
status = Column(String(255))
source_cgsnapshot_id = Column(String(36))

class ConsistencyGroupShareTypeMapping(BASE, ManilaBase):
"""Represents the share types in a consistency group"""
__tablename__ = 'consistency_group_share_type_mappings'
id = Column(Integer, primary_key=True)
deleted = Column(String(36), default='False')
consistency_group_id = Column(String(36),
ForeignKey('consistency_groups.id'),
nullable=False)
share_type_id = Column(String(36),
ForeignKey('share_types.id'),
nullable=False)
consistency_group = orm.relationship(
ConsistencyGroup,
backref="share_types",
foreign_keys=consistency_group_id,
primaryjoin=('and_('
'ConsistencyGroupShareTypeMapping.consistency_group_id == '
'ConsistencyGroups.id,'
'ConsistencyGroupShareTypeMapping.deleted == 0)')
)

class CGSnapshot(BASE, ManilaBase):
"""Represents a cgsnapshot."""
__tablename__ = 'cgsnapshots'
id = Column(String(36), primary_key=True)

consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'))
user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)

name = Column(String(255))
description = Column(String(255))
status = Column(String(255))

class CGSnapshotMembers(BASE, ManilaBase):
__tablename__ = 'cgsnapshot_members'
id = Column(String(36), primary_key=True)
cgsnapshot_id = Column(String(36), ForeignKey('cgsnapshots.id'))
share_id = Column(String(36), ForeignKey('shares.id'))
size = Column(Integer)
status = Column(String(255))
share_proto = Column(String(255))
share_type_id = Column(String(36), ForeignKey('share_types.id'),
nullable=True)
user_id = Column(String(255))
project_id = Column(String(255))
deleted = Column(String(36), default='False')

cgsnapshot = relationship(
ConsistencyGroup,
backref="cgsnapshot_members",
foreign_keys=cgsnapshot_id,
primaryjoin='CGSnapshotMembers.cgsnapshot_id == CGSnapshot.id')

* New fields in Share
consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'), nullable=True)
consistency_group = relationship(
ConsistencyGroup,
backref="shares",
foreign_keys=consistency_group_id,
primaryjoin='Share.consistency_group_id == ConsistencyGroup.id')

source_cgsnapshot_member_id = Column(String(36), nullable=True)

==API==
Schemas: https://wiki.openstack.org/wiki/Manila/design/manila-liberty-consistency-groups/api-schema

'''GET''' /os-consistency-groups/
'''GET''' /os-consistency-groups/detail
'''POST''' /os-consistency-groups
'''GET''' /os-consistency-groups/<id>
'''DELETE''' os-consistency-groups/<id>
'''PUT''' /os-consistency-groups/<id>

'''GET''' /os-cgsnapshots/
'''GET''' /os-cgsnapshots/detail
'''POST''' /os-cgsnapshots
'''PUT''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>/members
'''DELETE''' /os-cgsnapshots/<id>

Admin Actions
'''POST''' /os-consistency-groups/<id>/action # os-reset-status and os-force-delete
'''POST''' /os-cgsnapshots/<id>/action # os-reset-status and os-force-delete

Adds consistency_group_id to POST/shares
Add consistency_group_id query filter to /shares

==Policies==
policy.json - All consistency group policies should default to the default policy
"consistency_group:create" : "",
"consistency_group:delete": "",
"consistency_group:update": "",
"consistency_group:get": "",
"consistency_group:get_all": "",

"consistency_group:create_cgsnapshot" : "",
"consistency_group:delete_cgsnapshot": "",
"consistency_group:get_cgsnapshot": "",
"consistency_group:get_all_cgsnapshots": "",

==Driver API==

update_share_stats should now return 'consistency_group_support' pool/host/None.

def create_consistency_group(self, context, cg_dict, share_server=None):
"""Create a consistency group.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_consistency_group_from_cgsnapshot(self, context, cg_dict,
cgsnapshot_dict, share_server=None):
"""Create a consistency group from a cgsnapshot.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'shares': [<models.Share>], # The new shares being created
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:param cgsnapshot_dict:
EXAMPLE:
{'status': 'available',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': '0',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'consistency_group_id': '4b04fdc3-00b9-4909-ba1a-06e9b3f88b67',
'cgsnapshot_members': [
{'status': 'available',
'share_type_id': '1a9ed31e-ee70-483d-93ba-89690e028d7f',
'share_id': 'e14b5174-e534-4f35-bc4f-fe81c1575d6f',
'user_id': 'a0314a441ca842019b0952224aa39192',
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share': <models.Share>,
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share_proto': 'NFS',
'project_id': '13c0be6290934bd98596cfa004650049',
'cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'deleted_at': None,
'id': '6813e06b-a8f5-4784-b17d-f3e91afa370e',
'size': 1
}
],
'deleted_at': None,
'id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.

share_update_list - a list of dictionaries containing dicts for
every share created in the CG. Any share dicts should at a minimum
contain the 'id' key and 'export_locations'. Export locations
should be in the same format as returned by a share_create. This
list may be empty or None.
EXAMPLE:
[
{'id': 'uuid', 'export_locations': ['export_path']}
]
"""
raise NotImplementedError()

def delete_consistency_group(self, context, cg_dict, share_server=None):
"""Delete a consistency group

:param context:
:param cg_dict:
EXAMPLE:
:return: cg_model_update
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_cgsnapshot(self, context, snap_dict, share_server=None):
"""Create a consistency group snapshot.

:param context:
:param snap_dict:
EXAMPLE:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.

member_update_list - a list of dictionaries containing for every
member of the cgsnapshot. Each dict should contains values to be
updated for teh CGSnapshotMember in the database. This list may be
empty or None.
"""
raise NotImplementedError()

def delete_cgsnapshot(self, context, snap_dict, share_server=None):
"""Delete a consistency group snapshot

:param context:
:param snap_dict:
EXAMPLE:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.
"""
raise NotImplementedError()

== Manila Client ==
* Do we want to alias the commands so that there are Cinder consistent commands and the improved names? (I.E cg-create = consisgroup-create) Perhaps we alias the cinderclient commands to be more like this?
cg-snapshot-create Creates a cgsnapshot.
cg-snapshot-delete Removes one or more cgsnapshots.
cg-snapshot-list Lists all cgsnapshots.
cg-snapshot-show Shows cgsnapshot details.

cg-create Creates a consistency group (--cgsnapshot to create from existing cg snapshot)
cg-delete Removes one or more consistency groups.
cg-list Lists all consistencygroups.
cg-show Shows details of a consistency group.
cg-update Updates a consistencygroup.

create --consistency-group Creates a share and puts it into consistency group

= Outstanding Questions / FAQ =
# How do we avoid confusion if no backends in a deployment support CGs?
#* It appears that Cinder has the policies default to 'nobody' but this seems odd, is there a better way?
#** (ameade) I can't think of a way to have devstack set the policies automatically either.
#** (ameade) we have chosen to go with an os-consistency-groups extension for now
# Do we need a way to fake CGs with the generic driver?
#* <strike> NO, but it will not be a core feature if it cannot be supported in the generic driver.</strike>
#* Ben says that we should fake it in the generic driver so it can be tested by the gate.
# Do we need a safeguard for deleting shares that are in a CG? (I.E. protected shares?) Are shares in a CG more valuable than normal shares?
# Should we overload the Snapshots table to include snapshots in a CGSnapshot to avoid adding a new db table? Or continue with having a new table?
# Why not have the cg delete all of the shares in it when deleting the cg?
#* This could cause odd state to occur if the backend fails when deleting shares and it will not be obvious where it failed. This is also extra complexity in the implementation that could instead be hidden by a client.

= Notes on Cinder Impl =

* A consistency group may be specified on volume create
* When a consistency group is created it is scheduled to a pool and all volume created in the CG will go to that pol

Cinder cli:
cgsnapshot-create Creates a cgsnapshot and snapshots of every volume in the CG that show up in snapshot-list.
cgsnapshot-delete Removes one or more cgsnapshots.
cgsnapshot-list Lists all cgsnapshots.
cgsnapshot-show Shows cgsnapshot details. <-- /cgsnapshots/detail?all_tenants=1&name=blah
consisgroup-create Creates a consistency group.
consisgroup-create-from-src Creates a consistency group from a cgsnapshot filled with volumes from the snapshots in the cg.
consisgroup-delete Removes one or more consistency groups.
consisgroup-list Lists all consistencygroups.
consisgroup-show Shows details of a consistency group. <-- /consistencygroups/detail?all_tenants=1&name=blah
consisgroup-update Updates a consistencygroup.

create --consisgroup-id

Cinder API:
'''GET''' /consistencygroups/detail
'''POST''' /consistencygroups
{'consistencygroup':{
'name': <string>,
'description': <string>,
'volume_types': <string>,
'availability_zone': <string>,
}}
'''POST''' /consistencygroups/create_from_src
{'consistencygroup-from-src':{
'name': <string>,
'description': <string>,
'cgsnapshot_id': <string>,
}}
'''GET''' /consistencygroups/<id>
'''POST''' consistencygroups/<id>/delete
{"consistencygroup": {"force": false}}
'''PUT''' /consistencygroups/<id>
{'consistencygroup':{
'name': <string>,
'description': <string>,
'add_volumes': <string>,
'remove_volumes': <string>,
}}

'''GET''' /cgsnapshots/detail
'''GET''' /cgsnapshots/<id>
'''POST''' /cgsnapshots
{'cgsnapshot':{
'consistencygroup_id': <string>,
}}
'''DELETE''' /cgsnapshots/<id>

Adds consistencygroup_id to POST/volumes

Xings commit message:
1) Create a CG, specifying all volume types that can be supported by this
CG. The scheduler chooses a backend that supports all specified volume types.
The CG will be empty when it is first created. Backend needs to report
consistencygroup_support = True. Volume type can have the following in
extra specs: {'capabilities:consistencygroup_support': '<is> True'}.
If consistencygroup_support is not in volume type extra specs, it will be
added to filter_properties by the scheduler to make sure that the scheduler
will select the backend which reports consistency group support capability.

Create CG CLI:
cinder consisgroup-create --volume-type type1,type2 mycg1

This will add a CG entry in the new consistencygroups table.

2) After the CG is created, create a new volume and add to the CG.
Repeat until all volumes are created for the CG.

Create volume CLI (with CG):
cinder create --volume-type type1 --consisgroup-id <CG uuid> 10

This will add a consistencygroup_id foreign key in the new volume
entry in the db.

3) Create a snapshot of the CG (cgsnapshot).

Create cgsnapshot CLI:
cinder cgsnapshot-create <CG uuid>

This will add a cgsnapshot entry in the new cgsnapshots table, create
snapshot for each volume in the CG, and add a cgsnapshot_id foreign key
in each newly created snapshot entry in the db.

===Questions===
# Why does policy.json default to 'nobody' being able to perform CG commands?
# Why are volume types specified when creating a CG?
#* This is because all volumes within a CG need to be on the same backend in order for the backend to honor the CG. This is important to note in the docs so that conflicting volume-types are not specified.
# What occurs when a volume on a different backend is added to a CG via the API?
#* Currently the API ensures that the volume volume_type matches a supported volume_type of the CG and that the CG and volume are on the same host (but can be on different pools).
# Is a volume created with a CG going to end up on the same host?
#* Yes, if a volume has a CG it will be cast to the same pool as the CG.
#* What if the pool is full but there are other available pools on the same backend? Should this fail?
# What happens if a driver does not support create_from_src?
#* The CG goes to ERROR a and volumes are created from the snapshot and also in ERROR status
#* A volume in a consistency group cannot be deleted and a CG with a volume cannot be deleted, therefore a volume must be removed from the CG before it can be deleted. But what if you don't support modifying a CG?
# Why do we need a force delete for CGs?
# What do I do if I do not want to see Snapshots in a CG when doing snapshot-list?
#* I suppose API filtering could allow for someone to not see all of the snapshots if they do not want to.

=== Cinder Bugs to file===
# What happens when you force delete a CG?
#* The volumes remain but still think they are in a CG, which means you still cannot delete the volume
# consisgroup-create-from-src says cgsnapshot is and optional param in cinderclient
# Creating a volume in a CG may not end up in the cg. The only check that is performed is that the volume-type of the volume is supported by the CG but volume-types are not 1-to-1 with backends. There could be backend A and B that both match with volume type 1. A cg is created with volume type 1 and ends up on backend A. Then a volume is created with the CG specified but ends up on backend B

=== Main impl ===
299b99437c2de84222fd585f06c2d05ca213225b ConsistencyGroup: Return 400 instead of 500 for invalid body
adb4c80be82caacad83f1366a4b34e5653fd5dab Create Consistency Group from CG Snapshot API
1a62a6e60fda73bf31256fbf684dc03bb6cf0038 Modify Consistency Group API
fc7a4384be2ccf8ff24f6c0f72c681ad9133801a Add tests for consistency groups DB migration
9082273305b0b9c117eb677a0e34c2ffde4e66f0 Volume types need to be specified when creating CG
764c5ec749821d36bb0215dc6002d3caea95d3b1 Delete consistency group failed
cf961f83ac323dfad1fa5e227d1e502a17529ecc Consistency Groups

=== Ex driver impl ===
8d5b795b37ed021c2639066689645f8aa0b1012f PureISCSIDriver consistency group updates.
39c1a8e08dc854e22ada315a46c20c99df2facf8 Add support to PureISCSIDriver for Consistency Groups
92a817708a3938b1b734d2caaa206b310996d8d0 EMC VMAX driver Kilo update

Manila/design/manila-liberty-consistency-groups

2015-08-10T23:08:18Z

Alex Meade: /* Outstanding Questions / FAQ */

= Consistency Groups =
This document describes the design choices and steps for the consistency group feature in core Manila.

Consistency groups are a mechanism to allow multiple filesystem shares to be guaranteed snapshots are able to be created at the exact same point in time. For example, a database may have its tables, logs, and configuration on separate volumes. If we wanted to restore the database from a previous point in time, it would only make sense to restore the logs, tables, and configuration together from the exact same point in time.

= Manila Core Changes =
The core manila changes are very similar to what has been done in Cinder. At a high level, changes in the DB, share service, drivers, scheduler service and api service will need to be changed.

Phase 1:
* Create/Delete CGs
* Create a share within a CG
* Delete a share that is in a CG
* Snapshot entire CGs
* Create CG from a CGSnapshot

Potential future features:
* Add/remove shares in a CG
* CG migration
* Replication of a CG

==Manila vs Cinder==
* Different CLI syntax
* Restful api tweaks
** Change /create_from_src to just be a cgsnapshot_id in the body on POST /shares
** Deletes to use the DELETE HTTP verb, force_delete not supported until we have a standard API guideline
* At the Driver API, cgsnapshot objects and cg objects (with all the information about what volumes are in the CG) need to be passed to the driver instead of just the cg/cgsnapshot name. This is important for backends that aren't managing a CG construct such as NetApp cDOT as they need to do things such as loop through the snapshots in a cg to delete them.
* The default policy in policy.json for consistency group operations is the default policy versus in Cinder is it 'nobody'
* Unlike Cinder, snapshots in a CGsnapshot are not the same as a normal snapshot. A CGsnapshot is treated as a single unit instead of a collection of snapshots.
* consistency_group_support scheduler capability has multiple values instead of boolean. This is to accommodate backends that require all volumes in a CG to be on the same pool vs. just the same host. It also allows for adding additional values in the future if a backend can support some other paradigm like a CG spanning multiple backends.

== User Workflows ==
* Snapshot multiple shares
*# Create a Consistency Group
*# Create share(s) with a CG specified
*# Create a CGsnapshot

* Create a CG
*# POST to /os-consistency-group
*#* specifiy share_types or the default share type will be used

* Copy a consistency group
*# Create a CGsnapshot
*# Create a CG with the CGSnapshot id

* Delete a CG (and all of its shares)
*# Delete all CG snapshots of the CG
*# Delete all shares in the CG
*# Delete the CG (Only empty CGs can be deleted)

* List shares in a given CG
*# GET /shares?consistencygroup_id=<cg_id>

* List the shares that were captured in a given CGSnapshot
*# GET /cgsnapshots/<id>/members

== System Workflows ==
* Creating a CG
*# Create the CG in the DB
*# Cast the consistency group creation to the scheduler
*## Choose a host for the CG
*# Cast the creation to the share service
*## Get the CG info from DB
*## Call the driver
*## Update database with info returned from driver (ex: CG status)

* Creating a CG from a cgsnapshot
*# Create a new CG in the DB
*# Schedule the CG create to the scheduler with the orig CG host/pool+share_type (depending on capability)
*# For each member of the snapshot, schedule a create share on the same host/pool as CG

* Creating a cg snapshot
*# Create cgsnapshot entity in DB
*# For all shares in CG, create cgsnapshot_member in DB
*# cast create_cgsnapshot to the driver

* Adding a share to a CG
** On share creation:
**# Cast to scheduler with the host of the CG
**#* if perpool capable: Goes to the same pool as CG
**#* if per backend: goes to a pool on the same host that matches a CG share type

* Deleting a share (in a CG)
** Not allowed if the CG has snapshots of the share ( check if the share_id is in the CGSnapshotMembers tables)

== Scheduler ==
manila/scheduler/filter_scheduler.py should now look for consistency_group_support as a capability.

consistency_group_support
default: None 
Values:
* None - No support for CGs
* host - shares in a CG must be on pool(s) on the same host that also match the CG share type
* pool - shares in a CG must live in the same pool as the CG

==DB==
* New Tables:
class ConsistencyGroup(BASE, ManilaBase):
"""Represents a consistencygroup."""
__tablename__ = 'consistency_groups'
id = Column(String(36), primary_key=True)

user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)
deleted = Column(String(36), default='False')

host = Column(String(255))
name = Column(String(255))
description = Column(String(255))
status = Column(String(255))
source_cgsnapshot_id = Column(String(36))

class ConsistencyGroupShareTypeMapping(BASE, ManilaBase):
"""Represents the share types in a consistency group"""
__tablename__ = 'consistency_group_share_type_mappings'
id = Column(Integer, primary_key=True)
deleted = Column(String(36), default='False')
consistency_group_id = Column(String(36),
ForeignKey('consistency_groups.id'),
nullable=False)
share_type_id = Column(String(36),
ForeignKey('share_types.id'),
nullable=False)
consistency_group = orm.relationship(
ConsistencyGroup,
backref="share_types",
foreign_keys=consistency_group_id,
primaryjoin=('and_('
'ConsistencyGroupShareTypeMapping.consistency_group_id == '
'ConsistencyGroups.id,'
'ConsistencyGroupShareTypeMapping.deleted == 0)')
)

class CGSnapshot(BASE, ManilaBase):
"""Represents a cgsnapshot."""
__tablename__ = 'cgsnapshots'
id = Column(String(36), primary_key=True)

consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'))
user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)

name = Column(String(255))
description = Column(String(255))
status = Column(String(255))

class CGSnapshotMembers(BASE, ManilaBase):
__tablename__ = 'cgsnapshot_members'
id = Column(String(36), primary_key=True)
cgsnapshot_id = Column(String(36), ForeignKey('cgsnapshots.id'))
share_id = Column(String(36), ForeignKey('shares.id'))
size = Column(Integer)
status = Column(String(255))
share_proto = Column(String(255))
share_type_id = Column(String(36), ForeignKey('share_types.id'),
nullable=True)
user_id = Column(String(255))
project_id = Column(String(255))
deleted = Column(String(36), default='False')

cgsnapshot = relationship(
ConsistencyGroup,
backref="cgsnapshot_members",
foreign_keys=cgsnapshot_id,
primaryjoin='CGSnapshotMembers.cgsnapshot_id == CGSnapshot.id')

* New fields in Share
consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'), nullable=True)
consistency_group = relationship(
ConsistencyGroup,
backref="shares",
foreign_keys=consistency_group_id,
primaryjoin='Share.consistency_group_id == ConsistencyGroup.id')

source_cgsnapshot_member_id = Column(String(36), nullable=True)

==API==
Schemas: https://wiki.openstack.org/wiki/Manila/design/manila-liberty-consistency-groups/api-schema

'''GET''' /os-consistency-groups/
'''GET''' /os-consistency-groups/detail
'''POST''' /os-consistency-groups
'''GET''' /os-consistency-groups/<id>
'''DELETE''' os-consistency-groups/<id>
'''PUT''' /os-consistency-groups/<id>

'''GET''' /os-cgsnapshots/
'''GET''' /os-cgsnapshots/detail
'''POST''' /os-cgsnapshots
'''PUT''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>/members
'''DELETE''' /os-cgsnapshots/<id>

Admin Actions
'''POST''' /os-consistency-groups/<id>/action # os-reset-status and os-force-delete
'''POST''' /os-cgsnapshots/<id>/action # os-reset-status and os-force-delete

Adds consistency_group_id to POST/shares
Add consistency_group_id query filter to /shares

==Policies==
policy.json - All consistency group policies should default to the default policy
"consistency_group:create" : "",
"consistency_group:delete": "",
"consistency_group:update": "",
"consistency_group:get": "",
"consistency_group:get_all": "",

"consistency_group:create_cgsnapshot" : "",
"consistency_group:delete_cgsnapshot": "",
"consistency_group:get_cgsnapshot": "",
"consistency_group:get_all_cgsnapshots": "",

==Driver API==

update_share_stats should now return 'consistency_group_support' pool/host/None.

def create_consistency_group(self, context, cg_dict, share_server=None):
"""Create a consistency group.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_consistency_group_from_cgsnapshot(self, context, cg_dict,
cgsnapshot_dict, share_server=None):
"""Create a consistency group from a cgsnapshot.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'shares': [<models.Share>], # The new shares being created
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:param cgsnapshot_dict:
EXAMPLE:
{'status': 'available',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': '0',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'consistency_group_id': '4b04fdc3-00b9-4909-ba1a-06e9b3f88b67',
'cgsnapshot_members': [
{'status': 'available',
'share_type_id': '1a9ed31e-ee70-483d-93ba-89690e028d7f',
'share_id': 'e14b5174-e534-4f35-bc4f-fe81c1575d6f',
'user_id': 'a0314a441ca842019b0952224aa39192',
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share': <models.Share>,
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share_proto': 'NFS',
'project_id': '13c0be6290934bd98596cfa004650049',
'cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'deleted_at': None,
'id': '6813e06b-a8f5-4784-b17d-f3e91afa370e',
'size': 1
}
],
'deleted_at': None,
'id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.

share_update_list - a list of dictionaries containing dicts for
every share created in the CG. Any share dicts should at a minimum
contain the 'id' key and 'export_locations'. Export locations
should be in the same format as returned by a share_create. This
list may be empty or None.
EXAMPLE:
[
{'id': 'uuid', 'export_locations': ['export_path']}
]
"""
raise NotImplementedError()

def delete_consistency_group(self, context, cg_dict, share_server=None):
"""Delete a consistency group

:param context:
:param cg_dict:
EXAMPLE:
:return: cg_model_update
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_cgsnapshot(self, context, snap_dict, share_server=None):
"""Create a consistency group snapshot.

:param context:
:param snap_dict:
EXAMPLE:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.

member_update_list - a list of dictionaries containing for every
member of the cgsnapshot. Each dict should contains values to be
updated for teh CGSnapshotMember in the database. This list may be
empty or None.
"""
raise NotImplementedError()

def delete_cgsnapshot(self, context, snap_dict, share_server=None):
"""Delete a consistency group snapshot

:param context:
:param snap_dict:
EXAMPLE:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.
"""
raise NotImplementedError()

== Manila Client ==
* Do we want to alias the commands so that there are Cinder consistent commands and the improved names? (I.E cg-create = consisgroup-create) Perhaps we alias the cinderclient commands to be more like this?
cg-snapshot-create Creates a cgsnapshot.
cg-snapshot-delete Removes one or more cgsnapshots.
cg-snapshot-list Lists all cgsnapshots.
cg-snapshot-show Shows cgsnapshot details.

cg-create Creates a consistency group (--cgsnapshot to create from existing cg snapshot)
cg-delete Removes one or more consistency groups.
cg-list Lists all consistencygroups.
cg-show Shows details of a consistency group.
cg-update Updates a consistencygroup.

create --consistency-group Creates a share and puts it into consistency group

= Outstanding Questions / FAQ =
# How do we avoid confusion if no backends in a deployment support CGs?
#* It appears that Cinder has the policies default to 'nobody' but this seems odd, is there a better way?
#** (ameade) I can't think of a way to have devstack set the policies automatically either.
#** (ameade) we have chosen to go with an os-consistency-groups extension for now
# Do we need a way to fake CGs with the generic driver?
#* <strike> NO, but it will not be a core feature if it cannot be supported in the generic driver.</strike> Ben says that we should fake it in the generic driver so it can be tested by the gate.
# Do we need a safeguard for deleting shares that are in a CG? (I.E. protected shares?) Are shares in a CG more valuable than normal shares?
# Should we overload the Snapshots table to include snapshots in a CGSnapshot to avoid adding a new db table? Or continue with having a new table?
# Why not have the cg delete all of the shares in it when deleting the cg?
#* This could cause odd state to occur if the backend fails when deleting shares and it will not be obvious where it failed. This is also extra complexity in the implementation that could instead be hidden by a client.

= Notes on Cinder Impl =

* A consistency group may be specified on volume create
* When a consistency group is created it is scheduled to a pool and all volume created in the CG will go to that pol

Cinder cli:
cgsnapshot-create Creates a cgsnapshot and snapshots of every volume in the CG that show up in snapshot-list.
cgsnapshot-delete Removes one or more cgsnapshots.
cgsnapshot-list Lists all cgsnapshots.
cgsnapshot-show Shows cgsnapshot details. <-- /cgsnapshots/detail?all_tenants=1&name=blah
consisgroup-create Creates a consistency group.
consisgroup-create-from-src Creates a consistency group from a cgsnapshot filled with volumes from the snapshots in the cg.
consisgroup-delete Removes one or more consistency groups.
consisgroup-list Lists all consistencygroups.
consisgroup-show Shows details of a consistency group. <-- /consistencygroups/detail?all_tenants=1&name=blah
consisgroup-update Updates a consistencygroup.

create --consisgroup-id

Cinder API:
'''GET''' /consistencygroups/detail
'''POST''' /consistencygroups
{'consistencygroup':{
'name': <string>,
'description': <string>,
'volume_types': <string>,
'availability_zone': <string>,
}}
'''POST''' /consistencygroups/create_from_src
{'consistencygroup-from-src':{
'name': <string>,
'description': <string>,
'cgsnapshot_id': <string>,
}}
'''GET''' /consistencygroups/<id>
'''POST''' consistencygroups/<id>/delete
{"consistencygroup": {"force": false}}
'''PUT''' /consistencygroups/<id>
{'consistencygroup':{
'name': <string>,
'description': <string>,
'add_volumes': <string>,
'remove_volumes': <string>,
}}

'''GET''' /cgsnapshots/detail
'''GET''' /cgsnapshots/<id>
'''POST''' /cgsnapshots
{'cgsnapshot':{
'consistencygroup_id': <string>,
}}
'''DELETE''' /cgsnapshots/<id>

Adds consistencygroup_id to POST/volumes

Xings commit message:
1) Create a CG, specifying all volume types that can be supported by this
CG. The scheduler chooses a backend that supports all specified volume types.
The CG will be empty when it is first created. Backend needs to report
consistencygroup_support = True. Volume type can have the following in
extra specs: {'capabilities:consistencygroup_support': '<is> True'}.
If consistencygroup_support is not in volume type extra specs, it will be
added to filter_properties by the scheduler to make sure that the scheduler
will select the backend which reports consistency group support capability.

Create CG CLI:
cinder consisgroup-create --volume-type type1,type2 mycg1

This will add a CG entry in the new consistencygroups table.

2) After the CG is created, create a new volume and add to the CG.
Repeat until all volumes are created for the CG.

Create volume CLI (with CG):
cinder create --volume-type type1 --consisgroup-id <CG uuid> 10

This will add a consistencygroup_id foreign key in the new volume
entry in the db.

3) Create a snapshot of the CG (cgsnapshot).

Create cgsnapshot CLI:
cinder cgsnapshot-create <CG uuid>

This will add a cgsnapshot entry in the new cgsnapshots table, create
snapshot for each volume in the CG, and add a cgsnapshot_id foreign key
in each newly created snapshot entry in the db.

===Questions===
# Why does policy.json default to 'nobody' being able to perform CG commands?
# Why are volume types specified when creating a CG?
#* This is because all volumes within a CG need to be on the same backend in order for the backend to honor the CG. This is important to note in the docs so that conflicting volume-types are not specified.
# What occurs when a volume on a different backend is added to a CG via the API?
#* Currently the API ensures that the volume volume_type matches a supported volume_type of the CG and that the CG and volume are on the same host (but can be on different pools).
# Is a volume created with a CG going to end up on the same host?
#* Yes, if a volume has a CG it will be cast to the same pool as the CG.
#* What if the pool is full but there are other available pools on the same backend? Should this fail?
# What happens if a driver does not support create_from_src?
#* The CG goes to ERROR a and volumes are created from the snapshot and also in ERROR status
#* A volume in a consistency group cannot be deleted and a CG with a volume cannot be deleted, therefore a volume must be removed from the CG before it can be deleted. But what if you don't support modifying a CG?
# Why do we need a force delete for CGs?
# What do I do if I do not want to see Snapshots in a CG when doing snapshot-list?
#* I suppose API filtering could allow for someone to not see all of the snapshots if they do not want to.

=== Cinder Bugs to file===
# What happens when you force delete a CG?
#* The volumes remain but still think they are in a CG, which means you still cannot delete the volume
# consisgroup-create-from-src says cgsnapshot is and optional param in cinderclient
# Creating a volume in a CG may not end up in the cg. The only check that is performed is that the volume-type of the volume is supported by the CG but volume-types are not 1-to-1 with backends. There could be backend A and B that both match with volume type 1. A cg is created with volume type 1 and ends up on backend A. Then a volume is created with the CG specified but ends up on backend B

=== Main impl ===
299b99437c2de84222fd585f06c2d05ca213225b ConsistencyGroup: Return 400 instead of 500 for invalid body
adb4c80be82caacad83f1366a4b34e5653fd5dab Create Consistency Group from CG Snapshot API
1a62a6e60fda73bf31256fbf684dc03bb6cf0038 Modify Consistency Group API
fc7a4384be2ccf8ff24f6c0f72c681ad9133801a Add tests for consistency groups DB migration
9082273305b0b9c117eb677a0e34c2ffde4e66f0 Volume types need to be specified when creating CG
764c5ec749821d36bb0215dc6002d3caea95d3b1 Delete consistency group failed
cf961f83ac323dfad1fa5e227d1e502a17529ecc Consistency Groups

=== Ex driver impl ===
8d5b795b37ed021c2639066689645f8aa0b1012f PureISCSIDriver consistency group updates.
39c1a8e08dc854e22ada315a46c20c99df2facf8 Add support to PureISCSIDriver for Consistency Groups
92a817708a3938b1b734d2caaa206b310996d8d0 EMC VMAX driver Kilo update

Manila/design/manila-liberty-consistency-groups

2015-08-10T23:05:33Z

Alex Meade: /* API */

= Consistency Groups =
This document describes the design choices and steps for the consistency group feature in core Manila.

Consistency groups are a mechanism to allow multiple filesystem shares to be guaranteed snapshots are able to be created at the exact same point in time. For example, a database may have its tables, logs, and configuration on separate volumes. If we wanted to restore the database from a previous point in time, it would only make sense to restore the logs, tables, and configuration together from the exact same point in time.

= Manila Core Changes =
The core manila changes are very similar to what has been done in Cinder. At a high level, changes in the DB, share service, drivers, scheduler service and api service will need to be changed.

Phase 1:
* Create/Delete CGs
* Create a share within a CG
* Delete a share that is in a CG
* Snapshot entire CGs
* Create CG from a CGSnapshot

Potential future features:
* Add/remove shares in a CG
* CG migration
* Replication of a CG

==Manila vs Cinder==
* Different CLI syntax
* Restful api tweaks
** Change /create_from_src to just be a cgsnapshot_id in the body on POST /shares
** Deletes to use the DELETE HTTP verb, force_delete not supported until we have a standard API guideline
* At the Driver API, cgsnapshot objects and cg objects (with all the information about what volumes are in the CG) need to be passed to the driver instead of just the cg/cgsnapshot name. This is important for backends that aren't managing a CG construct such as NetApp cDOT as they need to do things such as loop through the snapshots in a cg to delete them.
* The default policy in policy.json for consistency group operations is the default policy versus in Cinder is it 'nobody'
* Unlike Cinder, snapshots in a CGsnapshot are not the same as a normal snapshot. A CGsnapshot is treated as a single unit instead of a collection of snapshots.
* consistency_group_support scheduler capability has multiple values instead of boolean. This is to accommodate backends that require all volumes in a CG to be on the same pool vs. just the same host. It also allows for adding additional values in the future if a backend can support some other paradigm like a CG spanning multiple backends.

== User Workflows ==
* Snapshot multiple shares
*# Create a Consistency Group
*# Create share(s) with a CG specified
*# Create a CGsnapshot

* Create a CG
*# POST to /os-consistency-group
*#* specifiy share_types or the default share type will be used

* Copy a consistency group
*# Create a CGsnapshot
*# Create a CG with the CGSnapshot id

* Delete a CG (and all of its shares)
*# Delete all CG snapshots of the CG
*# Delete all shares in the CG
*# Delete the CG (Only empty CGs can be deleted)

* List shares in a given CG
*# GET /shares?consistencygroup_id=<cg_id>

* List the shares that were captured in a given CGSnapshot
*# GET /cgsnapshots/<id>/members

== System Workflows ==
* Creating a CG
*# Create the CG in the DB
*# Cast the consistency group creation to the scheduler
*## Choose a host for the CG
*# Cast the creation to the share service
*## Get the CG info from DB
*## Call the driver
*## Update database with info returned from driver (ex: CG status)

* Creating a CG from a cgsnapshot
*# Create a new CG in the DB
*# Schedule the CG create to the scheduler with the orig CG host/pool+share_type (depending on capability)
*# For each member of the snapshot, schedule a create share on the same host/pool as CG

* Creating a cg snapshot
*# Create cgsnapshot entity in DB
*# For all shares in CG, create cgsnapshot_member in DB
*# cast create_cgsnapshot to the driver

* Adding a share to a CG
** On share creation:
**# Cast to scheduler with the host of the CG
**#* if perpool capable: Goes to the same pool as CG
**#* if per backend: goes to a pool on the same host that matches a CG share type

* Deleting a share (in a CG)
** Not allowed if the CG has snapshots of the share ( check if the share_id is in the CGSnapshotMembers tables)

== Scheduler ==
manila/scheduler/filter_scheduler.py should now look for consistency_group_support as a capability.

consistency_group_support
default: None 
Values:
* None - No support for CGs
* host - shares in a CG must be on pool(s) on the same host that also match the CG share type
* pool - shares in a CG must live in the same pool as the CG

==DB==
* New Tables:
class ConsistencyGroup(BASE, ManilaBase):
"""Represents a consistencygroup."""
__tablename__ = 'consistency_groups'
id = Column(String(36), primary_key=True)

user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)
deleted = Column(String(36), default='False')

host = Column(String(255))
name = Column(String(255))
description = Column(String(255))
status = Column(String(255))
source_cgsnapshot_id = Column(String(36))

class ConsistencyGroupShareTypeMapping(BASE, ManilaBase):
"""Represents the share types in a consistency group"""
__tablename__ = 'consistency_group_share_type_mappings'
id = Column(Integer, primary_key=True)
deleted = Column(String(36), default='False')
consistency_group_id = Column(String(36),
ForeignKey('consistency_groups.id'),
nullable=False)
share_type_id = Column(String(36),
ForeignKey('share_types.id'),
nullable=False)
consistency_group = orm.relationship(
ConsistencyGroup,
backref="share_types",
foreign_keys=consistency_group_id,
primaryjoin=('and_('
'ConsistencyGroupShareTypeMapping.consistency_group_id == '
'ConsistencyGroups.id,'
'ConsistencyGroupShareTypeMapping.deleted == 0)')
)

class CGSnapshot(BASE, ManilaBase):
"""Represents a cgsnapshot."""
__tablename__ = 'cgsnapshots'
id = Column(String(36), primary_key=True)

consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'))
user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)

name = Column(String(255))
description = Column(String(255))
status = Column(String(255))

class CGSnapshotMembers(BASE, ManilaBase):
__tablename__ = 'cgsnapshot_members'
id = Column(String(36), primary_key=True)
cgsnapshot_id = Column(String(36), ForeignKey('cgsnapshots.id'))
share_id = Column(String(36), ForeignKey('shares.id'))
size = Column(Integer)
status = Column(String(255))
share_proto = Column(String(255))
share_type_id = Column(String(36), ForeignKey('share_types.id'),
nullable=True)
user_id = Column(String(255))
project_id = Column(String(255))
deleted = Column(String(36), default='False')

cgsnapshot = relationship(
ConsistencyGroup,
backref="cgsnapshot_members",
foreign_keys=cgsnapshot_id,
primaryjoin='CGSnapshotMembers.cgsnapshot_id == CGSnapshot.id')

* New fields in Share
consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'), nullable=True)
consistency_group = relationship(
ConsistencyGroup,
backref="shares",
foreign_keys=consistency_group_id,
primaryjoin='Share.consistency_group_id == ConsistencyGroup.id')

source_cgsnapshot_member_id = Column(String(36), nullable=True)

==API==
Schemas: https://wiki.openstack.org/wiki/Manila/design/manila-liberty-consistency-groups/api-schema

'''GET''' /os-consistency-groups/
'''GET''' /os-consistency-groups/detail
'''POST''' /os-consistency-groups
'''GET''' /os-consistency-groups/<id>
'''DELETE''' os-consistency-groups/<id>
'''PUT''' /os-consistency-groups/<id>

'''GET''' /os-cgsnapshots/
'''GET''' /os-cgsnapshots/detail
'''POST''' /os-cgsnapshots
'''PUT''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>/members
'''DELETE''' /os-cgsnapshots/<id>

Admin Actions
'''POST''' /os-consistency-groups/<id>/action # os-reset-status and os-force-delete
'''POST''' /os-cgsnapshots/<id>/action # os-reset-status and os-force-delete

Adds consistency_group_id to POST/shares
Add consistency_group_id query filter to /shares

==Policies==
policy.json - All consistency group policies should default to the default policy
"consistency_group:create" : "",
"consistency_group:delete": "",
"consistency_group:update": "",
"consistency_group:get": "",
"consistency_group:get_all": "",

"consistency_group:create_cgsnapshot" : "",
"consistency_group:delete_cgsnapshot": "",
"consistency_group:get_cgsnapshot": "",
"consistency_group:get_all_cgsnapshots": "",

==Driver API==

update_share_stats should now return 'consistency_group_support' pool/host/None.

def create_consistency_group(self, context, cg_dict, share_server=None):
"""Create a consistency group.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_consistency_group_from_cgsnapshot(self, context, cg_dict,
cgsnapshot_dict, share_server=None):
"""Create a consistency group from a cgsnapshot.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'shares': [<models.Share>], # The new shares being created
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:param cgsnapshot_dict:
EXAMPLE:
{'status': 'available',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': '0',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'consistency_group_id': '4b04fdc3-00b9-4909-ba1a-06e9b3f88b67',
'cgsnapshot_members': [
{'status': 'available',
'share_type_id': '1a9ed31e-ee70-483d-93ba-89690e028d7f',
'share_id': 'e14b5174-e534-4f35-bc4f-fe81c1575d6f',
'user_id': 'a0314a441ca842019b0952224aa39192',
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share': <models.Share>,
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share_proto': 'NFS',
'project_id': '13c0be6290934bd98596cfa004650049',
'cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'deleted_at': None,
'id': '6813e06b-a8f5-4784-b17d-f3e91afa370e',
'size': 1
}
],
'deleted_at': None,
'id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.

share_update_list - a list of dictionaries containing dicts for
every share created in the CG. Any share dicts should at a minimum
contain the 'id' key and 'export_locations'. Export locations
should be in the same format as returned by a share_create. This
list may be empty or None.
EXAMPLE:
[
{'id': 'uuid', 'export_locations': ['export_path']}
]
"""
raise NotImplementedError()

def delete_consistency_group(self, context, cg_dict, share_server=None):
"""Delete a consistency group

:param context:
:param cg_dict:
EXAMPLE:
:return: cg_model_update
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_cgsnapshot(self, context, snap_dict, share_server=None):
"""Create a consistency group snapshot.

:param context:
:param snap_dict:
EXAMPLE:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.

member_update_list - a list of dictionaries containing for every
member of the cgsnapshot. Each dict should contains values to be
updated for teh CGSnapshotMember in the database. This list may be
empty or None.
"""
raise NotImplementedError()

def delete_cgsnapshot(self, context, snap_dict, share_server=None):
"""Delete a consistency group snapshot

:param context:
:param snap_dict:
EXAMPLE:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.
"""
raise NotImplementedError()

== Manila Client ==
* Do we want to alias the commands so that there are Cinder consistent commands and the improved names? (I.E cg-create = consisgroup-create) Perhaps we alias the cinderclient commands to be more like this?
cg-snapshot-create Creates a cgsnapshot.
cg-snapshot-delete Removes one or more cgsnapshots.
cg-snapshot-list Lists all cgsnapshots.
cg-snapshot-show Shows cgsnapshot details.

cg-create Creates a consistency group (--cgsnapshot to create from existing cg snapshot)
cg-delete Removes one or more consistency groups.
cg-list Lists all consistencygroups.
cg-show Shows details of a consistency group.
cg-update Updates a consistencygroup.

create --consistency-group Creates a share and puts it into consistency group

= Outstanding Questions / FAQ =
# How do we avoid confusion if no backends in a deployment support CGs?
#* It appears that Cinder has the policies default to 'nobody' but this seems odd, is there a better way?
#** (ameade) I can't think of a way to have devstack set the policies automatically either.
# Do we need a way to fake CGs with the generic driver?
#* NO, but it will not be a core feature if it cannot be supported in the generic driver.
# Do we need a safeguard for deleting shares that are in a CG? (I.E. protected shares?) Are shares in a CG more valuable than normal shares?
# Should we overload the Snapshots table to include snapshots in a CGSnapshot to avoid adding a new db table? Or continue with having a new table?
# Why not have the cg delete all of the shares in it when deleting the cg?
#* This could cause odd state to occur if the backend fails when deleting shares and it will not be obvious where it failed. This is also extra complexity in the implementation that could instead be hidden by a client.

= Notes on Cinder Impl =

* A consistency group may be specified on volume create
* When a consistency group is created it is scheduled to a pool and all volume created in the CG will go to that pol

Cinder cli:
cgsnapshot-create Creates a cgsnapshot and snapshots of every volume in the CG that show up in snapshot-list.
cgsnapshot-delete Removes one or more cgsnapshots.
cgsnapshot-list Lists all cgsnapshots.
cgsnapshot-show Shows cgsnapshot details. <-- /cgsnapshots/detail?all_tenants=1&name=blah
consisgroup-create Creates a consistency group.
consisgroup-create-from-src Creates a consistency group from a cgsnapshot filled with volumes from the snapshots in the cg.
consisgroup-delete Removes one or more consistency groups.
consisgroup-list Lists all consistencygroups.
consisgroup-show Shows details of a consistency group. <-- /consistencygroups/detail?all_tenants=1&name=blah
consisgroup-update Updates a consistencygroup.

create --consisgroup-id

Cinder API:
'''GET''' /consistencygroups/detail
'''POST''' /consistencygroups
{'consistencygroup':{
'name': <string>,
'description': <string>,
'volume_types': <string>,
'availability_zone': <string>,
}}
'''POST''' /consistencygroups/create_from_src
{'consistencygroup-from-src':{
'name': <string>,
'description': <string>,
'cgsnapshot_id': <string>,
}}
'''GET''' /consistencygroups/<id>
'''POST''' consistencygroups/<id>/delete
{"consistencygroup": {"force": false}}
'''PUT''' /consistencygroups/<id>
{'consistencygroup':{
'name': <string>,
'description': <string>,
'add_volumes': <string>,
'remove_volumes': <string>,
}}

'''GET''' /cgsnapshots/detail
'''GET''' /cgsnapshots/<id>
'''POST''' /cgsnapshots
{'cgsnapshot':{
'consistencygroup_id': <string>,
}}
'''DELETE''' /cgsnapshots/<id>

Adds consistencygroup_id to POST/volumes

Xings commit message:
1) Create a CG, specifying all volume types that can be supported by this
CG. The scheduler chooses a backend that supports all specified volume types.
The CG will be empty when it is first created. Backend needs to report
consistencygroup_support = True. Volume type can have the following in
extra specs: {'capabilities:consistencygroup_support': '<is> True'}.
If consistencygroup_support is not in volume type extra specs, it will be
added to filter_properties by the scheduler to make sure that the scheduler
will select the backend which reports consistency group support capability.

Create CG CLI:
cinder consisgroup-create --volume-type type1,type2 mycg1

This will add a CG entry in the new consistencygroups table.

2) After the CG is created, create a new volume and add to the CG.
Repeat until all volumes are created for the CG.

Create volume CLI (with CG):
cinder create --volume-type type1 --consisgroup-id <CG uuid> 10

This will add a consistencygroup_id foreign key in the new volume
entry in the db.

3) Create a snapshot of the CG (cgsnapshot).

Create cgsnapshot CLI:
cinder cgsnapshot-create <CG uuid>

This will add a cgsnapshot entry in the new cgsnapshots table, create
snapshot for each volume in the CG, and add a cgsnapshot_id foreign key
in each newly created snapshot entry in the db.

===Questions===
# Why does policy.json default to 'nobody' being able to perform CG commands?
# Why are volume types specified when creating a CG?
#* This is because all volumes within a CG need to be on the same backend in order for the backend to honor the CG. This is important to note in the docs so that conflicting volume-types are not specified.
# What occurs when a volume on a different backend is added to a CG via the API?
#* Currently the API ensures that the volume volume_type matches a supported volume_type of the CG and that the CG and volume are on the same host (but can be on different pools).
# Is a volume created with a CG going to end up on the same host?
#* Yes, if a volume has a CG it will be cast to the same pool as the CG.
#* What if the pool is full but there are other available pools on the same backend? Should this fail?
# What happens if a driver does not support create_from_src?
#* The CG goes to ERROR a and volumes are created from the snapshot and also in ERROR status
#* A volume in a consistency group cannot be deleted and a CG with a volume cannot be deleted, therefore a volume must be removed from the CG before it can be deleted. But what if you don't support modifying a CG?
# Why do we need a force delete for CGs?
# What do I do if I do not want to see Snapshots in a CG when doing snapshot-list?
#* I suppose API filtering could allow for someone to not see all of the snapshots if they do not want to.

=== Cinder Bugs to file===
# What happens when you force delete a CG?
#* The volumes remain but still think they are in a CG, which means you still cannot delete the volume
# consisgroup-create-from-src says cgsnapshot is and optional param in cinderclient
# Creating a volume in a CG may not end up in the cg. The only check that is performed is that the volume-type of the volume is supported by the CG but volume-types are not 1-to-1 with backends. There could be backend A and B that both match with volume type 1. A cg is created with volume type 1 and ends up on backend A. Then a volume is created with the CG specified but ends up on backend B

=== Main impl ===
299b99437c2de84222fd585f06c2d05ca213225b ConsistencyGroup: Return 400 instead of 500 for invalid body
adb4c80be82caacad83f1366a4b34e5653fd5dab Create Consistency Group from CG Snapshot API
1a62a6e60fda73bf31256fbf684dc03bb6cf0038 Modify Consistency Group API
fc7a4384be2ccf8ff24f6c0f72c681ad9133801a Add tests for consistency groups DB migration
9082273305b0b9c117eb677a0e34c2ffde4e66f0 Volume types need to be specified when creating CG
764c5ec749821d36bb0215dc6002d3caea95d3b1 Delete consistency group failed
cf961f83ac323dfad1fa5e227d1e502a17529ecc Consistency Groups

=== Ex driver impl ===
8d5b795b37ed021c2639066689645f8aa0b1012f PureISCSIDriver consistency group updates.
39c1a8e08dc854e22ada315a46c20c99df2facf8 Add support to PureISCSIDriver for Consistency Groups
92a817708a3938b1b734d2caaa206b310996d8d0 EMC VMAX driver Kilo update

Manila/design/manila-liberty-consistency-groups

2015-08-10T23:03:36Z

Alex Meade: /* Driver API */

= Consistency Groups =
This document describes the design choices and steps for the consistency group feature in core Manila.

Consistency groups are a mechanism to allow multiple filesystem shares to be guaranteed snapshots are able to be created at the exact same point in time. For example, a database may have its tables, logs, and configuration on separate volumes. If we wanted to restore the database from a previous point in time, it would only make sense to restore the logs, tables, and configuration together from the exact same point in time.

= Manila Core Changes =
The core manila changes are very similar to what has been done in Cinder. At a high level, changes in the DB, share service, drivers, scheduler service and api service will need to be changed.

Phase 1:
* Create/Delete CGs
* Create a share within a CG
* Delete a share that is in a CG
* Snapshot entire CGs
* Create CG from a CGSnapshot

Potential future features:
* Add/remove shares in a CG
* CG migration
* Replication of a CG

==Manila vs Cinder==
* Different CLI syntax
* Restful api tweaks
** Change /create_from_src to just be a cgsnapshot_id in the body on POST /shares
** Deletes to use the DELETE HTTP verb, force_delete not supported until we have a standard API guideline
* At the Driver API, cgsnapshot objects and cg objects (with all the information about what volumes are in the CG) need to be passed to the driver instead of just the cg/cgsnapshot name. This is important for backends that aren't managing a CG construct such as NetApp cDOT as they need to do things such as loop through the snapshots in a cg to delete them.
* The default policy in policy.json for consistency group operations is the default policy versus in Cinder is it 'nobody'
* Unlike Cinder, snapshots in a CGsnapshot are not the same as a normal snapshot. A CGsnapshot is treated as a single unit instead of a collection of snapshots.
* consistency_group_support scheduler capability has multiple values instead of boolean. This is to accommodate backends that require all volumes in a CG to be on the same pool vs. just the same host. It also allows for adding additional values in the future if a backend can support some other paradigm like a CG spanning multiple backends.

== User Workflows ==
* Snapshot multiple shares
*# Create a Consistency Group
*# Create share(s) with a CG specified
*# Create a CGsnapshot

* Create a CG
*# POST to /os-consistency-group
*#* specifiy share_types or the default share type will be used

* Copy a consistency group
*# Create a CGsnapshot
*# Create a CG with the CGSnapshot id

* Delete a CG (and all of its shares)
*# Delete all CG snapshots of the CG
*# Delete all shares in the CG
*# Delete the CG (Only empty CGs can be deleted)

* List shares in a given CG
*# GET /shares?consistencygroup_id=<cg_id>

* List the shares that were captured in a given CGSnapshot
*# GET /cgsnapshots/<id>/members

== System Workflows ==
* Creating a CG
*# Create the CG in the DB
*# Cast the consistency group creation to the scheduler
*## Choose a host for the CG
*# Cast the creation to the share service
*## Get the CG info from DB
*## Call the driver
*## Update database with info returned from driver (ex: CG status)

* Creating a CG from a cgsnapshot
*# Create a new CG in the DB
*# Schedule the CG create to the scheduler with the orig CG host/pool+share_type (depending on capability)
*# For each member of the snapshot, schedule a create share on the same host/pool as CG

* Creating a cg snapshot
*# Create cgsnapshot entity in DB
*# For all shares in CG, create cgsnapshot_member in DB
*# cast create_cgsnapshot to the driver

* Adding a share to a CG
** On share creation:
**# Cast to scheduler with the host of the CG
**#* if perpool capable: Goes to the same pool as CG
**#* if per backend: goes to a pool on the same host that matches a CG share type

* Deleting a share (in a CG)
** Not allowed if the CG has snapshots of the share ( check if the share_id is in the CGSnapshotMembers tables)

== Scheduler ==
manila/scheduler/filter_scheduler.py should now look for consistency_group_support as a capability.

consistency_group_support
default: None 
Values:
* None - No support for CGs
* host - shares in a CG must be on pool(s) on the same host that also match the CG share type
* pool - shares in a CG must live in the same pool as the CG

==DB==
* New Tables:
class ConsistencyGroup(BASE, ManilaBase):
"""Represents a consistencygroup."""
__tablename__ = 'consistency_groups'
id = Column(String(36), primary_key=True)

user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)
deleted = Column(String(36), default='False')

host = Column(String(255))
name = Column(String(255))
description = Column(String(255))
status = Column(String(255))
source_cgsnapshot_id = Column(String(36))

class ConsistencyGroupShareTypeMapping(BASE, ManilaBase):
"""Represents the share types in a consistency group"""
__tablename__ = 'consistency_group_share_type_mappings'
id = Column(Integer, primary_key=True)
deleted = Column(String(36), default='False')
consistency_group_id = Column(String(36),
ForeignKey('consistency_groups.id'),
nullable=False)
share_type_id = Column(String(36),
ForeignKey('share_types.id'),
nullable=False)
consistency_group = orm.relationship(
ConsistencyGroup,
backref="share_types",
foreign_keys=consistency_group_id,
primaryjoin=('and_('
'ConsistencyGroupShareTypeMapping.consistency_group_id == '
'ConsistencyGroups.id,'
'ConsistencyGroupShareTypeMapping.deleted == 0)')
)

class CGSnapshot(BASE, ManilaBase):
"""Represents a cgsnapshot."""
__tablename__ = 'cgsnapshots'
id = Column(String(36), primary_key=True)

consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'))
user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)

name = Column(String(255))
description = Column(String(255))
status = Column(String(255))

class CGSnapshotMembers(BASE, ManilaBase):
__tablename__ = 'cgsnapshot_members'
id = Column(String(36), primary_key=True)
cgsnapshot_id = Column(String(36), ForeignKey('cgsnapshots.id'))
share_id = Column(String(36), ForeignKey('shares.id'))
size = Column(Integer)
status = Column(String(255))
share_proto = Column(String(255))
share_type_id = Column(String(36), ForeignKey('share_types.id'),
nullable=True)
user_id = Column(String(255))
project_id = Column(String(255))
deleted = Column(String(36), default='False')

cgsnapshot = relationship(
ConsistencyGroup,
backref="cgsnapshot_members",
foreign_keys=cgsnapshot_id,
primaryjoin='CGSnapshotMembers.cgsnapshot_id == CGSnapshot.id')

* New fields in Share
consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'), nullable=True)
consistency_group = relationship(
ConsistencyGroup,
backref="shares",
foreign_keys=consistency_group_id,
primaryjoin='Share.consistency_group_id == ConsistencyGroup.id')

source_cgsnapshot_member_id = Column(String(36), nullable=True)

==API==
Schemas: https://wiki.openstack.org/wiki/Manila/design/manila-liberty-consistency-groups/api-schema

'''GET''' /os-consistency-groups/
'''GET''' /os-consistency-groups/detail
'''POST''' /os-consistency-groups
'''GET''' /os-consistency-groups/<id>
'''DELETE''' os-consistency-groups/<id>
'''PUT''' /os-consistency-groups/<id>

'''GET''' /os-cgsnapshots/
'''GET''' /os-cgsnapshots/detail
'''POST''' /os-cgsnapshots
'''PUT''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>/members
'''DELETE''' /os-cgsnapshots/<id>

Adds consistency_group_id to POST/shares
Add consistency_group_id query filter to /shares

==Policies==
policy.json - All consistency group policies should default to the default policy
"consistency_group:create" : "",
"consistency_group:delete": "",
"consistency_group:update": "",
"consistency_group:get": "",
"consistency_group:get_all": "",

"consistency_group:create_cgsnapshot" : "",
"consistency_group:delete_cgsnapshot": "",
"consistency_group:get_cgsnapshot": "",
"consistency_group:get_all_cgsnapshots": "",

==Driver API==

update_share_stats should now return 'consistency_group_support' pool/host/None.

def create_consistency_group(self, context, cg_dict, share_server=None):
"""Create a consistency group.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_consistency_group_from_cgsnapshot(self, context, cg_dict,
cgsnapshot_dict, share_server=None):
"""Create a consistency group from a cgsnapshot.

:param context:
:param cg_dict: The consistency group details
EXAMPLE:
{'status': 'creating',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 15, 14, 6),
'updated_at': None,
'source_cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'host': 'openstack2@cmodeSSVMNFS',
'deleted_at': None,
'shares': [<models.Share>], # The new shares being created
'share_types': [<models.ConsistencyGroupShareTypeMapping>],
'id': 'eda52174-0442-476d-9694-a58327466c14',
'name': None
}
:param cgsnapshot_dict:
EXAMPLE:
{'status': 'available',
'project_id': '13c0be6290934bd98596cfa004650049',
'user_id': 'a0314a441ca842019b0952224aa39192',
'description': None,
'deleted': '0',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'consistency_group_id': '4b04fdc3-00b9-4909-ba1a-06e9b3f88b67',
'cgsnapshot_members': [
{'status': 'available',
'share_type_id': '1a9ed31e-ee70-483d-93ba-89690e028d7f',
'share_id': 'e14b5174-e534-4f35-bc4f-fe81c1575d6f',
'user_id': 'a0314a441ca842019b0952224aa39192',
'deleted': 'False',
'created_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share': <models.Share>,
'updated_at': datetime.datetime(2015, 8, 10, 0, 5, 58),
'share_proto': 'NFS',
'project_id': '13c0be6290934bd98596cfa004650049',
'cgsnapshot_id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'deleted_at': None,
'id': '6813e06b-a8f5-4784-b17d-f3e91afa370e',
'size': 1
}
],
'deleted_at': None,
'id': 'f6aa3b59-57eb-421e-965c-4e182538e36a',
'name': None
}
:return: (cg_model_update, share_update_list)
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.

share_update_list - a list of dictionaries containing dicts for
every share created in the CG. Any share dicts should at a minimum
contain the 'id' key and 'export_locations'. Export locations
should be in the same format as returned by a share_create. This
list may be empty or None.
EXAMPLE:
[
{'id': 'uuid', 'export_locations': ['export_path']}
]
"""
raise NotImplementedError()

def delete_consistency_group(self, context, cg_dict, share_server=None):
"""Delete a consistency group

:param context:
:param cg_dict:
EXAMPLE:
:return: cg_model_update
cg_model_update - a dict containing any values to be updated
for the CG in the database. This value may be None.
"""
raise NotImplementedError()

def create_cgsnapshot(self, context, snap_dict, share_server=None):
"""Create a consistency group snapshot.

:param context:
:param snap_dict:
EXAMPLE:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.

member_update_list - a list of dictionaries containing for every
member of the cgsnapshot. Each dict should contains values to be
updated for teh CGSnapshotMember in the database. This list may be
empty or None.
"""
raise NotImplementedError()

def delete_cgsnapshot(self, context, snap_dict, share_server=None):
"""Delete a consistency group snapshot

:param context:
:param snap_dict:
EXAMPLE:
:return: (cgsnapshot_update, member_update_list)
cgsnapshot_update - a dict containing any values to be updated
for the CGSnapshot in the database. This value may be None.
"""
raise NotImplementedError()

== Manila Client ==
* Do we want to alias the commands so that there are Cinder consistent commands and the improved names? (I.E cg-create = consisgroup-create) Perhaps we alias the cinderclient commands to be more like this?
cg-snapshot-create Creates a cgsnapshot.
cg-snapshot-delete Removes one or more cgsnapshots.
cg-snapshot-list Lists all cgsnapshots.
cg-snapshot-show Shows cgsnapshot details.

cg-create Creates a consistency group (--cgsnapshot to create from existing cg snapshot)
cg-delete Removes one or more consistency groups.
cg-list Lists all consistencygroups.
cg-show Shows details of a consistency group.
cg-update Updates a consistencygroup.

create --consistency-group Creates a share and puts it into consistency group

= Outstanding Questions / FAQ =
# How do we avoid confusion if no backends in a deployment support CGs?
#* It appears that Cinder has the policies default to 'nobody' but this seems odd, is there a better way?
#** (ameade) I can't think of a way to have devstack set the policies automatically either.
# Do we need a way to fake CGs with the generic driver?
#* NO, but it will not be a core feature if it cannot be supported in the generic driver.
# Do we need a safeguard for deleting shares that are in a CG? (I.E. protected shares?) Are shares in a CG more valuable than normal shares?
# Should we overload the Snapshots table to include snapshots in a CGSnapshot to avoid adding a new db table? Or continue with having a new table?
# Why not have the cg delete all of the shares in it when deleting the cg?
#* This could cause odd state to occur if the backend fails when deleting shares and it will not be obvious where it failed. This is also extra complexity in the implementation that could instead be hidden by a client.

= Notes on Cinder Impl =

* A consistency group may be specified on volume create
* When a consistency group is created it is scheduled to a pool and all volume created in the CG will go to that pol

Cinder cli:
cgsnapshot-create Creates a cgsnapshot and snapshots of every volume in the CG that show up in snapshot-list.
cgsnapshot-delete Removes one or more cgsnapshots.
cgsnapshot-list Lists all cgsnapshots.
cgsnapshot-show Shows cgsnapshot details. <-- /cgsnapshots/detail?all_tenants=1&name=blah
consisgroup-create Creates a consistency group.
consisgroup-create-from-src Creates a consistency group from a cgsnapshot filled with volumes from the snapshots in the cg.
consisgroup-delete Removes one or more consistency groups.
consisgroup-list Lists all consistencygroups.
consisgroup-show Shows details of a consistency group. <-- /consistencygroups/detail?all_tenants=1&name=blah
consisgroup-update Updates a consistencygroup.

create --consisgroup-id

Cinder API:
'''GET''' /consistencygroups/detail
'''POST''' /consistencygroups
{'consistencygroup':{
'name': <string>,
'description': <string>,
'volume_types': <string>,
'availability_zone': <string>,
}}
'''POST''' /consistencygroups/create_from_src
{'consistencygroup-from-src':{
'name': <string>,
'description': <string>,
'cgsnapshot_id': <string>,
}}
'''GET''' /consistencygroups/<id>
'''POST''' consistencygroups/<id>/delete
{"consistencygroup": {"force": false}}
'''PUT''' /consistencygroups/<id>
{'consistencygroup':{
'name': <string>,
'description': <string>,
'add_volumes': <string>,
'remove_volumes': <string>,
}}

'''GET''' /cgsnapshots/detail
'''GET''' /cgsnapshots/<id>
'''POST''' /cgsnapshots
{'cgsnapshot':{
'consistencygroup_id': <string>,
}}
'''DELETE''' /cgsnapshots/<id>

Adds consistencygroup_id to POST/volumes

Xings commit message:
1) Create a CG, specifying all volume types that can be supported by this
CG. The scheduler chooses a backend that supports all specified volume types.
The CG will be empty when it is first created. Backend needs to report
consistencygroup_support = True. Volume type can have the following in
extra specs: {'capabilities:consistencygroup_support': '<is> True'}.
If consistencygroup_support is not in volume type extra specs, it will be
added to filter_properties by the scheduler to make sure that the scheduler
will select the backend which reports consistency group support capability.

Create CG CLI:
cinder consisgroup-create --volume-type type1,type2 mycg1

This will add a CG entry in the new consistencygroups table.

2) After the CG is created, create a new volume and add to the CG.
Repeat until all volumes are created for the CG.

Create volume CLI (with CG):
cinder create --volume-type type1 --consisgroup-id <CG uuid> 10

This will add a consistencygroup_id foreign key in the new volume
entry in the db.

3) Create a snapshot of the CG (cgsnapshot).

Create cgsnapshot CLI:
cinder cgsnapshot-create <CG uuid>

This will add a cgsnapshot entry in the new cgsnapshots table, create
snapshot for each volume in the CG, and add a cgsnapshot_id foreign key
in each newly created snapshot entry in the db.

===Questions===
# Why does policy.json default to 'nobody' being able to perform CG commands?
# Why are volume types specified when creating a CG?
#* This is because all volumes within a CG need to be on the same backend in order for the backend to honor the CG. This is important to note in the docs so that conflicting volume-types are not specified.
# What occurs when a volume on a different backend is added to a CG via the API?
#* Currently the API ensures that the volume volume_type matches a supported volume_type of the CG and that the CG and volume are on the same host (but can be on different pools).
# Is a volume created with a CG going to end up on the same host?
#* Yes, if a volume has a CG it will be cast to the same pool as the CG.
#* What if the pool is full but there are other available pools on the same backend? Should this fail?
# What happens if a driver does not support create_from_src?
#* The CG goes to ERROR a and volumes are created from the snapshot and also in ERROR status
#* A volume in a consistency group cannot be deleted and a CG with a volume cannot be deleted, therefore a volume must be removed from the CG before it can be deleted. But what if you don't support modifying a CG?
# Why do we need a force delete for CGs?
# What do I do if I do not want to see Snapshots in a CG when doing snapshot-list?
#* I suppose API filtering could allow for someone to not see all of the snapshots if they do not want to.

=== Cinder Bugs to file===
# What happens when you force delete a CG?
#* The volumes remain but still think they are in a CG, which means you still cannot delete the volume
# consisgroup-create-from-src says cgsnapshot is and optional param in cinderclient
# Creating a volume in a CG may not end up in the cg. The only check that is performed is that the volume-type of the volume is supported by the CG but volume-types are not 1-to-1 with backends. There could be backend A and B that both match with volume type 1. A cg is created with volume type 1 and ends up on backend A. Then a volume is created with the CG specified but ends up on backend B

=== Main impl ===
299b99437c2de84222fd585f06c2d05ca213225b ConsistencyGroup: Return 400 instead of 500 for invalid body
adb4c80be82caacad83f1366a4b34e5653fd5dab Create Consistency Group from CG Snapshot API
1a62a6e60fda73bf31256fbf684dc03bb6cf0038 Modify Consistency Group API
fc7a4384be2ccf8ff24f6c0f72c681ad9133801a Add tests for consistency groups DB migration
9082273305b0b9c117eb677a0e34c2ffde4e66f0 Volume types need to be specified when creating CG
764c5ec749821d36bb0215dc6002d3caea95d3b1 Delete consistency group failed
cf961f83ac323dfad1fa5e227d1e502a17529ecc Consistency Groups

=== Ex driver impl ===
8d5b795b37ed021c2639066689645f8aa0b1012f PureISCSIDriver consistency group updates.
39c1a8e08dc854e22ada315a46c20c99df2facf8 Add support to PureISCSIDriver for Consistency Groups
92a817708a3938b1b734d2caaa206b310996d8d0 EMC VMAX driver Kilo update

Manila/design/manila-liberty-consistency-groups/api-schema

2015-08-06T17:38:25Z

Alex Meade: /* CG Snapshot member */

= Resource schemas=

== Simple CG ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer"
},
"name": {
"description": "A non-unique name for a CG",
"type": "string"
},
},
"required": ["id", "name"]
}

== Detailed CG ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"created_at": {
"description": "The time at which the CG was created",
"type": "string"
},
"status": {
"description": "Status of the CG",
"type": "string",
"enum": ["creating", "available", "deleting", "error"]
},
"project_id": {
"description": "The owner of the CG",
"type": "string"
},
"host": {
"description": "The backend where the CG lives",
"type": "string"
},
"source_cgsnapshot_id": {
"description": "The uuid of the cgsnapshot from which this CG was created.",
"type": ['null', 'string'],
},
"share_types": {
"description": "The share types that shares in the CG are allowed to be a part of",
"type": "array",
"items": {
"type": "string"
},
},
},
"required": ["id", "name", "description", "created_at", "status", "project_id", "host"]
}

== Simple CG Snapshot ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CGsnapshot",
"type": "string"
},
},
"required": ["id", "name"]
}

== Detailed CG Snapshot ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CG snapshot",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG snapshot",
"type": "string"
},
"created_at": {
"description": "The time at which the CG snapshot was created",
"type": "string"
},
"status": {
"description": "Status of the CG snapshot",
"type": "string",
"enum": ["creating", "available", "deleting", "error"]
},
"project_id": {
"description": "The owner of the CG snapshot",
"type": "string"
},
"consistency_group_id": {
"description": "The uuid of the consistency group from which this CG snapshot was created.",
"type": ['null', 'string'],
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
},
"required": ["id", "name", "description", "created_at", "status", "project_id", "host"]
}

== CG Snapshot member ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot Member",
"description": "A resource representing a member of a CGSnapshot",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"cgsnapshot_id": {
"description": "The unique identifier for the parent cg_snapshot",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"share_id": {
"description": "The unique identifier for the share this snapshot was created from.",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"share_type_id": {
"description": "The unique identifier for the share_type of the share this snapshot was created from.",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"size": {
"description": "The storage size of this snapshot",
"type": "integer",
},
"share_protocol": {
"description": "The share protocol of the parent share",
"type": "string",
},
},
}

= Request schemas =

== POST /os-consistency-groups ==
{ "consistency_group" {
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"source_cgsnapshot_id": {
"description": "The uuid of the cgsnapshot from which this CG was created. Cannot be supplied when 'share_types' is provided.",
"type": ['null', 'string'],
},
"share_types": {
"description": "The share types that shares in the CG are allowed to be a part of. Cannot be supplied when 'source_cgsnapshot_id' is provided.",
"type": "string"
},
},
"required": []
}
}

Response: {'consistency_group": detailed CG}

== GET /os-consistency-groups ==
Response:
{'consistency_groups": [Simple CG]}

== GET /os-consistency-groups/detail ==
Response:
{'consistency_groups": [Detailed CG]}

== GET /os-consistency-groups/<id> ==
Response: {'consistency_group": Detailed CG}

== PUT /os-consistency-groups/<id> ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"required": []
}

Response: {'consistency_group": Detailed CG}

== POST /os-cgsnapshots ==
{'cgsnapshot": {
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"consistency_group_id": {
"description": "The uuid of the CG from which to create this CG snapshot.",
"type": 'string',
},
},
"required": ['consistency_group_id']
}}
Response: {'cgsnapshot": detailed CGSnapshot}

== PUT /os-cgsnapshots/<id> ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"required": []
}

Response: {'cgsnapshot": Detailed CGSnapshot}

Manila/design/manila-liberty-consistency-groups/api-schema

2015-08-06T17:36:32Z

Alex Meade: /* Resource schemas */

= Resource schemas=

== Simple CG ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer"
},
"name": {
"description": "A non-unique name for a CG",
"type": "string"
},
},
"required": ["id", "name"]
}

== Detailed CG ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"created_at": {
"description": "The time at which the CG was created",
"type": "string"
},
"status": {
"description": "Status of the CG",
"type": "string",
"enum": ["creating", "available", "deleting", "error"]
},
"project_id": {
"description": "The owner of the CG",
"type": "string"
},
"host": {
"description": "The backend where the CG lives",
"type": "string"
},
"source_cgsnapshot_id": {
"description": "The uuid of the cgsnapshot from which this CG was created.",
"type": ['null', 'string'],
},
"share_types": {
"description": "The share types that shares in the CG are allowed to be a part of",
"type": "array",
"items": {
"type": "string"
},
},
},
"required": ["id", "name", "description", "created_at", "status", "project_id", "host"]
}

== Simple CG Snapshot ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CGsnapshot",
"type": "string"
},
},
"required": ["id", "name"]
}

== Detailed CG Snapshot ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CG snapshot",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG snapshot",
"type": "string"
},
"created_at": {
"description": "The time at which the CG snapshot was created",
"type": "string"
},
"status": {
"description": "Status of the CG snapshot",
"type": "string",
"enum": ["creating", "available", "deleting", "error"]
},
"project_id": {
"description": "The owner of the CG snapshot",
"type": "string"
},
"consistency_group_id": {
"description": "The uuid of the consistency group from which this CG snapshot was created.",
"type": ['null', 'string'],
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
},
"required": ["id", "name", "description", "created_at", "status", "project_id", "host"]
}

== CG Snapshot member ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot Member",
"description": "A resource representing a member of a CGSnapshot",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"cgsnapshot_id": {
"description": "The unique identifier for the parent cg_snapshot",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"share_id": {
"description": "The unique identifier for the share this snapshot was created from.",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
},
"share_type_id": {
"description": "The unique identifier for the share_type of the share this snapshot was created from.",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
},
"size": {
"description": "The storage size of this snapshot",
"type": "integer",
},
},
"share_protocol": {
"description": "The share protocol of the parent share",
"type": "string",
},
},
}

= Request schemas =

== POST /os-consistency-groups ==
{ "consistency_group" {
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"source_cgsnapshot_id": {
"description": "The uuid of the cgsnapshot from which this CG was created. Cannot be supplied when 'share_types' is provided.",
"type": ['null', 'string'],
},
"share_types": {
"description": "The share types that shares in the CG are allowed to be a part of. Cannot be supplied when 'source_cgsnapshot_id' is provided.",
"type": "string"
},
},
"required": []
}
}

Response: {'consistency_group": detailed CG}

== GET /os-consistency-groups ==
Response:
{'consistency_groups": [Simple CG]}

== GET /os-consistency-groups/detail ==
Response:
{'consistency_groups": [Detailed CG]}

== GET /os-consistency-groups/<id> ==
Response: {'consistency_group": Detailed CG}

== PUT /os-consistency-groups/<id> ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"required": []
}

Response: {'consistency_group": Detailed CG}

== POST /os-cgsnapshots ==
{'cgsnapshot": {
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"consistency_group_id": {
"description": "The uuid of the CG from which to create this CG snapshot.",
"type": 'string',
},
},
"required": ['consistency_group_id']
}}
Response: {'cgsnapshot": detailed CGSnapshot}

== PUT /os-cgsnapshots/<id> ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"required": []
}

Response: {'cgsnapshot": Detailed CGSnapshot}

Manila/design/manila-liberty-consistency-groups/api-schema

2015-08-06T17:07:24Z

Alex Meade: /* Request schemas */

= Resource schemas=

== Simple CG ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer"
},
"name": {
"description": "A non-unique name for a CG",
"type": "string"
},
},
"required": ["id", "name"]
}

== Detailed CG ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"created_at": {
"description": "The time at which the CG was created",
"type": "string"
},
"status": {
"description": "Status of the CG",
"type": "string",
"enum": ["creating", "available", "deleting", "error"]
},
"project_id": {
"description": "The owner of the CG",
"type": "string"
},
"host": {
"description": "The backend where the CG lives",
"type": "string"
},
"source_cgsnapshot_id": {
"description": "The uuid of the cgsnapshot from which this CG was created.",
"type": ['null', 'string'],
},
"share_types": {
"description": "The share types that shares in the CG are allowed to be a part of",
"type": "array",
"items": {
"type": "string"
},
},
},
"required": ["id", "name", "description", "created_at", "status", "project_id", "host"]
}

== Simple CG Snapshot ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CGsnapshot",
"type": "string"
},
},
"required": ["id", "name"]
}

== Detailed CG Snapshot ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CG snapshot",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG snapshot",
"type": "string"
},
"created_at": {
"description": "The time at which the CG snapshot was created",
"type": "string"
},
"status": {
"description": "Status of the CG snapshot",
"type": "string",
"enum": ["creating", "available", "deleting", "error"]
},
"project_id": {
"description": "The owner of the CG snapshot",
"type": "string"
},
"consistency_group_id": {
"description": "The uuid of the consistency group from which this CG snapshot was created.",
"type": ['null', 'string'],
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
},
"required": ["id", "name", "description", "created_at", "status", "project_id", "host"]
}

== CG Snapshot member ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot Member",
"description": "A resource representing a member of a CGSnapshot",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"cgsnapshot_id": {
"description": "The unique identifier for the parent cg_snapshot",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"share_id": {
"description": "The unique identifier for the share this snapshot was created from.",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
},
}

= Request schemas =

== POST /os-consistency-groups ==
{ "consistency_group" {
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"source_cgsnapshot_id": {
"description": "The uuid of the cgsnapshot from which this CG was created. Cannot be supplied when 'share_types' is provided.",
"type": ['null', 'string'],
},
"share_types": {
"description": "The share types that shares in the CG are allowed to be a part of. Cannot be supplied when 'source_cgsnapshot_id' is provided.",
"type": "string"
},
},
"required": []
}
}

Response: {'consistency_group": detailed CG}

== GET /os-consistency-groups ==
Response:
{'consistency_groups": [Simple CG]}

== GET /os-consistency-groups/detail ==
Response:
{'consistency_groups": [Detailed CG]}

== GET /os-consistency-groups/<id> ==
Response: {'consistency_group": Detailed CG}

== PUT /os-consistency-groups/<id> ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"required": []
}

Response: {'consistency_group": Detailed CG}

== POST /os-cgsnapshots ==
{'cgsnapshot": {
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"consistency_group_id": {
"description": "The uuid of the CG from which to create this CG snapshot.",
"type": 'string',
},
},
"required": ['consistency_group_id']
}}
Response: {'cgsnapshot": detailed CGSnapshot}

== PUT /os-cgsnapshots/<id> ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"required": []
}

Response: {'cgsnapshot": Detailed CGSnapshot}

Manila/design/manila-liberty-consistency-groups/api-schema

2015-08-06T17:06:24Z

Alex Meade: /* Request schemas */

= Resource schemas=

== Simple CG ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer"
},
"name": {
"description": "A non-unique name for a CG",
"type": "string"
},
},
"required": ["id", "name"]
}

== Detailed CG ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"created_at": {
"description": "The time at which the CG was created",
"type": "string"
},
"status": {
"description": "Status of the CG",
"type": "string",
"enum": ["creating", "available", "deleting", "error"]
},
"project_id": {
"description": "The owner of the CG",
"type": "string"
},
"host": {
"description": "The backend where the CG lives",
"type": "string"
},
"source_cgsnapshot_id": {
"description": "The uuid of the cgsnapshot from which this CG was created.",
"type": ['null', 'string'],
},
"share_types": {
"description": "The share types that shares in the CG are allowed to be a part of",
"type": "array",
"items": {
"type": "string"
},
},
},
"required": ["id", "name", "description", "created_at", "status", "project_id", "host"]
}

== Simple CG Snapshot ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CGsnapshot",
"type": "string"
},
},
"required": ["id", "name"]
}

== Detailed CG Snapshot ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CG snapshot",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG snapshot",
"type": "string"
},
"created_at": {
"description": "The time at which the CG snapshot was created",
"type": "string"
},
"status": {
"description": "Status of the CG snapshot",
"type": "string",
"enum": ["creating", "available", "deleting", "error"]
},
"project_id": {
"description": "The owner of the CG snapshot",
"type": "string"
},
"consistency_group_id": {
"description": "The uuid of the consistency group from which this CG snapshot was created.",
"type": ['null', 'string'],
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
},
"required": ["id", "name", "description", "created_at", "status", "project_id", "host"]
}

== CG Snapshot member ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot Member",
"description": "A resource representing a member of a CGSnapshot",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"cgsnapshot_id": {
"description": "The unique identifier for the parent cg_snapshot",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"share_id": {
"description": "The unique identifier for the share this snapshot was created from.",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
},
}

= Request schemas =

== POST /os-consistency-groups ==
{ "consistency_group" {
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"source_cgsnapshot_id": {
"description": "The uuid of the cgsnapshot from which this CG was created. Cannot be supplied when 'share_types' is provided.",
"type": ['null', 'string'],
},
"share_types": {
"description": "The share types that shares in the CG are allowed to be a part of. Cannot be supplied when 'source_cgsnapshot_id' is provided.",
"type": "string"
},
},
"required": []
}
}

Response:
{'consistency_group": detailed CG}

== GET /os-consistency-groups ==
Response:
{'consistency_groups": [Simple CG]}

== GET /os-consistency-groups/detail ==
Response:
{'consistency_groups": [Detailed CG]}

== GET /os-consistency-groups/<id> ==
Response: {'consistency_group": Detailed CG}

== PUT /os-consistency-groups/<id> ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"required": []
}

Response: {'consistency_group": Detailed CG}

== POST /os-cgsnapshots ==
{'cgsnapshot": {
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"consistency_group_id": {
"description": "The uuid of the CG from which to create this CG snapshot.",
"type": 'string',
},
},
"required": ['consistency_group_id']
}}
Response: {'cgsnapshot": detailed CGSnapshot}

== PUT /os-cgsnapshots/<id> ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"required": []
}

Response: {'cgsnapshot": Detailed CGSnapshot}

Manila/design/manila-liberty-consistency-groups/api-schema

2015-08-06T16:59:34Z

Alex Meade: /* Simple CG Snapshot */

= Resource schemas=

== Simple CG ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer"
},
"name": {
"description": "A non-unique name for a CG",
"type": "string"
},
},
"required": ["id", "name"]
}

== Detailed CG ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"created_at": {
"description": "The time at which the CG was created",
"type": "string"
},
"status": {
"description": "Status of the CG",
"type": "string",
"enum": ["creating", "available", "deleting", "error"]
},
"project_id": {
"description": "The owner of the CG",
"type": "string"
},
"host": {
"description": "The backend where the CG lives",
"type": "string"
},
"source_cgsnapshot_id": {
"description": "The uuid of the cgsnapshot from which this CG was created.",
"type": ['null', 'string'],
},
"share_types": {
"description": "The share types that shares in the CG are allowed to be a part of",
"type": "array",
"items": {
"type": "string"
},
},
},
"required": ["id", "name", "description", "created_at", "status", "project_id", "host"]
}

== Simple CG Snapshot ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CGsnapshot",
"type": "string"
},
},
"required": ["id", "name"]
}

== Detailed CG Snapshot ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CG snapshot",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG snapshot",
"type": "string"
},
"created_at": {
"description": "The time at which the CG snapshot was created",
"type": "string"
},
"status": {
"description": "Status of the CG snapshot",
"type": "string",
"enum": ["creating", "available", "deleting", "error"]
},
"project_id": {
"description": "The owner of the CG snapshot",
"type": "string"
},
"consistency_group_id": {
"description": "The uuid of the consistency group from which this CG snapshot was created.",
"type": ['null', 'string'],
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
},
"required": ["id", "name", "description", "created_at", "status", "project_id", "host"]
}

== CG Snapshot member ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot Member",
"description": "A resource representing a member of a CGSnapshot",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"cgsnapshot_id": {
"description": "The unique identifier for the parent cg_snapshot",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"share_id": {
"description": "The unique identifier for the share this snapshot was created from.",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
},
}

= Request schemas =

== POST /os-consistency-groups ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"source_cgsnapshot_id": {
"description": "The uuid of the cgsnapshot from which this CG was created. Cannot be supplied when 'share_types' is provided.",
"type": ['null', 'string'],
},
"share_types": {
"description": "The share types that shares in the CG are allowed to be a part of. Cannot be supplied when 'source_cgsnapshot_id' is provided.",
"type": "string"
},
},
"required": []
}

Response: detailed CG

== GET /os-consistency-groups ==
Response: Simple CG

== GET /os-consistency-groups/detail ==
Response: Detailed CG

== GET /os-consistency-groups/<id> ==
Response: Detailed CG

== PUT /os-consistency-groups/<id> ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"required": []
}

Response: Detailed CG

== POST /os-cgsnapshots ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"consistency_group_id": {
"description": "The uuid of the CG from which to create this CG snapshot.",
"type": 'string',
},
},
"required": ['consistency_group_id']
}

Response: detailed CGSnapshot

== PUT /os-cgsnapshots/<id> ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"required": []
}

Response: Detailed CGSnapshot

Manila/design/manila-liberty-consistency-groups/api-schema

2015-08-06T16:58:36Z

Alex Meade: /* Detailed CG Snapshot */

= Resource schemas=

== Simple CG ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer"
},
"name": {
"description": "A non-unique name for a CG",
"type": "string"
},
},
"required": ["id", "name"]
}

== Detailed CG ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"created_at": {
"description": "The time at which the CG was created",
"type": "string"
},
"status": {
"description": "Status of the CG",
"type": "string",
"enum": ["creating", "available", "deleting", "error"]
},
"project_id": {
"description": "The owner of the CG",
"type": "string"
},
"host": {
"description": "The backend where the CG lives",
"type": "string"
},
"source_cgsnapshot_id": {
"description": "The uuid of the cgsnapshot from which this CG was created.",
"type": ['null', 'string'],
},
"share_types": {
"description": "The share types that shares in the CG are allowed to be a part of",
"type": "array",
"items": {
"type": "string"
},
},
},
"required": ["id", "name", "description", "created_at", "status", "project_id", "host"]
}

== Simple CG Snapshot ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer"
},
"name": {
"description": "A non-unique name for a CGsnapshot",
"type": "string"
},
},
"required": ["id", "name"]
}

== Detailed CG Snapshot ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CG snapshot",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG snapshot",
"type": "string"
},
"created_at": {
"description": "The time at which the CG snapshot was created",
"type": "string"
},
"status": {
"description": "Status of the CG snapshot",
"type": "string",
"enum": ["creating", "available", "deleting", "error"]
},
"project_id": {
"description": "The owner of the CG snapshot",
"type": "string"
},
"consistency_group_id": {
"description": "The uuid of the consistency group from which this CG snapshot was created.",
"type": ['null', 'string'],
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
},
"required": ["id", "name", "description", "created_at", "status", "project_id", "host"]
}

== CG Snapshot member ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot Member",
"description": "A resource representing a member of a CGSnapshot",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"cgsnapshot_id": {
"description": "The unique identifier for the parent cg_snapshot",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"share_id": {
"description": "The unique identifier for the share this snapshot was created from.",
"type": "string",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
},
}

= Request schemas =

== POST /os-consistency-groups ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"source_cgsnapshot_id": {
"description": "The uuid of the cgsnapshot from which this CG was created. Cannot be supplied when 'share_types' is provided.",
"type": ['null', 'string'],
},
"share_types": {
"description": "The share types that shares in the CG are allowed to be a part of. Cannot be supplied when 'source_cgsnapshot_id' is provided.",
"type": "string"
},
},
"required": []
}

Response: detailed CG

== GET /os-consistency-groups ==
Response: Simple CG

== GET /os-consistency-groups/detail ==
Response: Detailed CG

== GET /os-consistency-groups/<id> ==
Response: Detailed CG

== PUT /os-consistency-groups/<id> ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"required": []
}

Response: Detailed CG

== POST /os-cgsnapshots ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"consistency_group_id": {
"description": "The uuid of the CG from which to create this CG snapshot.",
"type": 'string',
},
},
"required": ['consistency_group_id']
}

Response: detailed CGSnapshot

== PUT /os-cgsnapshots/<id> ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"required": []
}

Response: Detailed CGSnapshot

Manila/design/manila-liberty-consistency-groups/api-schema

2015-08-06T16:51:00Z

Alex Meade: /* POST /os-cgsnapshots */

= Resource schemas=

== Simple CG ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer"
},
"name": {
"description": "A non-unique name for a CG",
"type": "string"
},
},
"required": ["id", "name"]
}

== Detailed CG ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"created_at": {
"description": "The time at which the CG was created",
"type": "string"
},
"status": {
"description": "Status of the CG",
"type": "string",
"enum": ["creating", "available", "deleting", "error"]
},
"project_id": {
"description": "The owner of the CG",
"type": "string"
},
"host": {
"description": "The backend where the CG lives",
"type": "string"
},
"source_cgsnapshot_id": {
"description": "The uuid of the cgsnapshot from which this CG was created.",
"type": ['null', 'string'],
},
"share_types": {
"description": "The share types that shares in the CG are allowed to be a part of",
"type": "array",
"items": {
"type": "string"
},
},
},
"required": ["id", "name", "description", "created_at", "status", "project_id", "host"]
}

== Simple CG Snapshot ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer"
},
"name": {
"description": "A non-unique name for a CGsnapshot",
"type": "string"
},
},
"required": ["id", "name"]
}

== Detailed CG Snapshot ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CG snapshot",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG snapshot",
"type": "string"
},
"created_at": {
"description": "The time at which the CG snapshot was created",
"type": "string"
},
"status": {
"description": "Status of the CG snapshot",
"type": "string",
"enum": ["creating", "available", "deleting", "error"]
},
"project_id": {
"description": "The owner of the CG snapshot",
"type": "string"
},
"consistency_group_id": {
"description": "The uuid of the consistency group from which this CG snapshot was created.",
"type": ['null', 'string'],
},
},
"required": ["id", "name", "description", "created_at", "status", "project_id", "host"]
}

= Request schemas =

== POST /os-consistency-groups ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"source_cgsnapshot_id": {
"description": "The uuid of the cgsnapshot from which this CG was created. Cannot be supplied when 'share_types' is provided.",
"type": ['null', 'string'],
},
"share_types": {
"description": "The share types that shares in the CG are allowed to be a part of. Cannot be supplied when 'source_cgsnapshot_id' is provided.",
"type": "string"
},
},
"required": []
}

Response: detailed CG

== GET /os-consistency-groups ==
Response: Simple CG

== GET /os-consistency-groups/detail ==
Response: Detailed CG

== GET /os-consistency-groups/<id> ==
Response: Detailed CG

== PUT /os-consistency-groups/<id> ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"required": []
}

Response: Detailed CG

== POST /os-cgsnapshots ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"consistency_group_id": {
"description": "The uuid of the CG from which to create this CG snapshot.",
"type": 'string',
},
},
"required": ['consistency_group_id']
}

Response: detailed CGSnapshot

== PUT /os-cgsnapshots/<id> ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"required": []
}

Response: Detailed CG

Manila/design/manila-liberty-consistency-groups/api-schema

2015-08-06T16:50:36Z

Alex Meade: /* POST /os-cgsnapshots */

= Resource schemas=

== Simple CG ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer"
},
"name": {
"description": "A non-unique name for a CG",
"type": "string"
},
},
"required": ["id", "name"]
}

== Detailed CG ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"created_at": {
"description": "The time at which the CG was created",
"type": "string"
},
"status": {
"description": "Status of the CG",
"type": "string",
"enum": ["creating", "available", "deleting", "error"]
},
"project_id": {
"description": "The owner of the CG",
"type": "string"
},
"host": {
"description": "The backend where the CG lives",
"type": "string"
},
"source_cgsnapshot_id": {
"description": "The uuid of the cgsnapshot from which this CG was created.",
"type": ['null', 'string'],
},
"share_types": {
"description": "The share types that shares in the CG are allowed to be a part of",
"type": "array",
"items": {
"type": "string"
},
},
},
"required": ["id", "name", "description", "created_at", "status", "project_id", "host"]
}

== Simple CG Snapshot ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer"
},
"name": {
"description": "A non-unique name for a CGsnapshot",
"type": "string"
},
},
"required": ["id", "name"]
}

== Detailed CG Snapshot ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CG snapshot",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG snapshot",
"type": "string"
},
"created_at": {
"description": "The time at which the CG snapshot was created",
"type": "string"
},
"status": {
"description": "Status of the CG snapshot",
"type": "string",
"enum": ["creating", "available", "deleting", "error"]
},
"project_id": {
"description": "The owner of the CG snapshot",
"type": "string"
},
"consistency_group_id": {
"description": "The uuid of the consistency group from which this CG snapshot was created.",
"type": ['null', 'string'],
},
},
"required": ["id", "name", "description", "created_at", "status", "project_id", "host"]
}

= Request schemas =

== POST /os-consistency-groups ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"source_cgsnapshot_id": {
"description": "The uuid of the cgsnapshot from which this CG was created. Cannot be supplied when 'share_types' is provided.",
"type": ['null', 'string'],
},
"share_types": {
"description": "The share types that shares in the CG are allowed to be a part of. Cannot be supplied when 'source_cgsnapshot_id' is provided.",
"type": "string"
},
},
"required": []
}

Response: detailed CG

== GET /os-consistency-groups ==
Response: Simple CG

== GET /os-consistency-groups/detail ==
Response: Detailed CG

== GET /os-consistency-groups/<id> ==
Response: Detailed CG

== PUT /os-consistency-groups/<id> ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"required": []
}

Response: Detailed CG

== POST /os-cgsnapshots ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"consistency_group_id": {
"description": "The uuid of the CG from which to create this CG snapshot.",
"type": 'string',
},
},
"required": ['consistency_group_id']
}

Response: detailed CG

== PUT /os-cgsnapshots/<id> ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"required": []
}

Response: Detailed CG

ThirdPartySystems/NetApp FC CI

2015-08-06T15:44:09Z

Alex Meade:

{{ThirdPartySystemInfo|name=NetApp FC CI
|account=netapp-fc-ci
|contact= xdl-openstack-jenkins@netapp.com, andrew.kerr@netapp.com (akerr), alex.meade@netapp.com (ameade)
|intent=Test Openstack with NetApp Fibre Channel Cinder drivers using storage arrays
|structure=Nodepool, Zuul, Jenkins, modified upstream components with local configuration
|method=Using Zuul to connect to Gerrit
|programs=Cinder
|status= On hold until https://bugs.launchpad.net/nova/+bug/1482092 is fixed}}

Manila/design/manila-liberty-consistency-groups

2015-08-01T19:25:14Z

Alex Meade: /* User Workflows */

= Consistency Groups =
This document describes the design choices and steps for the consistency group feature in core Manila.

Consistency groups are a mechanism to allow multiple filesystem shares to be guaranteed snapshots are able to be created at the exact same point in time. For example, a database may have its tables, logs, and configuration on separate volumes. If we wanted to restore the database from a previous point in time, it would only make sense to restore the logs, tables, and configuration together from the exact same point in time.

= Manila Core Changes =
The core manila changes are very similar to what has been done in Cinder. At a high level, changes in the DB, share service, drivers, scheduler service and api service will need to be changed.

Phase 1:
* Create/Delete CGs
* Create a share within a CG
* Delete a share that is in a CG
* Snapshot entire CGs
* Create CG from a CGSnapshot

Potential future features:
* Add/remove shares in a CG
* CG migration
* Replication of a CG

==Manila vs Cinder==
* Different CLI syntax
* Restful api tweaks
** Change /create_from_src to just be a cgsnapshot_id in the body on POST /shares
** Deletes to use the DELETE HTTP verb, force_delete not supported until we have a standard API guideline
* At the Driver API, cgsnapshot objects and cg objects (with all the information about what volumes are in the CG) need to be passed to the driver instead of just the cg/cgsnapshot name. This is important for backends that aren't managing a CG construct such as NetApp cDOT as they need to do things such as loop through the snapshots in a cg to delete them.
* The default policy in policy.json for consistency group operations is the default policy versus in Cinder is it 'nobody'
* Unlike Cinder, snapshots in a CGsnapshot are not the same as a normal snapshot. A CGsnapshot is treated as a single unit instead of a collection of snapshots.
* consistency_group_support scheduler capability has multiple values instead of boolean. This is to accommodate backends that require all volumes in a CG to be on the same pool vs. just the same host. It also allows for adding additional values in the future if a backend can support some other paradigm like a CG spanning multiple backends.

== User Workflows ==
* Snapshot multiple shares
*# Create a Consistency Group
*# Create share(s) with a CG specified
*# Create a CGsnapshot

* Create a CG
*# POST to /os-consistency-group
*#* specifiy share_types or the default share type will be used

* Copy a consistency group
*# Create a CGsnapshot
*# Create a CG with the CGSnapshot id

* Delete a CG (and all of its shares)
*# Delete all CG snapshots of the CG
*# Delete all shares in the CG
*# Delete the CG (Only empty CGs can be deleted)

* List shares in a given CG
*# GET /shares?consistencygroup_id=<cg_id>

* List the shares that were captured in a given CGSnapshot
*# GET /cgsnapshots/<id>/members

== System Workflows ==
* Creating a CG
*# Create the CG in the DB
*# Cast the consistency group creation to the scheduler
*## Choose a host for the CG
*# Cast the creation to the share service
*## Get the CG info from DB
*## Call the driver
*## Update database with info returned from driver (ex: CG status)

* Creating a CG from a cgsnapshot
*# Create a new CG in the DB
*# Schedule the CG create to the scheduler with the orig CG host/pool+share_type (depending on capability)
*# For each member of the snapshot, schedule a create share on the same host/pool as CG

* Creating a cg snapshot
*# Create cgsnapshot entity in DB
*# For all shares in CG, create cgsnapshot_member in DB
*# cast create_cgsnapshot to the driver

* Adding a share to a CG
** On share creation:
**# Cast to scheduler with the host of the CG
**#* if perpool capable: Goes to the same pool as CG
**#* if per backend: goes to a pool on the same host that matches a CG share type

* Deleting a share (in a CG)
** Not allowed if the CG has snapshots of the share ( check if the share_id is in the CGSnapshotMembers tables)

== Scheduler ==
manila/scheduler/filter_scheduler.py should now look for consistency_group_support as a capability.

consistency_group_support
default: None 
Values:
* None - No support for CGs
* host - shares in a CG must be on pool(s) on the same host that also match the CG share type
* pool - shares in a CG must live in the same pool as the CG

==DB==
* New Tables:
class ConsistencyGroup(BASE, ManilaBase):
"""Represents a consistencygroup."""
__tablename__ = 'consistency_groups'
id = Column(String(36), primary_key=True)

user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)
deleted = Column(String(36), default='False')

host = Column(String(255))
name = Column(String(255))
description = Column(String(255))
status = Column(String(255))
source_cgsnapshot_id = Column(String(36))

class ConsistencyGroupShareTypeMapping(BASE, ManilaBase):
"""Represents the share types in a consistency group"""
__tablename__ = 'consistency_group_share_type_mappings'
id = Column(Integer, primary_key=True)
deleted = Column(String(36), default='False')
consistency_group_id = Column(String(36),
ForeignKey('consistency_groups.id'),
nullable=False)
share_type_id = Column(String(36),
ForeignKey('share_types.id'),
nullable=False)
consistency_group = orm.relationship(
ConsistencyGroup,
backref="share_types",
foreign_keys=consistency_group_id,
primaryjoin=('and_('
'ConsistencyGroupShareTypeMapping.consistency_group_id == '
'ConsistencyGroups.id,'
'ConsistencyGroupShareTypeMapping.deleted == 0)')
)

class CGSnapshot(BASE, ManilaBase):
"""Represents a cgsnapshot."""
__tablename__ = 'cgsnapshots'
id = Column(String(36), primary_key=True)

consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'))
user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)

name = Column(String(255))
description = Column(String(255))
status = Column(String(255))

class CGSnapshotMembers(BASE, ManilaBase):
__tablename__ = 'cgsnapshot_members'
id = Column(String(36), primary_key=True)
cgsnapshot_id = Column(String(36), ForeignKey('cgsnapshots.id'))
share_id = Column(String(36), ForeignKey('shares.id'))
size = Column(Integer)
status = Column(String(255))
share_proto = Column(String(255))
share_type_id = Column(String(36), ForeignKey('share_types.id'),
nullable=True)
user_id = Column(String(255))
project_id = Column(String(255))
deleted = Column(String(36), default='False')

cgsnapshot = relationship(
ConsistencyGroup,
backref="cgsnapshot_members",
foreign_keys=cgsnapshot_id,
primaryjoin='CGSnapshotMembers.cgsnapshot_id == CGSnapshot.id')

* New fields in Share
consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'), nullable=True)
consistency_group = relationship(
ConsistencyGroup,
backref="shares",
foreign_keys=consistency_group_id,
primaryjoin='Share.consistency_group_id == ConsistencyGroup.id')

source_cgsnapshot_member_id = Column(String(36), nullable=True)

==API==
Schemas: https://wiki.openstack.org/wiki/Manila/design/manila-liberty-consistency-groups/api-schema

'''GET''' /os-consistency-groups/
'''GET''' /os-consistency-groups/detail
'''POST''' /os-consistency-groups
'''GET''' /os-consistency-groups/<id>
'''DELETE''' os-consistency-groups/<id>
'''PUT''' /os-consistency-groups/<id>

'''GET''' /os-cgsnapshots/
'''GET''' /os-cgsnapshots/detail
'''POST''' /os-cgsnapshots
'''PUT''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>/members
'''DELETE''' /os-cgsnapshots/<id>

Adds consistency_group_id to POST/shares
Add consistency_group_id query filter to /shares

==Policies==
policy.json - All consistency group policies should default to the default policy
"consistency_group:create" : "",
"consistency_group:delete": "",
"consistency_group:update": "",
"consistency_group:get": "",
"consistency_group:get_all": "",

"consistency_group:create_cgsnapshot" : "",
"consistency_group:delete_cgsnapshot": "",
"consistency_group:get_cgsnapshot": "",
"consistency_group:get_all_cgsnapshots": "",

==Driver API==

update_share_stats should now return 'consistency_group_support' pool/host/None.

def create_consistency_group(self, context, cg):

def delete_consistency_group(self, context, cg):

def create_cgsnapshot(self, context, cgsnapshot):

def delete_cgsnapshot(self, context, cgsnapshot):

Phase 2:

def update_consistency_group(self, context, cg):

== Manila Client ==
* Do we want to alias the commands so that there are Cinder consistent commands and the improved names? (I.E cg-create = consisgroup-create) Perhaps we alias the cinderclient commands to be more like this?
cg-snapshot-create Creates a cgsnapshot.
cg-snapshot-delete Removes one or more cgsnapshots.
cg-snapshot-list Lists all cgsnapshots.
cg-snapshot-show Shows cgsnapshot details.

cg-create Creates a consistency group (--cgsnapshot to create from existing cg snapshot)
cg-delete Removes one or more consistency groups.
cg-list Lists all consistencygroups.
cg-show Shows details of a consistency group.
cg-update Updates a consistencygroup.

create --consistency-group Creates a share and puts it into consistency group

= Outstanding Questions / FAQ =
# How do we avoid confusion if no backends in a deployment support CGs?
#* It appears that Cinder has the policies default to 'nobody' but this seems odd, is there a better way?
#** (ameade) I can't think of a way to have devstack set the policies automatically either.
# Do we need a way to fake CGs with the generic driver?
#* NO, but it will not be a core feature if it cannot be supported in the generic driver.
# Do we need a safeguard for deleting shares that are in a CG? (I.E. protected shares?) Are shares in a CG more valuable than normal shares?
# Should we overload the Snapshots table to include snapshots in a CGSnapshot to avoid adding a new db table? Or continue with having a new table?
# Why not have the cg delete all of the shares in it when deleting the cg?
#* This could cause odd state to occur if the backend fails when deleting shares and it will not be obvious where it failed. This is also extra complexity in the implementation that could instead be hidden by a client.

= Notes on Cinder Impl =

* A consistency group may be specified on volume create
* When a consistency group is created it is scheduled to a pool and all volume created in the CG will go to that pol

Cinder cli:
cgsnapshot-create Creates a cgsnapshot and snapshots of every volume in the CG that show up in snapshot-list.
cgsnapshot-delete Removes one or more cgsnapshots.
cgsnapshot-list Lists all cgsnapshots.
cgsnapshot-show Shows cgsnapshot details. <-- /cgsnapshots/detail?all_tenants=1&name=blah
consisgroup-create Creates a consistency group.
consisgroup-create-from-src Creates a consistency group from a cgsnapshot filled with volumes from the snapshots in the cg.
consisgroup-delete Removes one or more consistency groups.
consisgroup-list Lists all consistencygroups.
consisgroup-show Shows details of a consistency group. <-- /consistencygroups/detail?all_tenants=1&name=blah
consisgroup-update Updates a consistencygroup.

create --consisgroup-id

Cinder API:
'''GET''' /consistencygroups/detail
'''POST''' /consistencygroups
{'consistencygroup':{
'name': <string>,
'description': <string>,
'volume_types': <string>,
'availability_zone': <string>,
}}
'''POST''' /consistencygroups/create_from_src
{'consistencygroup-from-src':{
'name': <string>,
'description': <string>,
'cgsnapshot_id': <string>,
}}
'''GET''' /consistencygroups/<id>
'''POST''' consistencygroups/<id>/delete
{"consistencygroup": {"force": false}}
'''PUT''' /consistencygroups/<id>
{'consistencygroup':{
'name': <string>,
'description': <string>,
'add_volumes': <string>,
'remove_volumes': <string>,
}}

'''GET''' /cgsnapshots/detail
'''GET''' /cgsnapshots/<id>
'''POST''' /cgsnapshots
{'cgsnapshot':{
'consistencygroup_id': <string>,
}}
'''DELETE''' /cgsnapshots/<id>

Adds consistencygroup_id to POST/volumes

Xings commit message:
1) Create a CG, specifying all volume types that can be supported by this
CG. The scheduler chooses a backend that supports all specified volume types.
The CG will be empty when it is first created. Backend needs to report
consistencygroup_support = True. Volume type can have the following in
extra specs: {'capabilities:consistencygroup_support': '<is> True'}.
If consistencygroup_support is not in volume type extra specs, it will be
added to filter_properties by the scheduler to make sure that the scheduler
will select the backend which reports consistency group support capability.

Create CG CLI:
cinder consisgroup-create --volume-type type1,type2 mycg1

This will add a CG entry in the new consistencygroups table.

2) After the CG is created, create a new volume and add to the CG.
Repeat until all volumes are created for the CG.

Create volume CLI (with CG):
cinder create --volume-type type1 --consisgroup-id <CG uuid> 10

This will add a consistencygroup_id foreign key in the new volume
entry in the db.

3) Create a snapshot of the CG (cgsnapshot).

Create cgsnapshot CLI:
cinder cgsnapshot-create <CG uuid>

This will add a cgsnapshot entry in the new cgsnapshots table, create
snapshot for each volume in the CG, and add a cgsnapshot_id foreign key
in each newly created snapshot entry in the db.

===Questions===
# Why does policy.json default to 'nobody' being able to perform CG commands?
# Why are volume types specified when creating a CG?
#* This is because all volumes within a CG need to be on the same backend in order for the backend to honor the CG. This is important to note in the docs so that conflicting volume-types are not specified.
# What occurs when a volume on a different backend is added to a CG via the API?
#* Currently the API ensures that the volume volume_type matches a supported volume_type of the CG and that the CG and volume are on the same host (but can be on different pools).
# Is a volume created with a CG going to end up on the same host?
#* Yes, if a volume has a CG it will be cast to the same pool as the CG.
#* What if the pool is full but there are other available pools on the same backend? Should this fail?
# What happens if a driver does not support create_from_src?
#* The CG goes to ERROR a and volumes are created from the snapshot and also in ERROR status
#* A volume in a consistency group cannot be deleted and a CG with a volume cannot be deleted, therefore a volume must be removed from the CG before it can be deleted. But what if you don't support modifying a CG?
# Why do we need a force delete for CGs?
# What do I do if I do not want to see Snapshots in a CG when doing snapshot-list?
#* I suppose API filtering could allow for someone to not see all of the snapshots if they do not want to.

=== Cinder Bugs to file===
# What happens when you force delete a CG?
#* The volumes remain but still think they are in a CG, which means you still cannot delete the volume
# consisgroup-create-from-src says cgsnapshot is and optional param in cinderclient
# Creating a volume in a CG may not end up in the cg. The only check that is performed is that the volume-type of the volume is supported by the CG but volume-types are not 1-to-1 with backends. There could be backend A and B that both match with volume type 1. A cg is created with volume type 1 and ends up on backend A. Then a volume is created with the CG specified but ends up on backend B

=== Main impl ===
299b99437c2de84222fd585f06c2d05ca213225b ConsistencyGroup: Return 400 instead of 500 for invalid body
adb4c80be82caacad83f1366a4b34e5653fd5dab Create Consistency Group from CG Snapshot API
1a62a6e60fda73bf31256fbf684dc03bb6cf0038 Modify Consistency Group API
fc7a4384be2ccf8ff24f6c0f72c681ad9133801a Add tests for consistency groups DB migration
9082273305b0b9c117eb677a0e34c2ffde4e66f0 Volume types need to be specified when creating CG
764c5ec749821d36bb0215dc6002d3caea95d3b1 Delete consistency group failed
cf961f83ac323dfad1fa5e227d1e502a17529ecc Consistency Groups

=== Ex driver impl ===
8d5b795b37ed021c2639066689645f8aa0b1012f PureISCSIDriver consistency group updates.
39c1a8e08dc854e22ada315a46c20c99df2facf8 Add support to PureISCSIDriver for Consistency Groups
92a817708a3938b1b734d2caaa206b310996d8d0 EMC VMAX driver Kilo update

Manila/design/manila-liberty-consistency-groups/api-schema

2015-08-01T19:03:54Z

Alex Meade: /* Request schemas */

= Resource schemas=

== Simple CG ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer"
},
"name": {
"description": "A non-unique name for a CG",
"type": "string"
},
},
"required": ["id", "name"]
}

== Detailed CG ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"created_at": {
"description": "The time at which the CG was created",
"type": "string"
},
"status": {
"description": "Status of the CG",
"type": "string",
"enum": ["creating", "available", "deleting", "error"]
},
"project_id": {
"description": "The owner of the CG",
"type": "string"
},
"host": {
"description": "The backend where the CG lives",
"type": "string"
},
"source_cgsnapshot_id": {
"description": "The uuid of the cgsnapshot from which this CG was created.",
"type": ['null', 'string'],
},
"share_types": {
"description": "The share types that shares in the CG are allowed to be a part of",
"type": "array",
"items": {
"type": "string"
},
},
},
"required": ["id", "name", "description", "created_at", "status", "project_id", "host"]
}

== Simple CG Snapshot ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer"
},
"name": {
"description": "A non-unique name for a CGsnapshot",
"type": "string"
},
},
"required": ["id", "name"]
}

== Detailed CG Snapshot ==

{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"id": {
"description": "The unique identifier for a product",
"type": "integer",
'pattern': ('^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}'
'-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$'),
},
"name": {
"description": "A non-unique name for a CG snapshot",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG snapshot",
"type": "string"
},
"created_at": {
"description": "The time at which the CG snapshot was created",
"type": "string"
},
"status": {
"description": "Status of the CG snapshot",
"type": "string",
"enum": ["creating", "available", "deleting", "error"]
},
"project_id": {
"description": "The owner of the CG snapshot",
"type": "string"
},
"consistency_group_id": {
"description": "The uuid of the consistency group from which this CG snapshot was created.",
"type": ['null', 'string'],
},
},
"required": ["id", "name", "description", "created_at", "status", "project_id", "host"]
}

= Request schemas =

== POST /os-consistency-groups ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"source_cgsnapshot_id": {
"description": "The uuid of the cgsnapshot from which this CG was created. Cannot be supplied when 'share_types' is provided.",
"type": ['null', 'string'],
},
"share_types": {
"description": "The share types that shares in the CG are allowed to be a part of. Cannot be supplied when 'source_cgsnapshot_id' is provided.",
"type": "string"
},
},
"required": []
}

Response: detailed CG

== GET /os-consistency-groups ==
Response: Simple CG

== GET /os-consistency-groups/detail ==
Response: Detailed CG

== GET /os-consistency-groups/<id> ==
Response: Detailed CG

== PUT /os-consistency-groups/<id> ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group (CG)",
"description": "A resource representing a group of shares that can be snapshotted in unison",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"required": []
}

Response: Detailed CG

== POST /os-cgsnapshots ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"consistency_group_id": {
"description": "The uuid of the CG from which to create this CG snapshot.",
"type": 'string',
},
},
"required": []
}

Response: detailed CG

== PUT /os-cgsnapshots/<id> ==
{
"$schema": "http://json-schema.org/draft-04/schema#",
"title": "Consistency Group Snapshot",
"description": "A resource representing a snapshot of a group of shares",
"type": "object",
"properties": {
"name": {
"description": "A non-unique name for a CG",
"type": "string",
'maxLength': 255,
},
"description": {
"description": "A short description of the CG",
"type": "string"
},
"required": []
}

Response: Detailed CG

Manila/design/manila-liberty-consistency-groups

2015-08-01T17:10:03Z

Alex Meade: update to os-consistency-groups

= Consistency Groups =
This document describes the design choices and steps for the consistency group feature in core Manila.

Consistency groups are a mechanism to allow multiple filesystem shares to be guaranteed snapshots are able to be created at the exact same point in time. For example, a database may have its tables, logs, and configuration on separate volumes. If we wanted to restore the database from a previous point in time, it would only make sense to restore the logs, tables, and configuration together from the exact same point in time.

= Manila Core Changes =
The core manila changes are very similar to what has been done in Cinder. At a high level, changes in the DB, share service, drivers, scheduler service and api service will need to be changed.

Phase 1:
* Create/Delete CGs
* Create a share within a CG
* Delete a share that is in a CG
* Snapshot entire CGs
* Create CG from a CGSnapshot

Potential future features:
* Add/remove shares in a CG
* CG migration
* Replication of a CG

==Manila vs Cinder==
* Different CLI syntax
* Restful api tweaks
** Change /create_from_src to just be a cgsnapshot_id in the body on POST /shares
** Deletes to use the DELETE HTTP verb, force_delete not supported until we have a standard API guideline
* At the Driver API, cgsnapshot objects and cg objects (with all the information about what volumes are in the CG) need to be passed to the driver instead of just the cg/cgsnapshot name. This is important for backends that aren't managing a CG construct such as NetApp cDOT as they need to do things such as loop through the snapshots in a cg to delete them.
* The default policy in policy.json for consistency group operations is the default policy versus in Cinder is it 'nobody'
* Unlike Cinder, snapshots in a CGsnapshot are not the same as a normal snapshot. A CGsnapshot is treated as a single unit instead of a collection of snapshots.
* consistency_group_support scheduler capability has multiple values instead of boolean. This is to accommodate backends that require all volumes in a CG to be on the same pool vs. just the same host. It also allows for adding additional values in the future if a backend can support some other paradigm like a CG spanning multiple backends.

== User Workflows ==
* Snapshot multiple shares
*# Create a Consistency Group
*# Create share(s) with a CG specified
*# Create a CGsnapshot

* Copy a consistency group
*# Create a CGsnapshot
*# Create a CG with the CGSnapshot id

* Delete a CG (and all of its shares)
*# Delete all CG snapshots of the CG
*# Delete all shares in the CG
*# Delete the CG (Only empty CGs can be deleted)

* List shares in a given CG
*# GET /shares?consistencygroup_id=<cg_id>

* List the shares that were captured in a given CGSnapshot
*# GET /cgsnapshots/<id>/members

== System Workflows ==
* Creating a CG
*# Create the CG in the DB
*# Cast the consistency group creation to the scheduler
*## Choose a host for the CG
*# Cast the creation to the share service
*## Get the CG info from DB
*## Call the driver
*## Update database with info returned from driver (ex: CG status)

* Creating a CG from a cgsnapshot
*# Create a new CG in the DB
*# Schedule the CG create to the scheduler with the orig CG host/pool+share_type (depending on capability)
*# For each member of the snapshot, schedule a create share on the same host/pool as CG

* Creating a cg snapshot
*# Create cgsnapshot entity in DB
*# For all shares in CG, create cgsnapshot_member in DB
*# cast create_cgsnapshot to the driver

* Adding a share to a CG
** On share creation:
**# Cast to scheduler with the host of the CG
**#* if perpool capable: Goes to the same pool as CG
**#* if per backend: goes to a pool on the same host that matches a CG share type

* Deleting a share (in a CG)
** Not allowed if the CG has snapshots of the share ( check if the share_id is in the CGSnapshotMembers tables)

== Scheduler ==
manila/scheduler/filter_scheduler.py should now look for consistency_group_support as a capability.

consistency_group_support
default: None 
Values:
* None - No support for CGs
* host - shares in a CG must be on pool(s) on the same host that also match the CG share type
* pool - shares in a CG must live in the same pool as the CG

==DB==
* New Tables:
class ConsistencyGroup(BASE, ManilaBase):
"""Represents a consistencygroup."""
__tablename__ = 'consistency_groups'
id = Column(String(36), primary_key=True)

user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)
deleted = Column(String(36), default='False')

host = Column(String(255))
name = Column(String(255))
description = Column(String(255))
status = Column(String(255))
source_cgsnapshot_id = Column(String(36))

class ConsistencyGroupShareTypeMapping(BASE, ManilaBase):
"""Represents the share types in a consistency group"""
__tablename__ = 'consistency_group_share_type_mappings'
id = Column(Integer, primary_key=True)
deleted = Column(String(36), default='False')
consistency_group_id = Column(String(36),
ForeignKey('consistency_groups.id'),
nullable=False)
share_type_id = Column(String(36),
ForeignKey('share_types.id'),
nullable=False)
consistency_group = orm.relationship(
ConsistencyGroup,
backref="share_types",
foreign_keys=consistency_group_id,
primaryjoin=('and_('
'ConsistencyGroupShareTypeMapping.consistency_group_id == '
'ConsistencyGroups.id,'
'ConsistencyGroupShareTypeMapping.deleted == 0)')
)

class CGSnapshot(BASE, ManilaBase):
"""Represents a cgsnapshot."""
__tablename__ = 'cgsnapshots'
id = Column(String(36), primary_key=True)

consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'))
user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)

name = Column(String(255))
description = Column(String(255))
status = Column(String(255))

class CGSnapshotMembers(BASE, ManilaBase):
__tablename__ = 'cgsnapshot_members'
id = Column(String(36), primary_key=True)
cgsnapshot_id = Column(String(36), ForeignKey('cgsnapshots.id'))
share_id = Column(String(36), ForeignKey('shares.id'))
size = Column(Integer)
status = Column(String(255))
share_proto = Column(String(255))
share_type_id = Column(String(36), ForeignKey('share_types.id'),
nullable=True)
user_id = Column(String(255))
project_id = Column(String(255))
deleted = Column(String(36), default='False')

cgsnapshot = relationship(
ConsistencyGroup,
backref="cgsnapshot_members",
foreign_keys=cgsnapshot_id,
primaryjoin='CGSnapshotMembers.cgsnapshot_id == CGSnapshot.id')

* New fields in Share
consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'), nullable=True)
consistency_group = relationship(
ConsistencyGroup,
backref="shares",
foreign_keys=consistency_group_id,
primaryjoin='Share.consistency_group_id == ConsistencyGroup.id')

source_cgsnapshot_member_id = Column(String(36), nullable=True)

==API==
Schemas: https://wiki.openstack.org/wiki/Manila/design/manila-liberty-consistency-groups/api-schema

'''GET''' /os-consistency-groups/
'''GET''' /os-consistency-groups/detail
'''POST''' /os-consistency-groups
'''GET''' /os-consistency-groups/<id>
'''DELETE''' os-consistency-groups/<id>
'''PUT''' /os-consistency-groups/<id>

'''GET''' /os-cgsnapshots/
'''GET''' /os-cgsnapshots/detail
'''POST''' /os-cgsnapshots
'''PUT''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>
'''GET''' /os-cgsnapshots/<id>/members
'''DELETE''' /os-cgsnapshots/<id>

Adds consistency_group_id to POST/shares
Add consistency_group_id query filter to /shares

==Policies==
policy.json - All consistency group policies should default to the default policy
"consistency_group:create" : "",
"consistency_group:delete": "",
"consistency_group:update": "",
"consistency_group:get": "",
"consistency_group:get_all": "",

"consistency_group:create_cgsnapshot" : "",
"consistency_group:delete_cgsnapshot": "",
"consistency_group:get_cgsnapshot": "",
"consistency_group:get_all_cgsnapshots": "",

==Driver API==

update_share_stats should now return 'consistency_group_support' pool/host/None.

def create_consistency_group(self, context, cg):

def delete_consistency_group(self, context, cg):

def create_cgsnapshot(self, context, cgsnapshot):

def delete_cgsnapshot(self, context, cgsnapshot):

Phase 2:

def update_consistency_group(self, context, cg):

== Manila Client ==
* Do we want to alias the commands so that there are Cinder consistent commands and the improved names? (I.E cg-create = consisgroup-create) Perhaps we alias the cinderclient commands to be more like this?
cg-snapshot-create Creates a cgsnapshot.
cg-snapshot-delete Removes one or more cgsnapshots.
cg-snapshot-list Lists all cgsnapshots.
cg-snapshot-show Shows cgsnapshot details.

cg-create Creates a consistency group (--cgsnapshot to create from existing cg snapshot)
cg-delete Removes one or more consistency groups.
cg-list Lists all consistencygroups.
cg-show Shows details of a consistency group.
cg-update Updates a consistencygroup.

create --consistency-group Creates a share and puts it into consistency group

= Outstanding Questions / FAQ =
# How do we avoid confusion if no backends in a deployment support CGs?
#* It appears that Cinder has the policies default to 'nobody' but this seems odd, is there a better way?
#** (ameade) I can't think of a way to have devstack set the policies automatically either.
# Do we need a way to fake CGs with the generic driver?
#* NO, but it will not be a core feature if it cannot be supported in the generic driver.
# Do we need a safeguard for deleting shares that are in a CG? (I.E. protected shares?) Are shares in a CG more valuable than normal shares?
# Should we overload the Snapshots table to include snapshots in a CGSnapshot to avoid adding a new db table? Or continue with having a new table?
# Why not have the cg delete all of the shares in it when deleting the cg?
#* This could cause odd state to occur if the backend fails when deleting shares and it will not be obvious where it failed. This is also extra complexity in the implementation that could instead be hidden by a client.

= Notes on Cinder Impl =

* A consistency group may be specified on volume create
* When a consistency group is created it is scheduled to a pool and all volume created in the CG will go to that pol

Cinder cli:
cgsnapshot-create Creates a cgsnapshot and snapshots of every volume in the CG that show up in snapshot-list.
cgsnapshot-delete Removes one or more cgsnapshots.
cgsnapshot-list Lists all cgsnapshots.
cgsnapshot-show Shows cgsnapshot details. <-- /cgsnapshots/detail?all_tenants=1&name=blah
consisgroup-create Creates a consistency group.
consisgroup-create-from-src Creates a consistency group from a cgsnapshot filled with volumes from the snapshots in the cg.
consisgroup-delete Removes one or more consistency groups.
consisgroup-list Lists all consistencygroups.
consisgroup-show Shows details of a consistency group. <-- /consistencygroups/detail?all_tenants=1&name=blah
consisgroup-update Updates a consistencygroup.

create --consisgroup-id

Cinder API:
'''GET''' /consistencygroups/detail
'''POST''' /consistencygroups
{'consistencygroup':{
'name': <string>,
'description': <string>,
'volume_types': <string>,
'availability_zone': <string>,
}}
'''POST''' /consistencygroups/create_from_src
{'consistencygroup-from-src':{
'name': <string>,
'description': <string>,
'cgsnapshot_id': <string>,
}}
'''GET''' /consistencygroups/<id>
'''POST''' consistencygroups/<id>/delete
{"consistencygroup": {"force": false}}
'''PUT''' /consistencygroups/<id>
{'consistencygroup':{
'name': <string>,
'description': <string>,
'add_volumes': <string>,
'remove_volumes': <string>,
}}

'''GET''' /cgsnapshots/detail
'''GET''' /cgsnapshots/<id>
'''POST''' /cgsnapshots
{'cgsnapshot':{
'consistencygroup_id': <string>,
}}
'''DELETE''' /cgsnapshots/<id>

Adds consistencygroup_id to POST/volumes

Xings commit message:
1) Create a CG, specifying all volume types that can be supported by this
CG. The scheduler chooses a backend that supports all specified volume types.
The CG will be empty when it is first created. Backend needs to report
consistencygroup_support = True. Volume type can have the following in
extra specs: {'capabilities:consistencygroup_support': '<is> True'}.
If consistencygroup_support is not in volume type extra specs, it will be
added to filter_properties by the scheduler to make sure that the scheduler
will select the backend which reports consistency group support capability.

Create CG CLI:
cinder consisgroup-create --volume-type type1,type2 mycg1

This will add a CG entry in the new consistencygroups table.

2) After the CG is created, create a new volume and add to the CG.
Repeat until all volumes are created for the CG.

Create volume CLI (with CG):
cinder create --volume-type type1 --consisgroup-id <CG uuid> 10

This will add a consistencygroup_id foreign key in the new volume
entry in the db.

3) Create a snapshot of the CG (cgsnapshot).

Create cgsnapshot CLI:
cinder cgsnapshot-create <CG uuid>

This will add a cgsnapshot entry in the new cgsnapshots table, create
snapshot for each volume in the CG, and add a cgsnapshot_id foreign key
in each newly created snapshot entry in the db.

===Questions===
# Why does policy.json default to 'nobody' being able to perform CG commands?
# Why are volume types specified when creating a CG?
#* This is because all volumes within a CG need to be on the same backend in order for the backend to honor the CG. This is important to note in the docs so that conflicting volume-types are not specified.
# What occurs when a volume on a different backend is added to a CG via the API?
#* Currently the API ensures that the volume volume_type matches a supported volume_type of the CG and that the CG and volume are on the same host (but can be on different pools).
# Is a volume created with a CG going to end up on the same host?
#* Yes, if a volume has a CG it will be cast to the same pool as the CG.
#* What if the pool is full but there are other available pools on the same backend? Should this fail?
# What happens if a driver does not support create_from_src?
#* The CG goes to ERROR a and volumes are created from the snapshot and also in ERROR status
#* A volume in a consistency group cannot be deleted and a CG with a volume cannot be deleted, therefore a volume must be removed from the CG before it can be deleted. But what if you don't support modifying a CG?
# Why do we need a force delete for CGs?
# What do I do if I do not want to see Snapshots in a CG when doing snapshot-list?
#* I suppose API filtering could allow for someone to not see all of the snapshots if they do not want to.

=== Cinder Bugs to file===
# What happens when you force delete a CG?
#* The volumes remain but still think they are in a CG, which means you still cannot delete the volume
# consisgroup-create-from-src says cgsnapshot is and optional param in cinderclient
# Creating a volume in a CG may not end up in the cg. The only check that is performed is that the volume-type of the volume is supported by the CG but volume-types are not 1-to-1 with backends. There could be backend A and B that both match with volume type 1. A cg is created with volume type 1 and ends up on backend A. Then a volume is created with the CG specified but ends up on backend B

=== Main impl ===
299b99437c2de84222fd585f06c2d05ca213225b ConsistencyGroup: Return 400 instead of 500 for invalid body
adb4c80be82caacad83f1366a4b34e5653fd5dab Create Consistency Group from CG Snapshot API
1a62a6e60fda73bf31256fbf684dc03bb6cf0038 Modify Consistency Group API
fc7a4384be2ccf8ff24f6c0f72c681ad9133801a Add tests for consistency groups DB migration
9082273305b0b9c117eb677a0e34c2ffde4e66f0 Volume types need to be specified when creating CG
764c5ec749821d36bb0215dc6002d3caea95d3b1 Delete consistency group failed
cf961f83ac323dfad1fa5e227d1e502a17529ecc Consistency Groups

=== Ex driver impl ===
8d5b795b37ed021c2639066689645f8aa0b1012f PureISCSIDriver consistency group updates.
39c1a8e08dc854e22ada315a46c20c99df2facf8 Add support to PureISCSIDriver for Consistency Groups
92a817708a3938b1b734d2caaa206b310996d8d0 EMC VMAX driver Kilo update

Manila/design/manila-liberty-consistency-groups

2015-07-31T18:30:40Z

Alex Meade: /* Cinder Bugs to file */

= Consistency Groups =
This document describes the design choices and steps for the consistency group feature in core Manila.

Consistency groups are a mechanism to allow multiple filesystem shares to be guaranteed snapshots are able to be created at the exact same point in time. For example, a database may have its tables, logs, and configuration on separate volumes. If we wanted to restore the database from a previous point in time, it would only make sense to restore the logs, tables, and configuration together from the exact same point in time.

= Manila Core Changes =
The core manila changes are very similar to what has been done in Cinder. At a high level, changes in the DB, share service, drivers, scheduler service and api service will need to be changed.

Phase 1:
* Create/Delete CGs
* Create a share within a CG
* Delete a share that is in a CG
* Snapshot entire CGs
* Create CG from a CGSnapshot

Potential future features:
* Add/remove shares in a CG
* CG migration
* Replication of a CG

==Manila vs Cinder==
* Different CLI syntax
* Restful api tweaks
** Change /create_from_src to just be a cgsnapshot_id in the body on POST /shares
** Deletes to use the DELETE HTTP verb, force_delete not supported until we have a standard API guideline
* At the Driver API, cgsnapshot objects and cg objects (with all the information about what volumes are in the CG) need to be passed to the driver instead of just the cg/cgsnapshot name. This is important for backends that aren't managing a CG construct such as NetApp cDOT as they need to do things such as loop through the snapshots in a cg to delete them.
* The default policy in policy.json for consistency group operations is the default policy versus in Cinder is it 'nobody'
* Unlike Cinder, snapshots in a CGsnapshot are not the same as a normal snapshot. A CGsnapshot is treated as a single unit instead of a collection of snapshots.
* consistency_group_support scheduler capability has multiple values instead of boolean. This is to accommodate backends that require all volumes in a CG to be on the same pool vs. just the same host. It also allows for adding additional values in the future if a backend can support some other paradigm like a CG spanning multiple backends.

== User Workflows ==
* Snapshot multiple shares
*# Create a Consistency Group
*# Create share(s) with a CG specified
*# Create a CGsnapshot

* Copy a consistency group
*# Create a CGsnapshot
*# Create a CG with the CGSnapshot id

* Delete a CG (and all of its shares)
*# Delete all CG snapshots of the CG
*# Delete all shares in the CG
*# Delete the CG (Only empty CGs can be deleted)

* List shares in a given CG
*# GET /shares?consistencygroup_id=<cg_id>

* List the shares that were captured in a given CGSnapshot
*# GET /cgsnapshots/<id>/members

== System Workflows ==
* Creating a CG
*# Create the CG in the DB
*# Cast the consistency group creation to the scheduler
*## Choose a host for the CG
*# Cast the creation to the share service
*## Get the CG info from DB
*## Call the driver
*## Update database with info returned from driver (ex: CG status)

* Creating a CG from a cgsnapshot
*# Create a new CG in the DB
*# Schedule the CG create to the scheduler with the orig CG host/pool+share_type (depending on capability)
*# For each member of the snapshot, schedule a create share on the same host/pool as CG

* Creating a cg snapshot
*# Create cgsnapshot entity in DB
*# For all shares in CG, create cgsnapshot_member in DB
*# cast create_cgsnapshot to the driver

* Adding a share to a CG
** On share creation:
**# Cast to scheduler with the host of the CG
**#* if perpool capable: Goes to the same pool as CG
**#* if per backend: goes to a pool on the same host that matches a CG share type

* Deleting a share (in a CG)
** Not allowed if the CG has snapshots of the share ( check if the share_id is in the CGSnapshotMembers tables)

== Scheduler ==
manila/scheduler/filter_scheduler.py should now look for consistency_group_support as a capability.

consistency_group_support
default: None 
Values:
* None - No support for CGs
* host - shares in a CG must be on pool(s) on the same host that also match the CG share type
* pool - shares in a CG must live in the same pool as the CG

==DB==
* New Tables:
class ConsistencyGroup(BASE, ManilaBase):
"""Represents a consistencygroup."""
__tablename__ = 'consistency_groups'
id = Column(String(36), primary_key=True)

user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)
deleted = Column(String(36), default='False')

host = Column(String(255))
name = Column(String(255))
description = Column(String(255))
status = Column(String(255))
source_cgsnapshot_id = Column(String(36))

class ConsistencyGroupShareTypeMapping(BASE, ManilaBase):
"""Represents the share types in a consistency group"""
__tablename__ = 'consistency_group_share_type_mappings'
id = Column(Integer, primary_key=True)
deleted = Column(String(36), default='False')
consistency_group_id = Column(String(36),
ForeignKey('consistency_groups.id'),
nullable=False)
share_type_id = Column(String(36),
ForeignKey('share_types.id'),
nullable=False)
consistency_group = orm.relationship(
ConsistencyGroup,
backref="share_types",
foreign_keys=consistency_group_id,
primaryjoin=('and_('
'ConsistencyGroupShareTypeMapping.consistency_group_id == '
'ConsistencyGroups.id,'
'ConsistencyGroupShareTypeMapping.deleted == 0)')
)

class CGSnapshot(BASE, ManilaBase):
"""Represents a cgsnapshot."""
__tablename__ = 'cgsnapshots'
id = Column(String(36), primary_key=True)

consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'))
user_id = Column(String(255), nullable=False)
project_id = Column(String(255), nullable=False)

name = Column(String(255))
description = Column(String(255))
status = Column(String(255))

class CGSnapshotMembers(BASE, ManilaBase):
__tablename__ = 'cgsnapshot_members'
id = Column(String(36), primary_key=True)
cgsnapshot_id = Column(String(36), ForeignKey('cgsnapshots.id'))
share_id = Column(String(36), ForeignKey('shares.id'))
size = Column(Integer)
status = Column(String(255))
share_proto = Column(String(255))
share_type_id = Column(String(36), ForeignKey('share_types.id'),
nullable=True)
user_id = Column(String(255))
project_id = Column(String(255))
deleted = Column(String(36), default='False')

cgsnapshot = relationship(
ConsistencyGroup,
backref="cgsnapshot_members",
foreign_keys=cgsnapshot_id,
primaryjoin='CGSnapshotMembers.cgsnapshot_id == CGSnapshot.id')

* New fields in Share
consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'), nullable=True)
consistency_group = relationship(
ConsistencyGroup,
backref="shares",
foreign_keys=consistency_group_id,
primaryjoin='Share.consistency_group_id == ConsistencyGroup.id')

source_cgsnapshot_member_id = Column(String(36), nullable=True)

==API==
Schemas: https://wiki.openstack.org/wiki/Manila/design/manila-liberty-consistency-groups/api-schema

'''GET''' /consistencygroups/
'''GET''' /consistencygroups/detail
'''POST''' /consistencygroups
'''GET''' /consistencygroups/<id>
'''DELETE''' consistencygroups/<id>
'''PUT''' /consistencygroups/<id>

'''GET''' /cgsnapshots/
'''GET''' /cgsnapshots/detail
'''POST''' /cgsnapshots
'''PUT''' /cgsnapshots/<id>
'''GET''' /cgsnapshots/<id>
'''GET''' /cgsnapshots/<id>/members
'''DELETE''' /cgsnapshots/<id>

Adds consistency_group_id to POST/shares
Add consistency_group_id query filter to /shares

==Policies==
policy.json - All consistency group policies should default to the default policy
"consistency_group:create" : "",
"consistency_group:delete": "",
"consistency_group:update": "",
"consistency_group:get": "",
"consistency_group:get_all": "",

"consistency_group:create_cgsnapshot" : "",
"consistency_group:delete_cgsnapshot": "",
"consistency_group:get_cgsnapshot": "",
"consistency_group:get_all_cgsnapshots": "",

==Driver API==

update_share_stats should now return 'consistency_group_support' pool/host/None.

def create_consistency_group(self, context, cg):

def delete_consistency_group(self, context, cg):

def create_cgsnapshot(self, context, cgsnapshot):

def delete_cgsnapshot(self, context, cgsnapshot):

Phase 2:

def update_consistency_group(self, context, cg):

== Manila Client ==
* Do we want to alias the commands so that there are Cinder consistent commands and the improved names? (I.E cg-create = consisgroup-create) Perhaps we alias the cinderclient commands to be more like this?
cg-snapshot-create Creates a cgsnapshot.
cg-snapshot-delete Removes one or more cgsnapshots.
cg-snapshot-list Lists all cgsnapshots.
cg-snapshot-show Shows cgsnapshot details.

cg-create Creates a consistency group (--cgsnapshot to create from existing cg snapshot)
cg-delete Removes one or more consistency groups.
cg-list Lists all consistencygroups.
cg-show Shows details of a consistency group.
cg-update Updates a consistencygroup.

create --consistency-group Creates a share and puts it into consistency group

= Outstanding Questions / FAQ =
# How do we avoid confusion if no backends in a deployment support CGs?
#* It appears that Cinder has the policies default to 'nobody' but this seems odd, is there a better way?
#** (ameade) I can't think of a way to have devstack set the policies automatically either.
# Do we need a way to fake CGs with the generic driver?
#* NO, but it will not be a core feature if it cannot be supported in the generic driver.
# Do we need a safeguard for deleting shares that are in a CG? (I.E. protected shares?) Are shares in a CG more valuable than normal shares?
# Should we overload the Snapshots table to include snapshots in a CGSnapshot to avoid adding a new db table? Or continue with having a new table?
# Why not have the cg delete all of the shares in it when deleting the cg?
#* This could cause odd state to occur if the backend fails when deleting shares and it will not be obvious where it failed. This is also extra complexity in the implementation that could instead be hidden by a client.

= Notes on Cinder Impl =

* A consistency group may be specified on volume create
* When a consistency group is created it is scheduled to a pool and all volume created in the CG will go to that pol

Cinder cli:
cgsnapshot-create Creates a cgsnapshot and snapshots of every volume in the CG that show up in snapshot-list.
cgsnapshot-delete Removes one or more cgsnapshots.
cgsnapshot-list Lists all cgsnapshots.
cgsnapshot-show Shows cgsnapshot details. <-- /cgsnapshots/detail?all_tenants=1&name=blah
consisgroup-create Creates a consistency group.
consisgroup-create-from-src Creates a consistency group from a cgsnapshot filled with volumes from the snapshots in the cg.
consisgroup-delete Removes one or more consistency groups.
consisgroup-list Lists all consistencygroups.
consisgroup-show Shows details of a consistency group. <-- /consistencygroups/detail?all_tenants=1&name=blah
consisgroup-update Updates a consistencygroup.

create --consisgroup-id

Cinder API:
'''GET''' /consistencygroups/detail
'''POST''' /consistencygroups
{'consistencygroup':{
'name': <string>,
'description': <string>,
'volume_types': <string>,
'availability_zone': <string>,
}}
'''POST''' /consistencygroups/create_from_src
{'consistencygroup-from-src':{
'name': <string>,
'description': <string>,
'cgsnapshot_id': <string>,
}}
'''GET''' /consistencygroups/<id>
'''POST''' consistencygroups/<id>/delete
{"consistencygroup": {"force": false}}
'''PUT''' /consistencygroups/<id>
{'consistencygroup':{
'name': <string>,
'description': <string>,
'add_volumes': <string>,
'remove_volumes': <string>,
}}

'''GET''' /cgsnapshots/detail
'''GET''' /cgsnapshots/<id>
'''POST''' /cgsnapshots
{'cgsnapshot':{
'consistencygroup_id': <string>,
}}
'''DELETE''' /cgsnapshots/<id>

Adds consistencygroup_id to POST/volumes

Xings commit message:
1) Create a CG, specifying all volume types that can be supported by this
CG. The scheduler chooses a backend that supports all specified volume types.
The CG will be empty when it is first created. Backend needs to report
consistencygroup_support = True. Volume type can have the following in
extra specs: {'capabilities:consistencygroup_support': '<is> True'}.
If consistencygroup_support is not in volume type extra specs, it will be
added to filter_properties by the scheduler to make sure that the scheduler
will select the backend which reports consistency group support capability.

Create CG CLI:
cinder consisgroup-create --volume-type type1,type2 mycg1

This will add a CG entry in the new consistencygroups table.

2) After the CG is created, create a new volume and add to the CG.
Repeat until all volumes are created for the CG.

Create volume CLI (with CG):
cinder create --volume-type type1 --consisgroup-id <CG uuid> 10

This will add a consistencygroup_id foreign key in the new volume
entry in the db.

3) Create a snapshot of the CG (cgsnapshot).

Create cgsnapshot CLI:
cinder cgsnapshot-create <CG uuid>

This will add a cgsnapshot entry in the new cgsnapshots table, create
snapshot for each volume in the CG, and add a cgsnapshot_id foreign key
in each newly created snapshot entry in the db.

===Questions===
# Why does policy.json default to 'nobody' being able to perform CG commands?
# Why are volume types specified when creating a CG?
#* This is because all volumes within a CG need to be on the same backend in order for the backend to honor the CG. This is important to note in the docs so that conflicting volume-types are not specified.
# What occurs when a volume on a different backend is added to a CG via the API?
#* Currently the API ensures that the volume volume_type matches a supported volume_type of the CG and that the CG and volume are on the same host (but can be on different pools).
# Is a volume created with a CG going to end up on the same host?
#* Yes, if a volume has a CG it will be cast to the same pool as the CG.
#* What if the pool is full but there are other available pools on the same backend? Should this fail?
# What happens if a driver does not support create_from_src?
#* The CG goes to ERROR a and volumes are created from the snapshot and also in ERROR status
#* A volume in a consistency group cannot be deleted and a CG with a volume cannot be deleted, therefore a volume must be removed from the CG before it can be deleted. But what if you don't support modifying a CG?
# Why do we need a force delete for CGs?
# What do I do if I do not want to see Snapshots in a CG when doing snapshot-list?
#* I suppose API filtering could allow for someone to not see all of the snapshots if they do not want to.

=== Cinder Bugs to file===
# What happens when you force delete a CG?
#* The volumes remain but still think they are in a CG, which means you still cannot delete the volume
# consisgroup-create-from-src says cgsnapshot is and optional param in cinderclient
# Creating a volume in a CG may not end up in the cg. The only check that is performed is that the volume-type of the volume is supported by the CG but volume-types are not 1-to-1 with backends. There could be backend A and B that both match with volume type 1. A cg is created with volume type 1 and ends up on backend A. Then a volume is created with the CG specified but ends up on backend B

=== Main impl ===
299b99437c2de84222fd585f06c2d05ca213225b ConsistencyGroup: Return 400 instead of 500 for invalid body
adb4c80be82caacad83f1366a4b34e5653fd5dab Create Consistency Group from CG Snapshot API
1a62a6e60fda73bf31256fbf684dc03bb6cf0038 Modify Consistency Group API
fc7a4384be2ccf8ff24f6c0f72c681ad9133801a Add tests for consistency groups DB migration
9082273305b0b9c117eb677a0e34c2ffde4e66f0 Volume types need to be specified when creating CG
764c5ec749821d36bb0215dc6002d3caea95d3b1 Delete consistency group failed
cf961f83ac323dfad1fa5e227d1e502a17529ecc Consistency Groups

=== Ex driver impl ===
8d5b795b37ed021c2639066689645f8aa0b1012f PureISCSIDriver consistency group updates.
39c1a8e08dc854e22ada315a46c20c99df2facf8 Add support to PureISCSIDriver for Consistency Groups
92a817708a3938b1b734d2caaa206b310996d8d0 EMC VMAX driver Kilo update