Difference between revisions of "Manila/design/manila-liberty-consistency-groups"
Alex Meade (talk | contribs) m (→Questions) |
Alex Meade (talk | contribs) m (→Manila Client) |
||
Line 366: | Line 366: | ||
create --consistency-group Creates a share and puts it into consistency group | create --consistency-group Creates a share and puts it into consistency group | ||
+ | |||
+ | == Outstanding Questions == | ||
+ | # How do we avoid confusion if no backends in a deployment support CGs? | ||
+ | #* It appears that Cinder has the policies default to 'nobody' but this seems odd, is there a better way? | ||
= Notes on Cinder Impl = | = Notes on Cinder Impl = |
Revision as of 01:37, 29 July 2015
Contents
Consistency Groups
This document describes the design choices and steps for the consistency group feature in core Manila.
Consistency groups are a mechanism to allow multiple storage volumes to be guaranteed snapshots are able to be created at the exact same point in time. For example, a database may have its tables, logs, and configuration on separate volumes. If we wanted to restore the database from a previous point in time, it would only make sense to restore the logs, tables, and configuration together from the exact same point in time.
Manila Core Changes
The core manila changes are very similar to what has been done in Cinder. At a high level, changes in the DB, share service, drivers, scheduler service and api service will need to be changed.
Phase 1:
- Create/Delete CGs
- Create a share within a CG
- Delete a share that is in a CG
- Snapshot entire CGs
- Create CG from a CGSnapshot
Phase 2:
- Add/remove shares in a CG
- CG migration
Manila vs Cinder
- Different CLI syntax
- Restful api tweaks
- Change /create_from_src to just be a cgsnapshot_id in the body on POST /shares
- Deletes to use the DELETE HTTP verb, force_delete not supported until we have a standard API guideline
- At the Driver API, cgsnapshot objects and cg objects (with all the information about what volumes are in the CG) need to be passed to the driver instead of just the cg/cgsnapshot name. This is important for backends that aren't managing a CG construct such as NetApp cDOT as they need to do things such as loop through the snapshots in a cg to delete them.
- The default policy in policy.json for consistency group operations is the default policy versus in Cinder is it 'nobody'
User Workflows
- Snapshot multiple shares
- Create a Consistency Group
- Create share(s) with a CG specified
- Create a CGsnapshot
- Copy a consistency group
- Create a CGsnapshot
- Create a CG with the CGSnapshot id
- Delete a CG (and all of its shares)
- Delete all CG snapshots of the CG
- Delete the CG (all shares in the CG will be deleted)
- List shares in a given CG
- GET /shares?consistencygroup_id=<cg_id>
- List the shares that were captured in a given CGSnapshot
- GET /cgsnapshots/<id>/members
System Workflows
- Creating a CG
- Create the CG in the DB
- Cast the consistency group creation to the scheduler
- Choose a host for the CG
- Cast the creation to the share service
- Get the CG info from DB
- Call the driver
- Update database with info returned from driver (ex: CG status)
- Creating a CG from a cgsnapshot
- Create a new CG in the DB
- Schedule the CG create to the scheduler with the orig CG host/pool+share_type (depending on capability)
- For each member of the snapshot, schedule a create share on the same host/pool as CG
- Creating a cg snapshot
- Create cgsnapshot entity in DB
- For all shares in CG, create cgsnapshot_member in DB
- cast create_cgsnapshot to the driver
- Adding a share to a CG
- On share creation:
- Cast to scheduler with the host of the CG
- if perpool capable: Goes to the same pool as CG
- if per backend: goes to a pool on the same host that matches a CG share type
- Cast to scheduler with the host of the CG
- On share creation:
- Deleting a share (in a CG)
- Not allowed if the CG has snapshots of the share ( check if the share_id is in the CGSnapshotMembers tables)
Phase 2:
- After share creation:
- Cast update_cg with the list of shares to be in the CG
- Migrating a Share in a consistency group
- This is not allowed, a share must be removed from a consistency group before being migrated. If the backend cannot support modifying consistency groups, the share could be snapshotted and then created again as a workaround.
Scheduler
manila/scheduler/filter_scheduler.py should now look for consistency_group_support as a capability.
consistency_group_support
default: None
Values:
- None - No support for CGs
- host - shares in a CG must be on pool(s) on the same host that also match the CG share type
- pool - shares in a CG must live in the same pool as the CG
DB
- New Tables:
class ConsistencyGroup(BASE, CinderBase): """Represents a consistencygroup.""" __tablename__ = 'consistency_groups' id = Column(String(36), primary_key=True) user_id = Column(String(255), nullable=False) project_id = Column(String(255), nullable=False) host = Column(String(255)) name = Column(String(255)) description = Column(String(255)) share_type_ids = Column(String(36)) status = Column(String(255)) source_cgsnapshot_id = Column(String(36))
class CGSnapshot(BASE, CinderBase): """Represents a cgsnapshot.""" __tablename__ = 'cgsnapshots' id = Column(String(36), primary_key=True) consistency_group_id = Column(String(36)) user_id = Column(String(255), nullable=False) project_id = Column(String(255), nullable=False) share_type_ids = Column(String(36)) name = Column(String(255)) description = Column(String(255)) status = Column(String(255)) consistencygroup = relationship( ConsistencyGroup, backref="cgsnapshot_members", foreign_keys=consistency_group_id, primaryjoin='CGSnapshotMembers.consistency_group_id == ConsistencyGroup.id')
class CGSnapshotMembers(BASE, CinderBase): __tablename__ = 'cgsnapshot_members' id = Column(String(36), primary_key=True) consistency_group_id = Column(String(36)) share_id = Column(String(36)) size = Column(Integer) status = Column(String(255)) share_proto = Column(String(255)) share_type_id = Column(String(36), ForeignKey('share_types.id'), nullable=True) user_id = Column(String(255)) project_id = Column(String(255))
- New fields in Share
consistency_group_id = Column(String(36), ForeignKey('consistency_groups.id'), nullable=True) consistency_group = relationship( ConsistencyGroup, backref="shares", foreign_keys=consistency_group_id, primaryjoin='Share.consistency_group_id == ConsistencyGroup.id')
API
GET /consistencygroups/ Response: {'consistency_groups':[ { 'id': <int>, 'name': <string>, 'links':..., }, ]} GET /consistencygroups/detail Response: {'consistency_groups':[ { 'id': <string>, 'name': <string>, 'description': <string> 'created_at': <date>, 'source_cgsnapshot_id': <string>, 'status': <string>, 'share_types': <string>, 'project_id': <string>, 'host': <string>, 'links':..., }, ]} POST /consistencygroups Request: {'consistency_group':{ 'name': <string>, 'description': <string>, 'share_types': <string>, OR 'cgsnapshot_id': <string>, }} Response: {'consistency_group':{ 'id': <string>, 'name': <string>, 'description': <string> 'created_at': <date>, 'source_cgsnapshot_id': <string>, 'status': <string>, 'share_types': <string>, 'project_id': <string>, 'host': <string>, 'links':..., }, } GET /consistencygroups/<id> Response: {'consistency_group':{ 'id': <string>, 'name': <string>, 'description': <string> 'created_at': <date>, 'source_cgsnapshot_id': <string>, 'status': <string>, 'share_types': <string>, 'project_id': <string>, 'host': <string>, 'links':..., }, } DELETE consistencygroups/<id> PUT /consistencygroups/<id> Request: {'consistency_group':{ 'name': <string>, 'description': <string>, }} Response: {'consistency_group':{ 'id': <string>, 'name': <string>, 'description': <string> 'created_at': <date>, 'source_cgsnapshot_id': <string>, 'status': <string>, 'share_types': <string>, 'project_id': <string>, 'host': <string>, 'links':..., }, } GET /cgsnapshots/ Response: {'cgsnapshot':[ { 'id': <int>, 'name': <string>, 'links':..., }, ]} GET /cgsnapshots/detail Response: {'cgsnapshots':[ { 'id': <string>, 'name': <string>, 'description': <string> 'created_at': <date>, 'consistency_group_id': <string>, 'status': <string>, 'share_types': <string>, 'project_id': <string>, 'links':..., }, ]} GET /cgsnapshots/<id> {'cgsnapshot':{ 'id': <string>, 'name': <string>, 'description': <string> 'created_at': <date>, 'consistency_group_id': <string>, 'status': <string>, 'share_types': <string>, 'project_id': <string>, 'links':..., }, } GET /cgsnapshots/<id>/members {'cgsnapshot_members':[ { 'id': <string>, 'consistency_group_id': <string>, 'created_at': <date>, 'status': <string>, 'project_id': <string>, 'share_proto': <string>, 'size': <int>, 'share_id': <string>, 'share_type': <string>, 'links':..., }, ]} POST /cgsnapshots Request: {'cgsnapshot':{ 'consistency_group_id': <string>, }} Response: {'cgsnapshot':{ 'id': <string>, 'name': <string>, 'description': <string> 'created_at': <date>, 'consistency_group_id': <string>, 'status': <string>, 'project_id': <string>, 'links':..., }, } DELETE /cgsnapshots/<id>
Adds consistency_group_id to POST/shares Add consistency_group_id query filter to /shares
Policies
policy.json - All consistency group policies should default to the default policy
"consistency_group:create" : "", "consistency_group:delete": "", "consistency_group:update": "", "consistency_group:get": "", "consistency_group:get_all": "", "consistency_group:create_cgsnapshot" : "", "consistency_group:delete_cgsnapshot": "", "consistency_group:get_cgsnapshot": "", "consistency_group:get_all_cgsnapshots": "",
Driver API
update_share_stats should now return 'consistency_group_support' pool/host/None.
def create_consistency_group(self, context, cg):
def delete_consistency_group(self, context, cg):
def create_cgsnapshot(self, context, cgsnapshot):
def delete_cgsnapshot(self, context, cgsnapshot):
Phase 2:
def update_consistency_group(self, context, cg):
Manila Client
- Do we want to alias the commands so that there are Cinder consistent commands and the improved names? (I.E cg-create = consisgroup-create) Perhaps we alias the cinderclient commands to be more like this?
cg-snapshot-create Creates a cgsnapshot. cg-snapshot-delete Removes one or more cgsnapshots. cg-snapshot-list Lists all cgsnapshots. cg-snapshot-show Shows cgsnapshot details. cg-create Creates a consistency group (--cgsnapshot to create from existing cg snapshot) cg-delete Removes one or more consistency groups. cg-list Lists all consistencygroups. cg-show Shows details of a consistency group. cg-update Updates a consistencygroup.
create --consistency-group Creates a share and puts it into consistency group
Outstanding Questions
- How do we avoid confusion if no backends in a deployment support CGs?
- It appears that Cinder has the policies default to 'nobody' but this seems odd, is there a better way?
Notes on Cinder Impl
- A consistency group may be specified on volume create
- When a consistency group is created it is scheduled to a pool and all volume created in the CG will go to that pol
Cinder cli:
cgsnapshot-create Creates a cgsnapshot and snapshots of every volume in the CG that show up in snapshot-list. cgsnapshot-delete Removes one or more cgsnapshots. cgsnapshot-list Lists all cgsnapshots. cgsnapshot-show Shows cgsnapshot details. <-- /cgsnapshots/detail?all_tenants=1&name=blah consisgroup-create Creates a consistency group. consisgroup-create-from-src Creates a consistency group from a cgsnapshot filled with volumes from the snapshots in the cg. consisgroup-delete Removes one or more consistency groups. consisgroup-list Lists all consistencygroups. consisgroup-show Shows details of a consistency group. <-- /consistencygroups/detail?all_tenants=1&name=blah consisgroup-update Updates a consistencygroup. create --consisgroup-id
Cinder API:
GET /consistencygroups/detail POST /consistencygroups {'consistencygroup':{ 'name': <string>, 'description': <string>, 'volume_types': <string>, 'availability_zone': <string>, }} POST /consistencygroups/create_from_src {'consistencygroup-from-src':{ 'name': <string>, 'description': <string>, 'cgsnapshot_id': <string>, }} GET /consistencygroups/<id> POST consistencygroups/<id>/delete {"consistencygroup": {"force": false}} PUT /consistencygroups/<id> {'consistencygroup':{ 'name': <string>, 'description': <string>, 'add_volumes': <string>, 'remove_volumes': <string>, }} GET /cgsnapshots/detail GET /cgsnapshots/<id> POST /cgsnapshots {'cgsnapshot':{ 'consistencygroup_id': <string>, }} DELETE /cgsnapshots/<id> Adds consistencygroup_id to POST/volumes
Xings commit message:
1) Create a CG, specifying all volume types that can be supported by this CG. The scheduler chooses a backend that supports all specified volume types. The CG will be empty when it is first created. Backend needs to report consistencygroup_support = True. Volume type can have the following in extra specs: {'capabilities:consistencygroup_support': '<is> True'}. If consistencygroup_support is not in volume type extra specs, it will be added to filter_properties by the scheduler to make sure that the scheduler will select the backend which reports consistency group support capability. Create CG CLI: cinder consisgroup-create --volume-type type1,type2 mycg1 This will add a CG entry in the new consistencygroups table. 2) After the CG is created, create a new volume and add to the CG. Repeat until all volumes are created for the CG. Create volume CLI (with CG): cinder create --volume-type type1 --consisgroup-id <CG uuid> 10 This will add a consistencygroup_id foreign key in the new volume entry in the db. 3) Create a snapshot of the CG (cgsnapshot). Create cgsnapshot CLI: cinder cgsnapshot-create <CG uuid> This will add a cgsnapshot entry in the new cgsnapshots table, create snapshot for each volume in the CG, and add a cgsnapshot_id foreign key in each newly created snapshot entry in the db.
Questions
- Why does policy.json default to 'nobody' being able to perform CG commands?
- Why are volume types specified when creating a CG?
- This is because all volumes within a CG need to be on the same backend in order for the backend to honor the CG. This is important to note in the docs so that conflicting volume-types are not specified.
- What occurs when a volume on a different backend is added to a CG via the API?
- Currently the API ensures that the volume volume_type matches a supported volume_type of the CG and that the CG and volume are on the same host (but can be on different pools).
- Is a volume created with a CG going to end up on the same host?
- Yes, if a volume has a CG it will be cast to the same pool as the CG.
- What if the pool is full but there are other available pools on the same backend? Should this fail?
- What happens if a driver does not support create_from_src?
- The CG goes to ERROR a and volumes are created from the snapshot and also in ERROR status
- A volume in a consistency group cannot be deleted and a CG with a volume cannot be deleted, therefore a volume must be removed from the CG before it can be deleted. But what if you don't support modifying a CG?
- Why do we need a force delete for CGs?
- What do I do if I do not want to see Snapshots in a CG when doing snapshot-list?
- I suppose API filtering could allow for someone to not see all of the snapshots if they do not want to.
Cinder Bugs to file
- What happens when you force delete a CG?
- The volumes remain but still think they are in a CG, which means you still cannot delete the volume
- consisgroup-create-from-src says cgsnapshot is and optional param in cinderclient
Main impl
299b99437c2de84222fd585f06c2d05ca213225b ConsistencyGroup: Return 400 instead of 500 for invalid body adb4c80be82caacad83f1366a4b34e5653fd5dab Create Consistency Group from CG Snapshot API 1a62a6e60fda73bf31256fbf684dc03bb6cf0038 Modify Consistency Group API fc7a4384be2ccf8ff24f6c0f72c681ad9133801a Add tests for consistency groups DB migration 9082273305b0b9c117eb677a0e34c2ffde4e66f0 Volume types need to be specified when creating CG 764c5ec749821d36bb0215dc6002d3caea95d3b1 Delete consistency group failed cf961f83ac323dfad1fa5e227d1e502a17529ecc Consistency Groups
Ex driver impl
8d5b795b37ed021c2639066689645f8aa0b1012f PureISCSIDriver consistency group updates. 39c1a8e08dc854e22ada315a46c20c99df2facf8 Add support to PureISCSIDriver for Consistency Groups 92a817708a3938b1b734d2caaa206b310996d8d0 EMC VMAX driver Kilo update