HierarchicalMultitenancy

Meetings

We have regular meetings on Fridays at 1600 UTC in #openstack-meeting.

Hierarchical Multitenancy

OpenStack needs to grow support for hierarchical ownership of objects. This enables the management subsets of users and projects in a way that is much more comfortable for private clouds. Adding this type of feature will require work across all of the projects in OpenStack. Traditionally efforts like this have stalled out because of the amount of collaboration required. For this reason, the goal is to make a small cross-functional team to design and implement this feature by producing real code.

Use Case

Martha, the owner of ProductionIT provides it services to multiple Enterprise clients. She would like to offer cloud services to Joe at WidgetMaster, and Sam at SuperDevShop. Joe is a Development Manager for WidgetMaster and he has multiple QA and Development teams with many users. Joe needs the ability create users, projects, and quotas, as well as the ability to list and delete resources across WidgetMaster. Martha needs to be able to set the quotas for both WidgetMaster and SuperDevShop; manage users, projects, and objects across the entire system; and set quotas for the client companies as a whole. She also needs to ensure that Joe can't see or mess with anything owned by Sam.

Problems

* Multiple Ownership (items in projects have a single project_id owner)
* Quotas (quotas can only be set per project_id [+ per user in nova])
* Coarse Policy (most policy files only consider admin vs. non-admin)
* Data migration to the new model
* Tracking the creator vs. the owner of an object
* Domains are not well understood
* Other things we haven't thought of

Potential Solutions

In the interest of common terminology, I'm going to refer to a "domain" as a higher level grouping of projects. Ideally this would be a nested structure, but at the very least one level is necessary.

Mapping

We could allow the services to request mapping information from keystone in ownership checks. For example when checking the project_id of an instance, we could make a request back to keystone to see if the project belongs to the domain. While we could do some caching here, this seems particularly bad performance for things like list.

Remove Cross Project Functionality

Another solution would be to remove the cross-project listing from the server side completely. This would require domain users to authenticate in the project they would like to control resources for. Listing would be handled on the client side, which would aggregate a list of all resources across the domain by listing each project individually. This is probably the cleanest approach, but it is a crummy user experience and has some very troubling performance concerns as the number of projects gets high.

Multiple Ownership of Objects

We could allow the project_id (or 'owner' in some openstack projects) to be a list. This would allow us to grant ownership of the resource to both the project and the domain, so list/delete operations could be done in either context. This is very flexible but has the minor disadvantage of not providing clear information of the hierarchy of things, requiring matches against multiple owners. The main disadvantage of this approach is the large set of changes it would require across projects.

Hierarchical Ownership

A simplified version of Multiple Ownership, instead of a list of owners, the owner is limited to a single representation of the owner. For example, the owner of an instance might be 'domainfoo.projectbar' or 'WidgetMaster.DevUnit.WebFrontend'. The structure of ownership is more static then a list, but it is much simpler in most cases. It also translates nicely into wildcard queries for ownership which means performance in the backend should be excellent.

Path Forward

Based on some experimentation, we have come up with a basic proposal for moving forward. This involves an implementation of the fourth option above, with the option for projects to move to multiple ownership (3) when they are ready.

Keystone Changes

1. Projects in keystone will gain a new field, parent_project_id, to create a tree of projects
2. Roles will be inherited down the project hierarchy tree. (See examples at [1])
3. When a token is created, tbe full role list and hierarchy will be determined by walking up the tree (optionally controlled by an inherited flag [2]).
4. The hierarchy will be passed in two new fields in the token: hierarchical_ids, hierarchical_names. These fields could use the ASCII record separator (0x1e) so we can use more common characters like dots in project names. (Some example code for the two above here: [3])
5. The auth token middleware will be modified to pull these values out of the token and provide them in the wsgi env to the request context in the other projects.
6. Domains may be modified to exist as top-level projects, or they may still exist solely as containers for users with the project hierarchy remaining separate.

Openstack Common Changes

1. Code will be added to the policy module to deal with prefix matching so policies can be written to allow access to a group of resources in the project tree
2. Some helper code for generating SQL queries using prefix matching will be added to the common db code (Some example code for the two above here: [4])
3. Caching code to maintain a project_id -> name cache in memory will be added. (Sample code in nova here: [5])

Other Project Changes

1. Projects will modify their ownership fields to contain the entire hierarchical id.
2. Projects will modify their policy and db code using the above common libraries to allow any existing cross-project calls (ie. nva list --all-tenants) to use prefix matching. (Some example code for the two above here: [6])
3. Projects need a way to change the hierarchical ownership if a new value comes from keystone due to a project move. This could be a separate api, but it could also be done on demand as queries are made where the hierarchy doesn't match.
4. Projects will have a cache of project_names (using above common code) for display purposes, so they can display the full hierarchical owner of an object to users. This can be done using an existing field where it makes sense or a new field can be added to the response.
5. If a project wants sharing of objects, it can modify the ownership field (or create a new field like 'shares') to contain a list of hierarchical owners instead of a single owner. The listing code would have to be modified in this case to search through all the shares, so this will likely have to be stored in a separate table for efficient queries.