Difference between revisions of "Cinder Pagination"
| (4 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
== Problem description == | == Problem description == | ||
| − | Pagination allows for retrieving a limited number of results instead of the entire data set. The cinder ''/volumes'' and ''/volumes/detail'' APIs support pagination and only support a single sort key and sort direction. The REST APIs need to accept multiple sort keys and directions so that the caller has more control over the sort order | + | Pagination allows for retrieving a limited number of results instead of the entire data set. The cinder ''/volumes'' and ''/volumes/detail'' APIs support pagination and only support a single sort key and sort direction. The REST APIs need to accept multiple sort keys and directions so that the caller has more granular control over the data set sort order. |
=== Use Case === | === Use Case === | ||
A UI that displays a table with only the page of data that it has retrieved from the server. The items in this table need to be sorted by status first and by display name second. In order to retrieve data in this order, the APIs must accept multiple sort keys/directions. | A UI that displays a table with only the page of data that it has retrieved from the server. The items in this table need to be sorted by status first and by display name second. In order to retrieve data in this order, the APIs must accept multiple sort keys/directions. | ||
| Line 15: | Line 15: | ||
|} | |} | ||
The caller can specify these parameters multiple times in order to generate a list of sort keys and sort directions. The first key listed is the primary key, the next key is the secondary key, etc. | The caller can specify these parameters multiple times in order to generate a list of sort keys and sort directions. The first key listed is the primary key, the next key is the secondary key, etc. | ||
| − | <br /> | + | <br /><br/> |
| − | For example: /volumes?sort_key=status&sort_key=display_name&sort_key=created_at&sort_dir=desc&sort_dir=desc&sort_dir=desc | + | For example: ''/volumes?sort_key=status&sort_key=display_name&sort_key=created_at&sort_dir=desc&sort_dir=desc&sort_dir=desc'' |
<br /> | <br /> | ||
'''Note:''' The "created_at" and "id" sort keys are always appended at the end of the key list (descending sort order) if they are not already specified on the request. | '''Note:''' The "created_at" and "id" sort keys are always appended at the end of the key list (descending sort order) if they are not already specified on the request. | ||
| Line 22: | Line 22: | ||
The database layer already supports multiple sort keys and directions. This blueprint will update the API layer to retrieve the sort information from the API request and pass that information down to the database layer. | The database layer already supports multiple sort keys and directions. This blueprint will update the API layer to retrieve the sort information from the API request and pass that information down to the database layer. | ||
| − | == Alternatives == | + | All sorting is handled in the [https://github.com/openstack/nova/blob/master/nova/openstack/common/db/sqlalchemy/utils.py#L64 sqlalchemy.utils.paginate_query] function. This function accepts an ORM model class as an argument and the only valid sort keys are attributes on the given model class. Therefore, the valid sort keys are limited to the model attributes on the [https://github.com/openstack/cinder/blob/master/cinder/db/sqlalchemy/models.py#L71 models.Volume] class. |
| + | |||
| + | === Alternatives === | ||
Repeating parameter key/values was chosen because glance already did it: | Repeating parameter key/values was chosen because glance already did it: | ||
<br/> | <br/> | ||
| Line 35: | Line 37: | ||
The downside of this approach is that it would require pre-defined token characters. I'm open to this solution if it is deemed better. | The downside of this approach is that it would require pre-defined token characters. I'm open to this solution if it is deemed better. | ||
| − | == REST API impact == | + | === REST API impact === |
| − | The following APIs will support multiple sort keys and directions | + | The following v2 APIs will support multiple sort keys and directions: |
<br/> | <br/> | ||
* /volumes | * /volumes | ||
* /volumes/detail | * /volumes/detail | ||
| − | == Other End user impact == | + | The existing API documentation needs to be updated to include the following new Request Parameters: |
| + | |||
| + | {| class="wikitable" | ||
| + | |- | ||
| + | ! Parameter !! Style !! Type !! Description | ||
| + | |- | ||
| + | | sort_key || query || xsd:string || Sort key (repeated for multiple), keys default to 'created_at' and 'id' | ||
| + | |- | ||
| + | | sort_dir || query || xsd:string || Sort direction, either 'asc' or 'desc' (repeated for multiple), defaults to 'desc' | ||
| + | |} | ||
| + | |||
| + | Neither the API response format nor the return codes will be modified, only the order of the volumes that are returned. | ||
| + | |||
| + | === Other End user impact === | ||
None. However, the cinderclient could be updated to accept sort direction. | None. However, the cinderclient could be updated to accept sort direction. | ||
| − | == Deployer impact == | + | The cinderclient should be updated to accept sort keys and sort directions, new parameters: |
| + | |||
| + | {| class="wikitable" | ||
| + | |- | ||
| + | ! Parameter !! Description | ||
| + | |- | ||
| + | | --sort-keys || Comma-separated list of sort keys used to specify server ordering. Each key must be paired with a sort direction value. | ||
| + | |- | ||
| + | | --sort-dirs || Comma-separated list of sort directions used to specify server ordering. Each key Must be paired with a sort key value. Valid values are 'asc' and 'desc'. | ||
| + | |} | ||
| + | |||
| + | === Deployer impact === | ||
None | None | ||
| − | == Developer impact == | + | === Developer impact === |
None | None | ||
| + | |||
| + | === Performance Impact === | ||
| + | |||
| + | All sorting will be done in the database. The choice of sort keys is limited to attributes on the models.Instance ORM class -- not every attribute key returned from a detailed query is a valid sort key. | ||
| + | |||
| + | The choice of sort keys will affect the data retrieval performance. However, limiting the choice of sort keys to the attributes on the models.Instance class reduces the performance impact (as opposed to allowing any sort key from the various cinder volumes tables). | ||
== Implementation == | == Implementation == | ||
=== Assignee(s) === | === Assignee(s) === | ||
Primary assignee: Steven Kaufer | Primary assignee: Steven Kaufer | ||
| − | + | <br/> | |
| − | None | + | Other contributors: None |
| − | == Milestones == | + | === Milestones === |
Originally approved for: Unknown | Originally approved for: Unknown | ||
<br/> | <br/> | ||
Completed: None | Completed: None | ||
| − | == Work Items == | + | === Work Items === |
Ideally the logic for processing the sort parameters would be common to all components and would be done in oslo (a similar blueprint is also being proposed in nova: https://blueprints.launchpad.net/nova/+spec/nova-pagination) | Ideally the logic for processing the sort parameters would be common to all components and would be done in oslo (a similar blueprint is also being proposed in nova: https://blueprints.launchpad.net/nova/+spec/nova-pagination) | ||
<br/><br/> | <br/><br/> | ||
| Line 67: | Line 99: | ||
<br/> | <br/> | ||
* Create common function to process the API parameters and create a list of sort keys and directions | * Create common function to process the API parameters and create a list of sort keys and directions | ||
| − | * Update v2 | + | * Update the v2 APIs to retrieve the sort information and pass down to the DB layer (requires changes to volume/api.py, db/api.py, and db\sqlalchemy\api.py) |
== Dependencies == | == Dependencies == | ||
Latest revision as of 19:35, 3 April 2014
Contents
Problem description
Pagination allows for retrieving a limited number of results instead of the entire data set. The cinder /volumes and /volumes/detail APIs support pagination and only support a single sort key and sort direction. The REST APIs need to accept multiple sort keys and directions so that the caller has more granular control over the data set sort order.
Use Case
A UI that displays a table with only the page of data that it has retrieved from the server. The items in this table need to be sorted by status first and by display name second. In order to retrieve data in this order, the APIs must accept multiple sort keys/directions.
Proposed change
The /volumes and /volumes/detail APIs will support the following parameters being repeated on the request:
| Parameter | Description |
|---|---|
| sort_key | Key used to determine sort order |
| sort_dir | Direction for with the associated sort key (asc or desc) |
The caller can specify these parameters multiple times in order to generate a list of sort keys and sort directions. The first key listed is the primary key, the next key is the secondary key, etc.
For example: /volumes?sort_key=status&sort_key=display_name&sort_key=created_at&sort_dir=desc&sort_dir=desc&sort_dir=desc
Note: The "created_at" and "id" sort keys are always appended at the end of the key list (descending sort order) if they are not already specified on the request.
The database layer already supports multiple sort keys and directions. This blueprint will update the API layer to retrieve the sort information from the API request and pass that information down to the database layer.
All sorting is handled in the sqlalchemy.utils.paginate_query function. This function accepts an ORM model class as an argument and the only valid sort keys are attributes on the given model class. Therefore, the valid sort keys are limited to the model attributes on the models.Volume class.
Alternatives
Repeating parameter key/values was chosen because glance already did it:
https://github.com/openstack/glance/blob/master/glance/api/v2/images.py#L526
However, the list of sort keys and directions could be built by splitting the associated parameter values.
For example: /volumes?sort_key=status,display_name,created_at&sort_dir=desc,desc,desc
The downside of this approach is that it would require pre-defined token characters. I'm open to this solution if it is deemed better.
REST API impact
The following v2 APIs will support multiple sort keys and directions:
- /volumes
- /volumes/detail
The existing API documentation needs to be updated to include the following new Request Parameters:
| Parameter | Style | Type | Description |
|---|---|---|---|
| sort_key | query | xsd:string | Sort key (repeated for multiple), keys default to 'created_at' and 'id' |
| sort_dir | query | xsd:string | Sort direction, either 'asc' or 'desc' (repeated for multiple), defaults to 'desc' |
Neither the API response format nor the return codes will be modified, only the order of the volumes that are returned.
Other End user impact
None. However, the cinderclient could be updated to accept sort direction.
The cinderclient should be updated to accept sort keys and sort directions, new parameters:
| Parameter | Description |
|---|---|
| --sort-keys | Comma-separated list of sort keys used to specify server ordering. Each key must be paired with a sort direction value. |
| --sort-dirs | Comma-separated list of sort directions used to specify server ordering. Each key Must be paired with a sort key value. Valid values are 'asc' and 'desc'. |
Deployer impact
None
Developer impact
None
Performance Impact
All sorting will be done in the database. The choice of sort keys is limited to attributes on the models.Instance ORM class -- not every attribute key returned from a detailed query is a valid sort key.
The choice of sort keys will affect the data retrieval performance. However, limiting the choice of sort keys to the attributes on the models.Instance class reduces the performance impact (as opposed to allowing any sort key from the various cinder volumes tables).
Implementation
Assignee(s)
Primary assignee: Steven Kaufer
Other contributors: None
Milestones
Originally approved for: Unknown
Completed: None
Work Items
Ideally the logic for processing the sort parameters would be common to all components and would be done in oslo (a similar blueprint is also being proposed in nova: https://blueprints.launchpad.net/nova/+spec/nova-pagination)
Therefore, I see 2 work items:
- Create common function to process the API parameters and create a list of sort keys and directions
- Update the v2 APIs to retrieve the sort information and pass down to the DB layer (requires changes to volume/api.py, db/api.py, and db\sqlalchemy\api.py)
Dependencies
- Related (but independent) change being proposed in nova: https://blueprints.launchpad.net/nova/+spec/nova-pagination
Testing
Tests need to be created to ensure that the data is retrieved in the specified sort order. I feel this unit tests are sufficient since these APIs already query the DB with multiple sort keys and directions ("created_at" and "id"); this blueprint is not adding support for multiple sort keys at the DB layer, only for exposing this function at the API level.
Documentation Impact
The /volumes and /volumes/detail API documentation will need to be updated to reflect the sorting parameters.
