Difference between revisions of "Nova Pagination"
(Nova Pagination Enhancements) |
|||
| Line 1: | Line 1: | ||
== Overview == | == Overview == | ||
| − | Pagination allows for retrieving a limited number of results instead of the entire data set. The nova ''/servers'' and ''/servers/detail'' APIs support pagination but do not allow the caller to specify | + | Pagination allows for retrieving a limited number of results instead of the entire data set. The nova ''/servers'' and ''/servers/detail'' APIs support pagination but do not allow the caller to specify how the data set is sorted. Also, there is no mechanism to determine the total number of servers (with or without filters) without paging through the entire data set. Lastly, there is no mechanism to jump to the n-th page of data without retrieving a marker from the (n-1)-th page of data. |
== Proposal == | == Proposal == | ||
| Line 13: | Line 13: | ||
| sort_dir || Direction for with the associated sort key (''asc'' or ''desc'') | | sort_dir || Direction for with the associated sort key (''asc'' or ''desc'') | ||
|} | |} | ||
| − | The caller can | + | The caller can specify these parameters multiple times in order to generate a list of sort keys and sort directions. The first key listed is the primary key, the next key is the secondary key, etc. |
<br /> | <br /> | ||
| − | '''Note:''' The ''created_at'' and ''id'' sort keys are always appended at the end of the key list (descending sort order) if they | + | '''Note:''' The ''created_at'' and ''id'' sort keys are always appended at the end of the key list (descending sort order) if they are not already specified on the request. |
=== servers/count API === | === servers/count API === | ||
| − | This new API | + | This new API returns the total number of servers that match the given filters (optional). The filtering support is the same as the existing servers APIs. |
=== offset parameter support === | === offset parameter support === | ||
The current pagination support uses a marker to determine the first item on the page of data to return. The problem with this solution is that it does allow the caller to retrieve a page anywhere in the data set without first retrieving a marker. This blueprint is proposing the following new parameter on the ''/servers'' and ''/servers/detail'' request: | The current pagination support uses a marker to determine the first item on the page of data to return. The problem with this solution is that it does allow the caller to retrieve a page anywhere in the data set without first retrieving a marker. This blueprint is proposing the following new parameter on the ''/servers'' and ''/servers/detail'' request: | ||
| Line 26: | Line 26: | ||
| offset || Index used to determine the first server to return | | offset || Index used to determine the first server to return | ||
|} | |} | ||
| − | See [http://www.gossamer-threads.com/lists/openstack/dev/2777 Getting pagination right] for more background on the marker vs. offset approach. Supplying both 'offset' and 'marker' would not be supported; the caller can specific either or neither. | + | See [http://www.gossamer-threads.com/lists/openstack/dev/2777 Getting pagination right] for more background on the marker vs. offset approach. Supplying both ''offset'' and ''marker'' would not be supported; the caller can specific either or neither. |
<br /> | <br /> | ||
| − | Once the user determines the total number of servers (using the new 'count' API), they can use this information to intelligently create an offset index. | + | Once the user determines the total number of servers (using the new ''servers/count'' API), they can use this information to intelligently create an offset index. |
== Implementation Details == | == Implementation Details == | ||
=== Multiple sort keys and directions === | === Multiple sort keys and directions === | ||
| − | The nova database layer already supports multiple sort keys and sort directions (see paginate_query function in [https://github.com/openstack/nova/blob/master/nova/openstack/common/db/sqlalchemy/utils.py utils.py]). The layers above this need to be modified to accept a list of sort keys and sort direction. Also, [https://github.com/openstack/nova/blob/master/nova/api/openstack/common.py common.py] needs to be modified | + | The nova database layer already supports multiple sort keys and sort directions (see paginate_query function in [https://github.com/openstack/nova/blob/master/nova/openstack/common/db/sqlalchemy/utils.py utils.py]). The layers above this need to be modified to accept a list of sort keys and sort direction. Also, [https://github.com/openstack/nova/blob/master/nova/api/openstack/common.py common.py] needs to be modified to create the list of sort keys and directions from the request parameters. |
=== servers/count API === | === servers/count API === | ||
| − | This new API is being proposed for both V2 and V3. At the database layer, | + | This new API is being proposed for both V2 and V3. At the database layer, the ''count'' function can be invoked on query object in order to retrieve the total number of rows the match. In order to expose this data, new functions must be added to all of the layers between the database the API. |
<br /> | <br /> | ||
<br /> | <br /> | ||
| − | For example, a ''get_count'' function in the [https://github.com/openstack/nova/blob/master/nova/compute/api.py comput api.py], a ''get_count_by_filters'' function in the [https://github.com/openstack/nova/blob/master/nova/objects/instance.py insance.py InstanceList], an | + | For example, a ''get_count'' function in the [https://github.com/openstack/nova/blob/master/nova/compute/api.py comput api.py], a ''get_count_by_filters'' function in the [https://github.com/openstack/nova/blob/master/nova/objects/instance.py insance.py InstanceList], an ''instance_count_by_filters'' function in [https://github.com/openstack/nova/blob/master/nova/db/api.py db api.py], and an ''instance_count_by_filters'' function in [https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py sqlalchemy api.py]. |
<br /> | <br /> | ||
<br /> | <br /> | ||
| − | The reply to this query | + | The reply to this query contains the integer number of servers that match the given filters. |
<br /> | <br /> | ||
<br /> | <br /> | ||
| − | + | Programmatically, the logic to setup the search options (in the [https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/servers.py v2 API servers.py]) would be broken into a staticmethod so that it can be re-used by a new count API extension. Similarly, the logic to setup the query (in [https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py sqlalchemy api.py]) would be broken into a new private function so that it can be shared by the existing ''instance_get_all_by_filters'' function and the new ''instance_count_by_filters'' function. The count API would also be added to the V3 API. | |
=== offset parameter support === | === offset parameter support === | ||
| − | The nova database layer already supports an offset index (see paginate_query function in [https://github.com/openstack/nova/blob/master/nova/openstack/common/db/sqlalchemy/utils.py utils.py]) The implementation details are similar to those for the multiple sort keys and directions, the layers need to be modified to accept a new offset parameter. An exception will be thrown in the caller specifies both marker and offset. | + | The nova database layer already supports an offset index (see paginate_query function in [https://github.com/openstack/nova/blob/master/nova/openstack/common/db/sqlalchemy/utils.py utils.py]) The implementation details are similar to those for the multiple sort keys and directions, the layers between the API and database need to be modified to accept a new offset parameter. An exception will be thrown in the caller specifies both marker and offset. |
Revision as of 15:59, 27 February 2014
Contents
Overview
Pagination allows for retrieving a limited number of results instead of the entire data set. The nova /servers and /servers/detail APIs support pagination but do not allow the caller to specify how the data set is sorted. Also, there is no mechanism to determine the total number of servers (with or without filters) without paging through the entire data set. Lastly, there is no mechanism to jump to the n-th page of data without retrieving a marker from the (n-1)-th page of data.
Proposal
Multiple sort keys and directions
The /servers and /servers/detail APIs will support the following parameters being repeated on the request:
| Parameter | Description |
|---|---|
| sort_key | Key used to determine sort order |
| sort_dir | Direction for with the associated sort key (asc or desc) |
The caller can specify these parameters multiple times in order to generate a list of sort keys and sort directions. The first key listed is the primary key, the next key is the secondary key, etc.
Note: The created_at and id sort keys are always appended at the end of the key list (descending sort order) if they are not already specified on the request.
servers/count API
This new API returns the total number of servers that match the given filters (optional). The filtering support is the same as the existing servers APIs.
offset parameter support
The current pagination support uses a marker to determine the first item on the page of data to return. The problem with this solution is that it does allow the caller to retrieve a page anywhere in the data set without first retrieving a marker. This blueprint is proposing the following new parameter on the /servers and /servers/detail request:
| Parameter | Description |
|---|---|
| offset | Index used to determine the first server to return |
See Getting pagination right for more background on the marker vs. offset approach. Supplying both offset and marker would not be supported; the caller can specific either or neither.
Once the user determines the total number of servers (using the new servers/count API), they can use this information to intelligently create an offset index.
Implementation Details
Multiple sort keys and directions
The nova database layer already supports multiple sort keys and sort directions (see paginate_query function in utils.py). The layers above this need to be modified to accept a list of sort keys and sort direction. Also, common.py needs to be modified to create the list of sort keys and directions from the request parameters.
servers/count API
This new API is being proposed for both V2 and V3. At the database layer, the count function can be invoked on query object in order to retrieve the total number of rows the match. In order to expose this data, new functions must be added to all of the layers between the database the API.
For example, a get_count function in the comput api.py, a get_count_by_filters function in the insance.py InstanceList, an instance_count_by_filters function in db api.py, and an instance_count_by_filters function in sqlalchemy api.py.
The reply to this query contains the integer number of servers that match the given filters.
Programmatically, the logic to setup the search options (in the v2 API servers.py) would be broken into a staticmethod so that it can be re-used by a new count API extension. Similarly, the logic to setup the query (in sqlalchemy api.py) would be broken into a new private function so that it can be shared by the existing instance_get_all_by_filters function and the new instance_count_by_filters function. The count API would also be added to the V3 API.
offset parameter support
The nova database layer already supports an offset index (see paginate_query function in utils.py) The implementation details are similar to those for the multiple sort keys and directions, the layers between the API and database need to be modified to accept a new offset parameter. An exception will be thrown in the caller specifies both marker and offset.
