Jump to: navigation, search

Difference between revisions of "Nova Pagination"

Line 1: Line 1:
 
== Overview ==
 
== Overview ==
−
Pagination allows for retrieving a limited number of results instead of the entire data set.  The nova ''/servers'' and ''/servers/detail'' APIs support pagination but do not allow the caller to specify how the data set is sorted.  Also, there is no mechanism to determine the total number of servers (with or without filters) without paging through the entire data set.  Lastly, there is no mechanism to jump to the n-th page of data without retrieving a marker from the (n-1)-th page of data.
+
Pagination allows for retrieving a limited number of results instead of the entire data set.  The nova ''/servers'' and ''/servers/detail'' APIs support pagination but do not allow the caller to specify how the data set is sorted.
  
 
== Proposal ==
 
== Proposal ==
Line 16: Line 16:
 
<br />
 
<br />
 
'''Note:''' The ''created_at'' and ''id'' sort keys are always appended at the end of the key list (descending sort order) if they are not already specified on the request.
 
'''Note:''' The ''created_at'' and ''id'' sort keys are always appended at the end of the key list (descending sort order) if they are not already specified on the request.
−
=== New servers/count API ===
 
−
This new API returns the total number of servers that match the given filters (optional).  The filtering support is the same as the existing servers APIs.
 
−
=== New offset parameter support ===
 
−
The current pagination support uses a marker to determine the first item on the page of data to return.  The problem with this solution is that it does allow the caller to retrieve a page anywhere in the data set without first retrieving a marker.  This blueprint is proposing the following new parameter on the ''/servers'' and ''/servers/detail'' request:
 
−
{| class="wikitable"
 
−
|-
 
−
! Parameter !! Description
 
−
|-
 
−
| offset || Index used to determine the first server to return
 
−
|}
 
−
See [http://www.gossamer-threads.com/lists/openstack/dev/2777 Getting pagination right] for more background on the marker vs. offset approach.  Supplying both ''offset'' and ''marker'' would not be supported; the caller can specific either or neither.
 
−
<br />
 
−
Once the user determines the total number of servers (using the new ''servers/count'' API), they can use this information to intelligently create an offset index.
 
  
 
== Implementation Details ==
 
== Implementation Details ==
 
=== Multiple sort keys and directions ===
 
=== Multiple sort keys and directions ===
 
The nova database layer already supports multiple sort keys and sort directions (see paginate_query function in [https://github.com/openstack/nova/blob/master/nova/openstack/common/db/sqlalchemy/utils.py utils.py]).  The layers above this need to be modified to accept a list of sort keys and sort direction.  Also, [https://github.com/openstack/nova/blob/master/nova/api/openstack/common.py common.py] needs to be modified to create the list of sort keys and directions from the request parameters.
 
The nova database layer already supports multiple sort keys and sort directions (see paginate_query function in [https://github.com/openstack/nova/blob/master/nova/openstack/common/db/sqlalchemy/utils.py utils.py]).  The layers above this need to be modified to accept a list of sort keys and sort direction.  Also, [https://github.com/openstack/nova/blob/master/nova/api/openstack/common.py common.py] needs to be modified to create the list of sort keys and directions from the request parameters.
−
 
−
=== New servers/count API ===
 
−
This new API is being proposed for both V2 and V3.  At the database layer, the ''count'' function can be invoked on query object in order to retrieve the total number of rows the match.  In order to expose this data, new functions must be added to all of the layers between the database the API.
 
−
<br />
 
−
<br />
 
−
For example, a ''get_count'' function in the [https://github.com/openstack/nova/blob/master/nova/compute/api.py compute api.py], a ''get_count_by_filters'' function in the [https://github.com/openstack/nova/blob/master/nova/objects/instance.py insance.py InstanceList], an ''instance_count_by_filters'' function in [https://github.com/openstack/nova/blob/master/nova/db/api.py db api.py], and an ''instance_count_by_filters'' function in [https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py sqlalchemy api.py].
 
−
<br />
 
−
<br />
 
−
The reply to this query contains the integer number of servers that match the given filters.
 
−
<br />
 
−
<br />
 
−
Programmatically, the logic to setup the search options (in the [https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/servers.py v2 API servers.py]) would be broken into a staticmethod so that it can be re-used by a new count API extension.  Similarly, the logic to setup the query (in [https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py sqlalchemy api.py]) would be broken into a new private function so that it can be shared by the existing ''instance_get_all_by_filters'' function and the new ''instance_count_by_filters'' function.  The count API would also be added to the V3 API.
 
−
 
−
=== New offset parameter support ===
 
−
The nova database layer already supports an offset index (see paginate_query function in [https://github.com/openstack/nova/blob/master/nova/openstack/common/db/sqlalchemy/utils.py utils.py])  The implementation details are similar to those for the multiple sort keys and directions, the layers between the API and database need to be modified to accept a new offset parameter.  An exception will be thrown in the caller specifies both marker and offset.
 
−
 
−
== Implementation Questions ==
 
−
* How is the v2 API extension written to create ''/servers/count''?  There are many examples of how to write an extension that targets a specifc server (ie, ''/servers/<uuid>/count'').  How would the extension be written so that it is non-targeted?
 

Revision as of 13:37, 20 March 2014

Overview

Pagination allows for retrieving a limited number of results instead of the entire data set. The nova /servers and /servers/detail APIs support pagination but do not allow the caller to specify how the data set is sorted.

Proposal

Multiple sort keys and directions

The /servers and /servers/detail APIs will support the following parameters being repeated on the request:

Parameter Description
sort_key Key used to determine sort order
sort_dir Direction for with the associated sort key (asc or desc)

The caller can specify these parameters multiple times in order to generate a list of sort keys and sort directions. The first key listed is the primary key, the next key is the secondary key, etc.
Note: The created_at and id sort keys are always appended at the end of the key list (descending sort order) if they are not already specified on the request.

Implementation Details

Multiple sort keys and directions

The nova database layer already supports multiple sort keys and sort directions (see paginate_query function in utils.py). The layers above this need to be modified to accept a list of sort keys and sort direction. Also, common.py needs to be modified to create the list of sort keys and directions from the request parameters.

Retrieved from "https://wiki.openstack.org/w/index.php?title=Nova_Pagination&oldid=46054"