Zaqar/specs/storage/message-pagination

Message Pagination

1. Markers must be decoupled from IDs to allow for future optimizations in storage drivers.

 a. Return as last item in the iterator
 b. Return in body of StopIteration exception
 c. Return marker as part of each doc

   [
       {
           "id": "foo",
           "marker": "foo",
           "ttl": 400,
           "body": {}
       }
   ]

(c) is the current recomendation, because it is both efficient and conforms to the Principle of Least Surprise. It is efficient because messages can be streamed and the transport can pull the marker off the last message, meaning the driver does not have to do an additional query to create a (marker, iter) tuple. Furthermore, if marker and id are the same string, it does not cost much of anything to duplicate those fields.

2. Marker semantics

Requirements

Duplicates are not allowed to be returned between pages
No message may be skipped/dropped/lost

Ergo, if markers are timestamp-based, the storage driver must track the messages that were returned for each client (based on uuid) for a moving window of time that is wide enough to guarantee duplicates can be filtered out.

Alternatively, a driver for a distributed store such as MongoDB could use a ZMQ-based oracle to generate monotonic, unique IDs. This needs to be benchmarked.

In any case, it is the driver's responsibility to ensure markers unambiguously determine the next page of results to return, without any risk of duplicating messages from a previous page, or skipping any messages.