Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to implement robust pagination with a RESTful API when the resultset can change?

I'm implementing a RESTful API which exposes Orders as a resource and supports pagination through the resultset:

GET /orders?start=1&end=30

where the orders to paginate are sorted by ordered_at timestamp, descending. This is basically approach #1 from the SO question Pagination in a REST web application.

If the user requests the second page of orders (GET /orders?start=31&end=60), the server simply re-queries the orders table, sorts by ordered_at DESC again and returns the records in positions 31 to 60.

The problem I have is: what happens if the resultset changes (e.g. a new order is added) while the user is viewing the records? In the case of a new order being added, the user would see the old order #30 in first position on the second page of results (because the same order is now #31). Worse, in the case of a deletion, the user sees the old order #32 in first position on the second page (#31) and wouldn't see the old order #31 (now #30) at all.

I can't see a solution to this without somehow making the RESTful server stateful (urg) or building some pagination intelligence into each client... What are some established techniques for dealing with this?

For completeness: my back-end is implemented in Scala/Spray/Squeryl/Postgres; I'm building two front-end clients, one in backbone.js and the other in Python Django.

like image 962
Alex Dean Avatar asked Dec 13 '11 10:12

Alex Dean


People also ask

What is the best approach for pagination when exploring record sets?

So if you're looking for a quick way to implement pagination then offset is the way to go. Especially if that data is not real-time. No need to complicate things where it is not needed. Having implemented cursors myself, I would say that it wasn't too hard to get it working for 90% of cases.

Which method is considered most efficient for pagination in REST API?

The Offset method is the most common way to paginate resources (with an offset on query), but it's less efficient than the Search-after method.

What is the best approach for pagination in API?

Offset Pagination This is the simplest form of paging. Limit/Offset became popular with apps using SQL databases which already have LIMIT and OFFSET as part of the SQL SELECT Syntax. Very little business logic is required to implement Limit/Offset paging.

How does REST API implement pagination in Java?

To add pagination to a RestController we first need the input parameters. Using the Pageable class, the "page", "size" and "sort" parameters can be passed to the endpoint. These are automatically bound by Spring Boot into the object.


2 Answers

The way I'd do it, is to make the indices from old to new. So they never change. And then when querying without any start parameter, return the newest page. Also the response should contain an index indicating what elements are contained, so you can calculate the indices you need to request for the next older page. While this is not exactly what you want, it seems like the easiest and cleanest solution to me.

Initial request: GET /orders?count=30 returns:

{
  "start"=1039;
  "count"=30;
  ...//data
}

From this the consumer calculates that he wants to request:

Next requests: GET /orders?start=1009&count=30 which then returns:

{
  "start"=1009;
  "count"=30;
  ...//data
}

Instead of raw indices you could also return a link to the next page:

{
  "next"="/orders?start=1009&count=30";
}

This approach breaks if items get inserted or deleted in the middle. In that case you should use some auto incrementing persistent value instead of an index.

like image 77
CodesInChaos Avatar answered Oct 25 '22 01:10

CodesInChaos


The sad truth is that all the sites I see have pagination "broken" in that sense, so there must not be an easy way to achieve that.

A quick workaround could be reversing the ordering, so the position of the items is absolute and unchanging with new additions. From your front page you can give the latest indices to ensure consistent navigation from up there.

  • Pros: same url gives the same results
  • Cons: there's no evident way to get the latest elements... Maybe you could use negative indices and redirect the result page to the absolute indices.
like image 32
fortran Avatar answered Oct 25 '22 01:10

fortran