Pagination issue in RESTful API design

Question

I am designing a RESTful API for a mobile application I am working on. My problem is with large collections containing many items. I understand that a good practice is to paginate large number of results in a collection.

I have read the Facebook Graph API doc (https://developers.facebook.com/docs/graph-api/using-graph-api/v2.2), Twitter cursors doc (https://dev.twitter.com/overview/api/cursoring), GitHub API doc (https://developer.github.com/v3/) and this post (API pagination best practices).

Consider an example collection /resources in my API that contains 100 items named resource1 to resource100 and sorted descending. This is the response you will get upon a GET request (GET http://api.path.com/resources?limit=5):

{
    "_links": {
        "self": { "href": "/resources?limit=5&page=1" },
        "last": { "href": "/resources?limit=5&page=7" },
        "next": { "href": "/resources?limit=5&page=2" }
    },

    "_embedded": {
        "records": [ 
            { resource 100 },
            { resource 99 },
            { resource 98 },
            { resource 97 },
            { resource 96 }
        ]
    }
}

Now my problem is a scenario like this:

1- I GET /resources with above contents.

2- After that, something is added to the resources collection (say another device adds a new resource for this account). So now I have 101 resources.

3- I GET /resources?limit=5&page=2 as the initial response suggests will contain the next page of my results. The response would be like this:

{
    "_links": {
        "self": { "href": "/history?page=2&limit=5" },
        "last": { "href": "/history?page=7&limit=5" },
        "next": { "href": "/history?page=3&limit=5" }
    },

    "_embedded": {
        "records": [ 
            { resource 96 },
            { resource 95 },
            { resource 94 },
            { resource 93 },
            { resource 92 }
        ]
    }
}

As you can see resource 96 is repeated in both pages (Or similar problem may happen if a resource gets deleted in step 2, in that case one resource will be lost).

Since I want to use this in a mobile app and in one list, I have to append the resources of each API call to the one before it so I can have a complete list. But this is troubling. Please let me know if you have a suggestion. Thank you in advance.

P.S: I have considered timestamp like query strings instead of cursor based pagination, but that will make problems somewhere else for me. (let me know if you need more info about that.)

Answer 1

We just implemented something similar to this for a mobile app via a REST API. The mobile app passed an additional query parameter which represents a timestamp at which elements in the page should be "frozen".

So your first request would look something like GET /resources?limit=5&page=1&from=2015-01-25T05:10:31.000Z and then the second page request (some time later) would increment the page count but keep the same timestamp: GET /resources?limit=5&page=2&from=2015-01-25T05:10:31.000Z

This also gives the mobile app control if it wants to differentiate a "soft" page (preserving the timestamp of the request of page 1) from a "hard refresh" page (resetting the timestamp to the current time).