Google Cloud datastore pagination

Question

We store data in Google Cloud Datastore. We want to provide APIs to our users. Our APIs's pagination specification is header based same as github API. We want users use page parameter.

pagination specification

e.g.

Link: <https://api.github.com/search/code?q=addClass+user%3Amozilla&page=15>; rel="next",
  <https://api.github.com/search/code?q=addClass+user%3Amozilla&page=34>; rel="last",
  <https://api.github.com/search/code?q=addClass+user%3Amozilla&page=1>; rel="first",
  <https://api.github.com/search/code?q=addClass+user%3Amozilla&page=13>; rel="prev"

https://developer.github.com/guides/traversing-with-pagination/

API

End-Users <--- Backend Code (PHP) <--- Google Cloud Datastore

As you know Google Cloud Datastore recommends using cursor for improving performance and cost. But we don't want end-users use cursor. Is it possible that end-users use integer page number instead of cursor and cursor is used in backend?

We use Google's PHP client.

https://github.com/google/google-api-php-client-services

Answer 1

I believe you can have something like an OFFSET by using GQL, but such operation will cost you a lot of money (doing the equivalent of LIMIT 1000, 10 will count as 1,010 reads - not just the 10 you actually get back).

Cut down costs of OFFSET for pagination

Let's say your page size is 10 items and a user asks to jump to page 5. You'll need to query for the first 40 entities, get the cursor and run the query again, now providing the cursor and limiting to 10.

It's recommended that, in the first query, you fetch with keys_only=True. This way you can:

• Save money, since a keys-only query is counted as a single entity read for the query itself (see Datastore pricing info)
• Have faster queries (see this benchmark - the page takes a while to load).