Django Rest Framework Pagination Since ID

Question

I have built an API with Django Rest Framework. I want to change pagination for a better user experience.

The problem:

The client makes a call to request all posts. The request looks like:

http://website.com/api/v1/posts/?page=1

This returns the first page of posts. However, new posts are always being created. So when the user requests:

http://website.com/api/v1/posts/?page=2

The posts are almost always the same as page 1 (since new data is always coming in and we are ordering by -created).

Possible Solution?

I had the idea of sending an object id along with the request so that when we grab the posts. We grab them with respect to the last query.

http://website.com/api/v1/posts/?page=2&post_id=12345

And when we paginate we filter where post_id < 12345

But this only works assuming our post_id is an integer.

Right now I'm currently only using a basic ListAPIView

class PostList(generics.ListAPIView):
    """
    API endpoint that allows posts to be viewed
    """
    serializer_class = serializers.PostSerializer # just a basic serializer
    model = Post

Is there a better way of paginating? so that the next request for that user's session doesn't look like the first when new data is being created.

Answer 1

You're looking for CursorPagination, from the DRF docs:

Cursor Pagination provides the following benefits:

• Provides a consistent pagination view. When used properly CursorPagination ensures that the client will never see the same item twice when paging through records, even when new items are being inserted by other clients during the pagination process.
• Supports usage with very large datasets. With extremely large datasets pagination using offset-based pagination styles may become inefficient or unusable. Cursor based pagination schemes instead have fixed-time properties, and do not slow down as the dataset size increases.

You can also use -created as the ordering field as you mentioned above.

Answer 2

How about caching the queryset? So that the next page is served from the same query set, and not from a new one. And then you could use a parameter to get a new queryset when you want.

Something like this:

from django.core.cache import cache

class PostList(generics.ListAPIView):

    def get_queryset(self):
        qs_key = str(self.request.user.id) + '_key'
        if 'refresh' in self.request.QUERY_PARAMS:
            # get a new query set
            qs = Post.objects.all()
            cache.set(qs_key, qs)
        return cache.get(qs_key)

So basically, only when your url will be like this:

http://website.com/api/v1/posts/?refersh=whatever

the request will return new data.

UPDATE

In order to provide each user with it's own set of posts, the cache key must contain an unique identifier (which might be the user's ID): I also updated the code.

The downside to this approach is that for a very large number of users and a large number of posts for each user, it might not work very well.

So, here is my second idea

Use a TimeStamped model for the Post model, and filter the query set based on the created field.

I don't know much about your models and how exactly they are built, so I guess you will have to choose which solution is best for your app :)