How to do result post-processing before rendering with django endless-pagination?

Question

I am trying to figure out if it's possible to do view post-processing on my queryset before rendering it in a django template that uses django-endless-pagination for infinite scroll.

I have view-specific logic that omits certain results from the queryset based on context, as well as adding attributes to the objects in the list for use by the templates. This logic cannot be executed via SQL as it is not inherent to the model. It must be done in python.

With django-endless-pagination and other pre-rolled django pagination modules, all the logic seems to be executed by templatetags, thus preventing the ability to do business logic before the rendering stage (which is a django tenet).

Because my view logic runs through the result set before the template tags execute, I'm losing the optimizations offered by this module (like SQL queries with paging e.g. limit 20; offset 20). My code traverses the entire unpaged result list every time the user pages, bypassing the lazy pagination benefits offered by the template tag.

Short of moving my code into the pagination module directly (which I'd rather not do and would require adding a bunch of extra data into the request context for use in a tag), is there any alternative?

Thanks!

Answer 1

If you look lazy_paginate tag use LazyPaginator class to process the queryset. You can override that class to serve your purpose. In order to do that you need to write Custom Template Tag. More instructions in the code comments.

*my_app/templatetags/custom_pagination_tags.py*

from django import template
from endless_pagination.templatetags.endless import paginate
from endless_pagination.paginators import LazyPaginator

register = template.Library()

Class CustomLazyPaginator(LazyPaginator):

    def page(self, number):
        page_obj = super(CustomLazyPaginator, self).page(number)
        # page function returns page object from that you could access object_list
        object_list = page_obj.object_list
        # Do some processing here for your queryset
        # Do not remove elements otherwise you will put your self in trouble
        # Just add some values in objects as you wanted to
        page_obj.object_list = object_list # override here
        return page_obj

@register.tag
def custom_lazy_paginate(parser, token):
    return paginate(parser, token, paginator_class=CustomLazyPaginator)

Now in template load your custom template tags and use that instead:

{% load custom_pagination_tags %}

{% custom_lazy_paginate queryset %}

Difficult: First Approach To Access Request Context In CustomLazyPaginator Class

Yes there is a way to pass the request context, but in order to do that you need to override paginate tag and also the render method of PaginateNode as you can see here when it calls the paginator_class it does not pass any context information. Below are the steps to achieve that:

Add __init__ method in CustomLazyPaginator:

def __init__(self, *args, **kwargs):
    self.context = kwargs.pop('context', None)
    super(CustomLazyPaginator, self).__init__(*args, **kwargs)

Copy the paginate tag and change the return statement from PaginateNode(paginator_class, objects, **kwargs) to CustomPaginateNode(paginator_class, objects, **kwargs) we will write CustomPaginateNode below.

from endless_pagination.templatetags.endless import PAGINATE_EXPRESSION

@register.tag
def paginate(parser, token, paginator_class=None):
    # Validate arguments.
    try:
        tag_name, tag_args = token.contents.split(None, 1)
    except ValueError:
        msg = '%r tag requires arguments' % token.contents.split()[0]
        raise template.TemplateSyntaxError(msg)

    # Use a regexp to catch args.
    match = PAGINATE_EXPRESSION.match(tag_args)
    if match is None:
        msg = 'Invalid arguments for %r tag' % tag_name
        raise template.TemplateSyntaxError(msg)

    # Retrieve objects.
    kwargs = match.groupdict()
    objects = kwargs.pop('objects')

    # The variable name must be present if a nested context variable is passed.
    if '.' in objects and kwargs['var_name'] is None:
        msg = (
            '%(tag)r tag requires a variable name `as` argumnent if the '
            'queryset is provided as a nested context variable (%(objects)s). '
            'You must either pass a direct queryset (e.g. taking advantage '
            'of the `with` template tag) or provide a new variable name to '
            'store the resulting queryset (e.g. `%(tag)s %(objects)s as '
            'objects`).'
        ) % {'tag': tag_name, 'objects': objects}
        raise template.TemplateSyntaxError(msg)

    # Call the node.
    return CustomPaginateNode(paginator_class, objects, **kwargs)

Remove the following import which we call earlier to avoid calling original paginate function:

from endless_pagination.templatetags.endless import paginate

Override the render method of PaginateNode to pass context to our CustomLazyPaginator class:

from endless_pagination.templatetags.endless import PaginateNode
from endless_pagination import (
    settings,
    utils,
)

class CustomPaginateNode(PaginateNode):
    def render(self, context):
        # Handle page number when it is not specified in querystring.
        if self.page_number_variable is None:
            default_number = self.page_number
        else:
            default_number = int(self.page_number_variable.resolve(context))

        # Calculate the number of items to show on each page.
        if self.per_page_variable is None:
            per_page = self.per_page
        else:
            per_page = int(self.per_page_variable.resolve(context))

        # Calculate the number of items to show in the first page.
        if self.first_page_variable is None:
            first_page = self.first_page or per_page
        else:
            first_page = int(self.first_page_variable.resolve(context))

        # User can override the querystring key to use in the template.
        # The default value is defined in the settings file.
        if self.querystring_key_variable is None:
            querystring_key = self.querystring_key
        else:
            querystring_key = self.querystring_key_variable.resolve(context)

        # Retrieve the override path if used.
        if self.override_path_variable is None:
            override_path = self.override_path
        else:
            override_path = self.override_path_variable.resolve(context)

        # Retrieve the queryset and create the paginator object.
        objects = self.objects.resolve(context)
        paginator = self.paginator(
            objects, per_page, first_page=first_page, orphans=settings.ORPHANS,
            context=context) # <--- passing context here

    # Normalize the default page number if a negative one is provided.
    if default_number < 0:
        default_number = utils.normalize_page_number(
            default_number, paginator.page_range)

    # The current request is used to get the requested page number.
    page_number = utils.get_page_number_from_request(
        context['request'], querystring_key, default=default_number)

    # Get the page.
    try:
        page = paginator.page(page_number)
    except EmptyPage:
        page = paginator.page(1)

    # Populate the context with required data.
    data = {
        'default_number': default_number,
        'override_path': override_path,
        'page': page,
        'querystring_key': querystring_key,
    }
    context.update({'endless': data, self.var_name: page.object_list})
    return ''

Simple: Second Approach To Access Request Context In CustomLazyPaginator Class

Just install django-contrib-requestprovider and add it in middleware in django's settings.py and access current request any where you want as:

from gadjo.requestprovider.signals import get_request

http_request = get_request()