django-filter messing around with empty field

Question

I setup django-filter to filter some of my listings. Here is one of them, with a custom form:

class BookingListFiltersForm(forms.Form):

    state__in = forms.MultipleChoiceField(
        choices=Booking.STATE_CHOICES, required=False,
        label=_("État"), widget=forms.CheckboxSelectMultiple)
    source__in = forms.ModelMultipleChoiceField(
        queryset=Platform.objects.all(), required=False,
        label=_("Source"), widget=ModelSelect2Multiple(
            url='autocomplete:platform'))


class BookingManagerFilter(filters.FilterSet):

    payments__date = filters.DateFilter(method='payments__date_filter')
    payments__method = filters.ChoiceFilter(
        method='payments__method_filter',
        choices=BookingPayment.METHOD_CHOICES,
    )

    class Meta:
        model = Booking
        fields = {
            'period': [
                'endswith', 'endswith__gte', 'endswith__lte',
                'startswith', 'startswith__gte', 'startswith__lte',
            ],
            'state': ['in'],
            'source': ['in'],
            'booking_date': ['date', 'date__lte', 'date__gte'],
            'accommodation': ['in'],
            'guest': ['exact']
        }

    def get_form_class(self):
        return BookingListFiltersForm

    def payments__date_filter(self, queryset, name, value):
        return queryset.filter(**{name: value})

    def payments__method_filter(self, queryset, name, value):
        return queryset.filter(**{name: value})

The form is submitted by GET method. When the field "source__in" is empty, the querystring looks like this "?state__in=1". In such case, I have no result in my page (which is unexpected, if a field is not filled in, I would expect that the results are not filtered upon this field).

I looked at debug toolbar to have more information about the executed SQL query. Surprisingly, I found no SQL query for the related queryset! (while if querystring is "?state__in=1&source__in=2" for instance, the result is as expected, and I can find related queries in debug toolbar)

So I tried to force the impression of the SQL query using print(str(filters.qs.query)). New surprise, this triggered an EmptyResultSet exception:

Traceback:

File "/home/tony/.venvs/cocoonr/lib/python3.6/site-packages/django/core/handlers/exception.py" in inner
  35.             response = get_response(request)

File "/home/tony/.venvs/cocoonr/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  128.                 response = self.process_exception_by_middleware(e, request)

File "/home/tony/.venvs/cocoonr/lib/python3.6/site-packages/django/core/handlers/base.py" in _get_response
  126.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/home/tony/.venvs/cocoonr/lib/python3.6/site-packages/django/views/generic/base.py" in view
  69.             return self.dispatch(request, *args, **kwargs)

File "/home/tony/.venvs/cocoonr/lib/python3.6/site-packages/django/utils/decorators.py" in _wrapper
  62.             return bound_func(*args, **kwargs)

File "/home/tony/.venvs/cocoonr/lib/python3.6/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
  21.                 return view_func(request, *args, **kwargs)

File "/home/tony/.venvs/cocoonr/lib/python3.6/site-packages/django/utils/decorators.py" in bound_func
  58.                 return func.__get__(self, type(self))(*args2, **kwargs2)

File "/home/tony/Workspace/cocoonr/utils/views/manager.py" in dispatch
  29.         return super().dispatch(*args, **kwargs)

File "/home/tony/.venvs/cocoonr/lib/python3.6/site-packages/django/views/generic/base.py" in dispatch
  89.         return handler(request, *args, **kwargs)

File "/home/tony/.venvs/cocoonr/lib/python3.6/site-packages/django/views/generic/list.py" in get
  142.         self.object_list = self.get_queryset()

File "/home/tony/Workspace/cocoonr/booking/views/manager.py" in get_queryset
  73.         queryset = super().get_queryset()

File "/home/tony/Workspace/cocoonr/utils/views/common.py" in get_queryset
  118.         print(self.filters.qs.query)

File "/home/tony/.venvs/cocoonr/lib/python3.6/site-packages/django/db/models/sql/query.py" in __str__
  252.         sql, params = self.sql_with_params()

File "/home/tony/.venvs/cocoonr/lib/python3.6/site-packages/django/db/models/sql/query.py" in sql_with_params
  260.         return self.get_compiler(DEFAULT_DB_ALIAS).as_sql()

File "/home/tony/.venvs/cocoonr/lib/python3.6/site-packages/django/db/models/sql/compiler.py" in as_sql
  461.                 where, w_params = self.compile(self.where) if self.where is not None else ("", [])

File "/home/tony/.venvs/cocoonr/lib/python3.6/site-packages/django/db/models/sql/compiler.py" in compile
  393.             sql, params = node.as_sql(self, self.connection)

File "/home/tony/.venvs/cocoonr/lib/python3.6/site-packages/django/db/models/sql/where.py" in as_sql
  98.                     raise EmptyResultSet

Exception Type: EmptyResultSet at /manager/booking/bookings/
Exception Value: 

Now I am stuck, I have no idea what is going wrong and how to debug any further.

I tried to pass the following querystring for the sake of testing: "?state__in=1&source__in=". In such case, the filtering works properly, but the filter form shows an error "« » is not a valid value" for the field "source__in".

In addition, here is the relevant mixin in utils/views/common.py:

class ListFilterMixin:

    filters_class = None
    default_filters = None

    @cached_property
    def filters(self):
        return self.get_filters()

    def get_filters(self):
        if self.filters_class:
            qstring = self.request.GET
            if not qstring and self.default_filters:
                qstring = QueryDict(self.default_filters)
            return self.filters_class(
                qstring, self.get_unfiltered_queryset(), request=self.request)
        else:
            return None

    def get_queryset(self):
        print(self.filters.qs.query)  # <--- Line 118
        # ...

    def get_unfiltered_queryset(self):
        return super().get_queryset()

And the view class in booking/views/manager.py:

class BookingListView(ListView):
    """List of all bookings."""

    model = Booking
    default_filters = 'state__in=1'
    filters_class = BookingManagerFilter
    paginate_by = 30
    ordering = '-pk'

    def get_queryset(self):
        queryset = super().get_queryset()  # <--- Line 73
        # ...

Also, so you have the full inheritence tree, note that the ListView used above is utils.views.manager.ListView:

class ListView(BulkActionsMixin, ManagerMixin, BaseListView):
    pass

And BaseListView is utils.views.common.ListView:

class ListView(ListFilterMixin, AgencyMixin, ContextMixin, BaseListView):
    pass

The last BaseListView is django.views.generic.list.ListView.

---

Using ipdb to debug as suggested by Kamil, I noticed a strange thing that is likely the cause of this behavior:

ipdb> next
> /home.tony/.venvs/cocoonr/lib/python3.6/site-packages/django_filters/filters.py(167)filter()
    166     def filter(self, qs, value):
--> 167         if value != self.null_value:
    168             return super().filter(qs, value)

ipdb> self.null_value
'null'
ipdb> value
<QuerySet []>
ipdb> self.field_name
'source'
ipdb> self.lookup_expr
'in'
ipdb> 

So the subsequent code, considers source__in is not empty and adds source__in=empty_queryset to the filters. I guess django then guess that the result cannot evaluate to a non-empty queryset and saves a useless query.

Is it a bug in django-filters or am I doing something wrong?

Answer 1

I finally figured out the issue.

Apparently django-filters does not handle correctly the lookup in for foreign keys. The default filter for source__in for instance, is ModelChoiceFilter. So I had to explicitely define it as a ModelMultipleChoiceFilter.

However I faced another issue which is that source__in=10&source__in=7 roughly translates into Q(source__in=10) | Q(source__in=7). Which is raises an exception as 10 and 7 are not iterables. So I changed my code to use the exact lookup instead of in but still use the ModelMultipleChoiceFilter. Which, in the end, gives the following:

class BookingListFiltersForm(forms.Form):

    state__in = forms.MultipleChoiceField(
        choices=Booking.STATE_CHOICES, required=False,
        label=_("État"), widget=forms.CheckboxSelectMultiple)
    source = forms.ModelMultipleChoiceField(
        queryset=Platform.objects.all(), required=False,
        label=_("Source"), widget=ModelSelect2Multiple(
            url='autocomplete:platform'))


class BookingManagerFilter(filters.FilterSet):

    source = filters.ModelMultipleChoiceFilter(
        queryset=Platform.objects.all())
    payments__date = filters.DateFilter(method='payments__date_filter')
    payments__method = filters.ChoiceFilter(
        method='payments__method_filter',
        choices=BookingPayment.METHOD_CHOICES,
    )

    class Meta:
        model = Booking
        fields = {
            'period': [
                'endswith', 'endswith__gte', 'endswith__lte',
                'startswith', 'startswith__gte', 'startswith__lte',
            ],
            'state': ['in'],
            'source': ['exact'],
            'booking_date': ['date', 'date__lte', 'date__gte'],
            'accommodation': ['exact'],
            'guest': ['exact']
        }

    def get_form_class(self):
        return BookingListFiltersForm

Answer 2

I think documentation answers your question:

Filtering by an empty string

It’s not currently possible to filter by an empty string, since empty values are interpreted as a skipped filter.

GET http://localhost/api/my-model?myfield=

Further in the docs you have examples of possible solutions. Im putting here one of them

Solution 1: Magic values

You can override the filter() method of a filter class to specifically check for magic values. This is similar to the ChoiceFilter’s null value handling.

GET http://localhost/api/my-model?myfield=EMPTY

class MyCharFilter(filters.CharFilter):
    empty_value = 'EMPTY'

    def filter(self, qs, value):
        if value != self.empty_value:
            return super(MyCharFilter, self).filter(qs, value)

        qs = self.get_method(qs)(**{'%s__%s' % (self.name, self.lookup_expr): ""})
        return qs.distinct() if self.distinct else qs

---

Right now i feel there is not enough information to solve your problem. I left a comment under your question. If you can provide that extra information it would greatly help understand what is going on.

For here are some tips that can help you track this bug:

• Install ipdb. it will help you execute code step by step and inspect each variables.

• Drop breakpoint import ipdb;ipdb.set_trace() before line

  File "/home/tony/.venvs/cocoonr/lib/python3.6/site-packages/django/views/generic/list.py" in get
    142.         self.object_list = self.get_queryset()
  

I suspect you should find the culprit in https://github.com/carltongibson/django-filter/blob/82a47fb7bbddedf179f110723003f3b28682d7fe/django_filters/filterset.py#L215

You can do something like this

class BookingManagerFilter(filters.FilterSet):
    # your previous code here

    def filter_queryset(self, queryset):
        import ipdb;ipdb.set_trace()
        return super(BookingManagerFilter, self)filter_queryset(queryset):

And run your endpoint, ipdb will stop the app and you will be able to step into the code and inspect it.