How to filter haystack results with db query

Question

I need to text-search across my model and filter with db queries at the same time.

For example:

class MyModel(models.Model):
    text = models.TextField()
    users = models.ManyToMany(User)

class MyModelIndexIndex(indexes.SearchIndex, indexes.Indexable):
    text = indexes.CharField(document=True, model_attr='text')

    def get_model(self):
        return MyModel

So I want to filter all MyModel objects by user AND by some text via full-text search. Smth like these:

qs = MyModel.objects.filter(users=request.user)
sqs = MyModelIndex.objects.filter(text=request.GET['q'])
intersection = some_magic_function(qs, sqs)

or

intersection = some_other_magic_function(
    qs_kwargs={'users': request.user},
    sqs_kwargs={'text': request.GET['q']}
)

Of course desired db queries could be much more complicated.

I see some possible solutions, all with major flaws:

1. Make intersection in django: extract ids from qs and use them in sqs filter or vice versa. Problem: performance. We can workaround itby using pagination and do intersection only for given page and its predecessors. In this case we lose total count (

2. Index all m2m related fields. Problem: performance, duplicate functionality (I believe db will do such queries much better), db-features such as annotations etc.

3. Do not use haystack ( Go for mysql or posgresql built-in full-text search.

I believe I miss something obvious. Case seems to be quite common. Is there a conventional solution?

Answer 1

In the general case, it's (probably) not possible to solve your problem using just one query. For instance, if you are using ElasticSearch as a search backend engine and MySQL for django models, there is no way MySQL and ElasticSearch will communicate to produce a single, common query.

However, there should be a workaround if you are using a common SQL database for your Django models and your Haystack backend engine. You would have to create a custom haystack engine that would parse the query and filter the available models.

For example, to modify the behaviour of the SimpleSearchBackend, all you need to do is patch the search method:

class CustomSimpleSearchBackend(haystack.backends.SimpleSearchBackend):

    def search(self, query_string, **kwargs):
        ...
        if query_string:
            for model in models:
                ...
                if 'users' in kwargs:
                    qs = qs.filter(users=kwargs['users'])
                ...

class CustomSimpleEngine(haystack.backends.BaseEngine):
    backend = CustomSimpleSearchBackend
    query = haystack.backends.simple_backend.SimpleSearchQuery

And in settings.py:

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'myapp.backends.CustomSimpleEngine',
    },
}

Depending on which connection backend you use, the required patch will be different of course, but I suspect it should not be too hard to implement.