Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Performance impact of or-ing Q objects in django query

I am performing a query that is or-ing a bunch Q's together and it seems to be taking a lot of time. Here is some psuedo code

query_params = []
for i in range(80): #there are about 80ish Q objects being created
    query_params.append(Q(filter_stuff))

Then I or them all together

query_params = reduce(or_, query_params)

And when I execute the query

query = list(MyModel.objects.filter(query_params))

It hangs for a LONG time. I know this is a pretty general question and it's hard to given a diagnostic without an intimate understanding of the data structure (which would be difficult to give here). But I'm just curious if there is an inherent performance impact of or-ing Q objects in a django query

like image 496
sedavidw Avatar asked Sep 29 '22 01:09

sedavidw


1 Answers

Was able to shrink down the length of the query significantly by reducing the number of of Qobjects. They were all of the format like:

q1 = Q(field1=field1_val1, field2=field2_val1)
q2 = Q(field1=field1_val2, field2=field2_val2)
#...etc

I ended up grouping them by field1 values:

q_dict = {field1_val1: [all field_2 values paired with field1_val1], ...}

Then my q objects looked like this:

for field1_val, field2_vals = q_dict.items():
    query_params.append(Q(field1=field1_val, field2__in=field2_vals))

This ultimately shrank down my number of Q objects significantly and the query ran much faster

like image 81
sedavidw Avatar answered Oct 02 '22 15:10

sedavidw