Using list comprehension instead of for loop when working with Django QuerySets

Question

I'm trying to tweak an application that is suffering in speed department. Because of that, I've started converting all the for-loop statements to list comprehensions when possible.

Currently, I'm working on a function that needs to iterate through a dictionary of Django querysets. The old code uses for-loop statement to iterate through this and it works fine. My code that uses list comprehension returns django querysets instead of my model object.

Here is the code:

def get_children(parent):
  # The following works
  children = []
  for value in get_data_map(parent).itervalues():
    children += list(value)
  # This part doesn't work as intended.
  booms = [value for value in get_data_map(parent).itervalues() if value]
  import pdb
  pdb.set_trace()


(Pdb) type(children[0])
<class 'site.myapp.models.Children'>

(Pdb) type(booms[0])
<class 'django.db.models.query.QuerySet'>

Note that get_data_map returns a dictionary whose values are <class 'django.db.models.query.QuerySet'>

This part of code is one of the most time consuming part of the application. If I get this working on list comprehensions, the application speed will hopefully get faster by two times.

Any idea how I can speed up this part of the code?

Answer 1

Your problem isn't for loops vs. list comprehensions (better would be generators). Your problem is way too many queries to the database.

Since you're trying to get one list, you should be trying to get it from one query. If you explained what was in your typical QuerySet, we could show you how best to merge them. Maybe use OR merges on Q objects. Or maybe building up a set of integers and feeding it to an __in= filter.