Django Threading and Tests

Question

I have a multithreaded django app which creates some objects in separate threads in order to return faster. The created objects are just used to track what the user has done and are not in any way time sensitive.

The view function used to look something like this:

def foo(request):
    #... do important computation...
    bar(x, y, z)
    return HttpResponse()

Everything worked fine here, but when I change it to look like this and use threading:

def foo(request):
    #... do important computation...
    thread = Thread(target=bar, args=(x, y, z))
    thread.start()
    if testing_mode:
        thread.join()
    return HttpResponse()

The second version fails. This is all being done using TransactionTestCase and mySQL.

Any ideas?

Answer 1

Using threads for offloading request is not so good idea. There are many traps, and few benefits. Main problems (and your question relates to these) are:

• every thread in Django uses separate DB connections so:
  • you loose benefits of transactions
  • you have to close connections manually in threads
  • if you don't close connections in threads in right way (and it is hard to do right) you will have hundreds of open connection to your DB and this will bring problems on you
  • you have problem with testing, because testing framework does some tricks on DB connection, and it can't do it on connection from thread
• translation framework does not work in threads
• you thread can be prematurely killed if wsgi server decides to reload, but there is no request to handle
• Django error handling does not work for threads

Correct ways to do that would be:

• Optimize your code to serve requests faster or
• use task systems like Celery or RQ to offload your work into background (this have some of the problems above, but is more straightforward).

PS. Don't try to setup Celery or RQ for tests. You should just mock the task and test it separately.