Acquire Thread.Lock takes 50% of response time in Google App Engine running Python 2.7

Question

I'm using python27 on GAE and am looking to reduce the response time of an app.

According to cProfile, it's taking 40s (obviously too long) to run 1.6M function calls (seems very high). The only clue I can find right now is that "{method 'acquire' of 'thread.lock' objects}" takes 20 of the 40s. (Note: setting threadsafe set to false or true in app.yaml does not seem to have much of an effect.)

Any thoughts on where I'm going wrong or where to investigate next? I'm aware that the large time taken by aquiring locks may just be symptom and not the cause, but if that's the case, how can I find the underlying cause? All the other times and ncalls listed for my functions in cProfile seem reasonable.

I also wonder if it's due to performance issues GAE had with 2.7 at the end of 2011:

• https://groups.google.com/forum/?fromgroups#!topic/google-appengine/jMah6aWfZWg
• https://code.google.com/p/googleappengine/issues/detail?id=6323
• Appengine, performance degradation with python27

For reference, here is an example cProfile output (with lines removed):

Profile data:
1662549 function calls (1652247 primitive calls) in 39.545 seconds
Ordered by: cumulative time
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
[lines removed]
10816   19.245    0.002   19.245    0.002 {method 'acquire' of 'thread.lock' objects}
[lines removed]

Thanks in advance for any help you can provide!

Answer 1

If your application is very RPC heavy (making long calls to datastore/urlfetch/etc) then you will notice a lot of time spent waiting in thread.lock.acquire().

You may want to enable AppStats in your application, and look to see how long each rpc is taking.