App Engine: Is time.sleep() counting towards my quotas?

Question

Hey. I'm working on an App Engine app that involves queries to the Google Maps API for geocoding. Google Maps doesn't like too much requests so I put a 1 second delay between each request with time.sleep(1).

I noticed that my quotas are running low in my GAE dashboard and decided to run a short test:

import cProfile
import time

def foo():
    time.sleep(3)

cProfile.run('foo()')

Which gave me the following output:

   4 function calls in 3.003 CPU seconds
   Ordered by: standard name

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000    3.003    3.003 <stdin>:1(foo)
        1    0.000    0.000    3.003    3.003 <string>:1(<module>)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}
        1    3.003    3.003    3.003    3.003 {time.sleep}

So it says that it's consuming 3 CPU seconds for a time.sleep(3). Now I'm wondering if calls like these are counted towards the quota limits that GAE provides. And if it does, what is the other way of making delays between API calls for geocoding?

Thanks.

Answer 1

You certainly don't want to be trying to sleep in a system that's designed completely from the ground up to finish requests in the absolute shortest time possible :D

What you could do instead, is create a task for each geocode, (check out the deferred library). You'd want to specify a queue for this task, then just set the rate limit on the queue to whatever you feel the maps geocoder might be comfortable with.

This way every geocode will run, and you'll never go faster than the rate limit you set, and you don't need to do any plumbing.

Answer 2

I am fairly certain that queue tasks also count towards your CPU usage in GAP. Regarding sleep(), i don't think there will be CPU "penalty" from that but I think it's a bad style.

Why sleep at all? In your task, do a single geocoding and simply post another invocation to yourself in the queue in 3secs. See the parameter countdown When invoking http://code.google.com/intl/el/appengine/docs/python/taskqueue/functions.html#add .

Answer 3

Your experiment proves that the time.sleep time counts against your quota. Have a look at the experimental Task Queue API. If your task isn't user initiated, you could also use Cron tasks, but I don't know if this will work well with so small intervals.

Answer 4

This Issue reports that the reporter has not been billed for cpu seconds incurred by time.sleep(), but that they show up on their appstats. It is very likely appstats uses cprofile as well. Sleep is important for people trying to make better asyncronous proxies which he could use for geocoding larger set of items.

http://code.google.com/p/googleappengine/issues/detail?id=3291