google app engine -- Asynchronous blocks of code? python

Question

So I have some code that functions something like this generic code:

for blah in blahs:
   blah.take_forever()

for blah2 in blah2s:
  vars = blah2.take_forever()
  for var in vars:
     var.also_take_forever()

I want something that functions like async, such as

async_start_blah2_loop_then_do_someting_else()

do_the_first_blah_loop()

gather_results_and_send_them_out()

However, I'm not using the datastore or a urlfetch for this, so what other options will speed up this process?

"

map_async(callback, pass_batch_into_callback=None, merge_future=None, **q_options)
Asynchronously map a callback function or tasklet over the query results. This is the asynchronous version of map().

"

seems to only apply to datastore queries.

Suggestions?

Answer 1

You can use Python's native threading module in App Engine to execute functions asynchronously (see threading.Thread). This also works fine on automatic scaled instances as it uses 'green' threads instead of OS threads.

def blahFunc():
  for blah in blahs:
    blah.take_forever()

def blah2Func():
  for blah2 in blah2s:
    vars = blah2.take_forever()
    for var in vars:
      var.also_take_forever()

# Execute both loops 'at the same time'
background = threading.Thread(target=blah2Func)
background.start() 
blah1Func()

Note that you won't necessarily gain any speed from this (I assume yours was a contrived example for simplicity) as the request still executes inside the same 'real' OS thread, but it is useful if you need to avoid blocking on a long operation.

If you have really long running tasks that may need to run longer than the request a better solution is to use Task Queues.