Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Callback with multiprocessing.pools.map_async not working?

Tags:

python

I can't seem to get my callbacks to work when using map_async(). It works when I use slightly modified code to loop over my array adding tasks via apply_async() instead. From the documentation it seems I should be able to use the callbacks with map_async(), but maybe it's some kind of newb mistake...

from multiprocessing import Pool,TimeoutError
from time import sleep

servers=["s1","s2","s3","s4","s5","s6"]

def f(x):
    print("start f(" + x + ")")
    sleep(5)
    print("end   f(" + x + ")")
    return "did " + x

def mycallback(x):
    print("My callback " + str(x))

def myerrorcallback(r):
    print("My errorcallback " + str(r))

if __name__ == '__main__':
    pool = Pool(processes=4)
    results = pool.map_async(f, servers,  chunksize=1, callback=mycallback, error_callback=myerrorcallback)
    print(results.get(timeout=11))

When run I get:

D:\python> f.py
start f(s1)
start f(s2)
start f(s3)
start f(s4)
end   f(s1)
start f(s5)
end   f(s2)
start f(s6)
end   f(s4)
end   f(s3)
end   f(s5)
end   f(s6)
['did s1', 'did s2', 'did s3', 'did s4', 'did s5', 'did s6']

When I use modified code with apply_async() instead I get the print output from the callbacks. Modified code is to just change the last part to:

if __name__ == '__main__':
    pool = Pool(processes=4)
    for server in servers:
        pool.apply_async(f, (server,),  callback=mycallback, error_callback=myerrorcallback)
    pool.close()
    pool.join()

Results in:

D:\python\>fb.py
start f(s1)
start f(s2)
start f(s3)
start f(s4)
end   f(s1)
start f(s5)
My callback did s1
end   f(s2)
My callback did s2
start f(s6)
end   f(s3)
My callback did s3
end   f(s4)
My callback did s4
end   f(s5)
My callback did s5
end   f(s6)
My callback did s6
like image 782
Bbb Avatar asked Oct 24 '12 06:10

Bbb


1 Answers

Ok I took a chance and logged a bug for it. Turns out it is actually a bug in 3.3 and a patch is in progress.

http://bugs.python.org/issue16307

like image 111
Bbb Avatar answered Sep 18 '22 19:09

Bbb