Python multiprocessing pool, join; not waiting to go on?

Question

(1) I'm trying to use pool.map followed by pool.join(), but python doesn't seem to be waiting for pool.map to finish before going on past the pool.join(). Here's a simple example of what I've tried:

from multiprocessing import Pool  foo = {1: []}  def f(x):     foo[1].append(x)     print foo  def main():     pool = Pool()     pool.map(f, range(100))     pool.close()     pool.join()     print foo  if __name__ == '__main__':     main() 

The printed output is just {1: []}, as if python just ignored the join command and ran print foo before it had a chance to run f. The intended result is that foo is {1:[0,1,...,99]}, and using the ordinary built-in python map gives this result. Why is the pooled version printing {1: []}, and how can I change my code to make it print the intended result?

(2) Ideally I'd also like to define foo as a local variable in main() and pass it to f, but doing this by making foo the first argument of f and using

pool.map(functools.partial(f, foo), range(100))

produces the same output. (and possibly also has the problem that each process now has its own copy of foo?) Though again, it works using the normal map instead.

Answer 1

This is not the correct way to use map.

1. Using a global variable that way is absolutely wrong. Processes do not share the same memory (normally) so every f will have his own copy of foo. To share a variable between different processes you should use a Manager
2. Function passed to map are, usually, expected to return a value.

I suggest you to read some documentation.

However here is a dummy example of how you could implement it:

from multiprocessing import Pool  foo = {1: []}  def f(x):     return x  def main():     pool = Pool()     foo[1] = pool.map(f, range(100))     pool.close()     pool.join()     print foo  if __name__ == '__main__':     main() 

You may also do something like pool.map(functools.partial(f, foo), range(100)) where foo is a Manager.