All example concurrent.futures code is failing with "BrokenProcessPool"

Question

I am trying to get a basic understanding of this before I create the actual application I need. I recently moved over from 2.7 to 3.3.

A direct copy-paste of this code from the python docs fails, as does a slightly simpler example from here.

This is my code, derived from the second example:

import concurrent.futures  nums = [1,2,3,4,5,6,7,8,9,10]  def f(x):     return x * x  # Make sure the map and function are working print([val for val in map(f, nums)])  # Test to make sure concurrent map is working with concurrent.futures.ProcessPoolExecutor() as executor:     for item in executor.map(f, nums):         print(item) 

And this is the output:

[1, 4, 9, 16, 25, 36, 49, 64, 81, 100] Traceback (most recent call last):   File "<string>", line 420, in run_nodebug   File "<module1>", line 13, in <module>   File "C:\Python33\lib\concurrent\futures\_base.py", line 546, in result_iterator     yield future.result()   File "C:\Python33\lib\concurrent\futures\_base.py", line 399, in result     return self.__get_result()   File "C:\Python33\lib\concurrent\futures\_base.py", line 351, in __get_result     raise self._exception concurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending. 

How can I get this code to work as expected? I was hoping the examples would just work out of the box.

Answer 1

This was my fault, for two reasons:

1. The code was un-guarded, i.e no if __name__
2. The strange looking Traceback was because the file was not saved. Never caused me an issue before, but did in this case.

Correcting both of those fixed the error.

Final test code:

import concurrent.futures  nums = [1,2,3,4,5,6,7,8,9,10]  def f(x):     return x * x def main():     # Make sure the map and function are working     print([val for val in map(f, nums)])      # Test to make sure concurrent map is working     with concurrent.futures.ProcessPoolExecutor() as executor:         print([val for val in executor.map(f, nums)])  if __name__ == '__main__':     main() 

Output, as expected:

[1, 4, 9, 16, 25, 36, 49, 64, 81, 100] [1, 4, 9, 16, 25, 36, 49, 64, 81, 100] 

Answer 2

Under Windows, it is important to protect the main loop of code to avoid recursive spawning of subprocesses when using processpoolexecutor or any other parallel code which spawns new processes.

Basically, all your code which creates new processes must be under if __name__ == '__main__': , for the same reason you cannot execute it in interpreter.