Concurrent.futures > works well in command line, not when compiled with pyinstaller or py2exe

Question

I have a very simple script based on concurrent.futures that works well in command line (Python 2.7), but crashes when compiled with py2exe or Pyinstaller (the compiled program opens an increasing number of processes and eventually completely blocks windows if I don't kill them all first).

The code is very standard/simple so I struggle to understand the origin of this issue... Has anyone experienced this earlier? (I found discussions related to a similar issue with multiprocessing... but nothing that I could use to solve my problem)

# -*- coding: utf8 -*-
import os
import socket
import concurrent.futures

def simple_checkDomain(aDomain):
    print aDomain 
    # Do other stuff

def main():

    with concurrent.futures.ProcessPoolExecutor(max_workers=4) as executor:
        for domain in ["google.com","yahoo.com"]:
            job = executor.submit(simple_checkDomain, domain)

if __name__ == "__main__":
    main()

Best regards, S

Answer 1

I was having this issue with cx_freeze in Python 3.4. After a bit more google searches I landed on this bug report: http://bugs.python.org/issue21505

That sent me to the following docs and in my app, it seemed to fix the issue.

The suggested solution for Python 2.7, straight from the docs: multiprocessing.freeze_support

Not sure if this will fix it for the py2exe or Pyinstaller that you are using but figured I would post in case.

Answer 2

Complementing rdp's answer, which takes cues from the multiprocessing docs about freeze support:

What you need to do is add these lines to the very beginning of your app's execution:

from multiprocessing import freeze_support

if __name__ == '__main__':
    freeze_support()

And also from the docs:

One needs to call this function straight after the if __name__ == '__main__' line of the main module.

In my tests, I found that I needed to add those to the main app module and not the module that used concurrent.futures, which means that when the process forks, the very executable will be started again and the module will be run before execution resumes to the pool. This was specially important in my case where a PyQt app would try to start a new GUI.

As a side note, the bug report in the link from the other answer suggests that the RuntimeError might not have been raised, and that was my case.