Menu
Related to Python Multiprocessing error: AttributeError: module '__main__' has no attribute '__spec__' , but arising from different circumstances.
I'm encountering an issue in Python 3.7.4 when I try to run multiprocessing code with pdb. The issue replicates with the basic multiprocessing example from https://docs.python.org/3.6/library/multiprocessing.html :
from multiprocessing import Pool
def f(x):
return x*x
if __name__ == '__main__':
with Pool(5) as p:
print(p.map(f, [1, 2, 3]))
This runs fine (outputs [1, 4, 9]) when run directly from Python via python.exe testcase.py. However, it does not work under pdb; python.exe -m pdb testcase.py fails with an error:
Traceback (most recent call last):
File "c:\python37\lib\pdb.py", line 1697, in main
pdb._runscript(mainpyfile)
File "c:\python37\lib\pdb.py", line 1566, in _runscript
self.run(statement)
File "c:\python37\lib\bdb.py", line 585, in run
exec(cmd, globals, locals)
File "<string>", line 1, in <module>
File "c:\users\max\desktop\projects\errortest.py", line 1, in <module>
from multiprocessing import Pool
File "c:\python37\lib\multiprocessing\context.py", line 119, in Pool
context=self.get_context())
File "c:\python37\lib\multiprocessing\pool.py", line 176, in __init__
self._repopulate_pool()
File "c:\python37\lib\multiprocessing\pool.py", line 241, in _repopulate_pool
w.start()
File "c:\python37\lib\multiprocessing\process.py", line 112, in start
self._popen = self._Popen(self)
File "c:\python37\lib\multiprocessing\context.py", line 322, in _Popen
return Popen(process_obj)
File "c:\python37\lib\multiprocessing\popen_spawn_win32.py", line 33, in __init__
prep_data = spawn.get_preparation_data(process_obj._name)
File "c:\python37\lib\multiprocessing\spawn.py", line 172, in get_preparation_data
main_mod_name = getattr(main_module.__spec__, "name", None)
AttributeError: module '__main__' has no attribute '__spec__'
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> c:\python37\lib\multiprocessing\spawn.py(172)get_preparation_data()
-> main_mod_name = getattr(main_module.__spec__, "name", None)
I hesitate to think that I've found a bug in a pair of modules that have been important parts of Python for over a decade. Is something incorrect here?
486
While using multiprocessing in Python, Pipes acts as the communication channel. Pipes are helpful when you want to initiate communication between multiple processes. They return two connection objects, one for each end of the Pipe, and use the send() & recv() methods to communicate.
Use the multiprocessing. Pool class when you need to execute tasks that may or may not take arguments and may or may not return a result once the tasks are complete. Use the multiprocessing. Pool class when you need to execute different types of ad hoc tasks, such as calling different target task functions.
multiprocessing has been distributed as part of the standard library since python 2.6. multiprocess is part of pathos, a python framework for heterogeneous computing.
Python provides real system-level processes via the multiprocessing. Process class in the multiprocessing module. The underlying operating system controls how new processes are created.
This is a limitation of multiprocessing in windows. This question contains a good explanation for why this is so. A quick google search shows that the puDB may be able to help with debugging multi-processing code, but I have not used it before.
The following is from the python docs:
Functionality within this package requires that the main module be importable by the children. This is covered in Programming guidelines however it is worth pointing out here. This means that some examples, such as the
multiprocessing.pool.Poolexamples will not work in the interactive interpreter.
185
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
