Menu
I am considering using Python to implement a program, which requires extensive multi-threading. Another requirement is that it will run on desktops, so having many processes, will make the application appear to be messy and harder to kill (in Task Manager). Therefore, I am considering using both the Threading and the Multiprocessing modules to reduce the number of processes. As far as I understand the GIL will apply only to a single process. My question is: Is there any reasons not to mix using the two modules?
420
If your program is IO-bound, both multithreading and multiprocessing in Python will work smoothly. However, If the code is CPU-bound and your machine has multiple cores, multiprocessing would be a better choice.
Both multithreading and multiprocessing allow Python code to run concurrently. Only multiprocessing will allow your code to be truly parallel. However, if your code is IO-heavy (like HTTP requests), then multithreading will still probably speed up your code.
Python doesn't support multi-threading because Python on the Cpython interpreter does not support true multi-core execution via multithreading. However, Python does have a threading library. The GIL does not prevent threading.
This includes queues in the multiprocessing.Queues are thread and process safe. This means that processes may get() and put() items from and to the queue concurrently without fear of a race condition. You can learn more about to how to use queues with multiple processes in the tutorial: Multiprocessing Queue in Python.
Note: This warning does not apply to windows.
Be careful! There is a nasty bug lurking with locks when combining threading and multiprocessing which is exposed when using the logging module as well. I've been bitten for the last week with child processes occasionally hanging. Now that I've disabled logging, so far so good (though not the best solution!).:
https://twiki.cern.ch/twiki/bin/view/Main/PythonLoggingThreadingMultiprocessingIntermixedStudy
http://bugs.python.org/issue6721
141
Other than basic principle of KISS.... go for it, there shouldn't be any issues.
36
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
