Menu
Suppose I have a python script called my_parallel_script.py that involves using multiprocessing to parallelize several things and I run it with the following command:
python -m cProfile my_parallel_script.py
This generates profiling output for the parent process only. Calls made in child processes are not recorded at all. Is it possible to profile the child processes as well?
If the only option is to modify the source, what would be the simplest way to do this?
694
Profiling is a technique to figure out how time is spent in a program. With these statistics, we can find the “hot spot” of a program and think about ways of improvement. Sometimes, a hot spot in an unexpected location may hint at a bug in the program as well.
There are several common ways to parallelize Python code. You can launch several application instances or a script to perform jobs in parallel. This approach is great when you don't need to exchange data between parallel jobs.
The general way to parallelize any operation is to take a particular function that should be run multiple times and make it run parallelly in different processors. To do this, you initialize a Pool with n number of processors and pass the function you want to parallelize to one of Pool s parallization methods.
If we want to execute the function one at a time, we can use the submit() method. It schedules the target function for execution and returns a futures object. This futures object encapsulates the function's execution and allows us to check that it's running or if it's done and fetch the return value using result() .
cProfile only works within a single process, so you will not automatically get the child process profiled.
I would recommend that you tweak the child process code so that you can invoke it separately as a single process. Then run it under the profiler. You probably don't need to run your system multi-process while profiling, and it will simplify the job to have only a single child running.
124
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
