How to improve the code performance if the most time-consuming part comes to posix.waitpid?

Question

I use cprofile to profile a python program. The most time-consuming part turns out to be posix.waitpid. What can I do to improve the code performance? See the screenshot below for the one line of my cprofile result

Answer 1

subprocess.call spends time in posix.waitpid

When you use subprocess.call(other_process), that starts the other process and waits for it to finish. If the other process does a lot of work, that takes a lot of time.

That is what your profiler is showing: most of the time, you are waiting for the other process to finish.

Quite possibly, the other process actually has a good reason to take that much time, but profiler can not tell you anything about that, because it is profiling this process, not the other process.

import instead of call?

You wrote in comment that you could import the other script and run it's main function. That would certainly take away the overhead of creating a new process, but probably most of that time was in fact in the main function of the other process.

Well, at least with import, everything would be running in the same process and profiler would be able to give more useful information.

On the other hand, you wanted a subprocess?

It is not clear why you created a subprocess in the first place, so I'll guess you wanted parallel execution of its code.

Well, you can start a subprocess and not wait for it to finish:

Instead of:

subprocess.call(other_process)

Do this:

process = subprocess.Popen(other_process)

# the code which follows is executed in parallel with the other process

Now the subprocess is started and you can use the process variable to see when it finishes.

Other options: threading, multiprocessing

If you want to run in parallel some code which you can import, then it is quite possible that threading and multiprocessing modules are better options than subprocess.