Lowering process priority of multiprocessing.Pool on Windows

Question

I use multiprocessing.Pool() to parallelize some heavy Pandas processing but find that it is a bit too successful. My CPU usage goes to 100% and my entire computer becomes very unresponsive. Even the mouse becomes difficult to use.

I can change the process priority of my process with this code.

import psutil
p = psutil.Process(os.getpid())
p.nice = psutil.BELOW_NORMAL_PRIORITY_CLASS

However, when I look in Windows Task Manager I find that only the main python.exe process has been changed to below normal priority.

Is there a good way to reduce the priority of the pool processes?

Answer 1

You can try setting priority of your process' children after you spawned them. Something like:

import psutil

# spawn children and/or launch process pool here

parent = psutil.Process()
parent.nice(psutil.BELOW_NORMAL_PRIORITY_CLASS)
for child in parent.children():
    child.nice(psutil.BELOW_NORMAL_PRIORITY_CLASS)