Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Auto kill process and child process of multiprocessing Pool

I am using multiprocessing module for parallel processing. Bellow code snippet search the string filename in X location and return the file name where the string found. But in some cases it take long time to search process so i was trying to kill the search process with take more than 300 seconds.For that i used timeout == 300 as given bellow , this kills the search process but it dosent kill the child process spawn by bellow code.

I tried to find multiple way but no success :/

How can i kill parent process from Pool along with its child process ?

import os
from multiprocessing import Pool

def runCmd(cmd):
     lresult = os.popen(cmd).read()
     return lresult

main ():
     p = Pool(4)
     data_paths = [list of paths of store data]
     search_cmds = [ "SearchText.exe %s < %s"%(data_path, filename) for data_path in data_paths ]
     results = [p.apply_async(runCmd, (cmd,), callback = log_result) for cmd in search_cmds]
     try:
        for result in results:
            root.append(result.get(timeout=300))
        #rool holds the result of search process
     except TimeoutError:
        for c in multiprocessing.active_children():
            print '----->',c.pid
            os.kill(c.pid, signal.SIGTERM)
     p.close()
     p.join()

if __name__ == '__main__':
    main()

Process Tree in Process Explorer :

cmd.exe
------python.exe
----------------python.exe
--------------------------cmd.exe
---------------------------------SearchText.exe
----------------python.exe
--------------------------cmd.exe
---------------------------------SearchText.exe
----------------python.exe
--------------------------cmd.exe
---------------------------------SearchText.exe
----------------python.exe
--------------------------cmd.exe
---------------------------------SearchText.exe

above code snippet dosnt kill the child process

--------------------------cmd.exe
---------------------------------SearchText.exe
--------------------------cmd.exe
---------------------------------SearchText.exe
--------------------------cmd.exe
---------------------------------SearchText.exe
--------------------------cmd.exe
---------------------------------SearchText.exe

Theses child search process retain , these child process also get killed .

Please guild.

Thanks

like image 675
Shashi Avatar asked Apr 01 '14 11:04

Shashi


People also ask

How do I use pool in multiprocessing in Python?

Pool An easy way to use multiprocessing is to use the Pool object to create child processes. from multiprocessing import Pool import os def f(x): print('Child process id:', os.getpid()) return x*2 if __name__ == '__main__': print('Parent process id:', os.getpid()) with Pool(5) as p: print(p.map(f, [1, 2, 3]))

How does the child process replace itself with a different program?

The child process replaces itself with a different program using the execve () system call (or one of its variants, e.g. execl () ). The thing is, there’s nothing preventing you from just doing fork () .

What happens if a process is not provided by the pool?

If not provided any, the processes will exist as long as the pool does. Consider the following example that calculates the square of the number and sleeps for 1 second. Here, we import the Pool class from the multiprocessing module. In the main function, we create an object of the Pool class.

Why is my multiprocessing pool full of sharks?

Why your multiprocessing Pool is stuck (it’s full of sharks!) You’re using multiprocessing to run some code across multiple processes, and it just—sits there. It’s stuck. You check CPU usage—nothing happening, it’s not doing any work.


1 Answers

I am able to solve my Issue using psutil module

Found solution on bellow post:

import psutil, os

def kill_proc_tree(pid, including_parent=True):    
    parent = psutil.Process(pid)
    for child in parent.get_children(recursive=True):
        child.kill()
    if including_parent:
        parent.kill()

me = os.getpid()
kill_proc_tree(me)

https://stackoverflow.com/a/4229404/420557

like image 143
Shashi Avatar answered Nov 05 '22 08:11

Shashi