subprocess kills child processes but not the processes the child spawns

Question

I have been having an issue whereby I can kill the processes that spawns the nodes but the nodes do not get killed. Does anyone have any suggest how I can do this?

Some of my latest failed attempts to accomplish this are:

node.terminate()

and

node.send_signal(signal.SIGINT)

below is the code:

from subprocess import Popen
import json
import sys
import os
import signal
import requests

FNULL = open(os.devnull, 'w')

json_data = open('nodes.json', 'r').read()
data = json.loads(json_data)

port = data['port']

# launch hub
hub = Popen('java -jar selenium-server-standalone-2.37.0.jar -role hub -port %s' % port, stdout=FNULL, stderr=FNULL, shell=True)

#launch nodes
nodes = []
for node in data['nodes']:
    options = ''
    if node['name'] == 'CHROME':
        options += '-Dwebdriver.chrome.driver=../grid/chromedriver '
    #options += ' -browser browserName='+node['name']+' maxInstances='+str(node['maxInstances'])
    nodes.append(Popen('java -jar selenium-server-standalone-2.37.0.jar -role node -hub http://localhost:%i/grid/register %s' % (port, options), stdout=FNULL, stderr=FNULL, shell=True))

# wait for user input
print "type 'q' and ENTER to close the grid:"
while True:
    line = sys.stdin.readline()
    if line == 'q\n':
        break

# close nodes
for node in nodes:
    #node.terminate()
    node.send_signal(signal.SIGINT)

# close hub   
r = requests.get('http://localhost:'+str(port)+'/lifecycle-manager?action=shutdown')

As far as im aware, I'm basically forced to use shell=True, to get redirections to work Processing the child's stdout/stderr in the parent python process is not an option, since I couldn't find functionality for doing it in a non-waiting way (and the parent python process must do other things while the child is running)

# close nodes
for node in nodes:
    node.send_signal(signal.SIGINT)
    node.terminate()    

this seems to kill all the processes except for 1 of the nodes. Not always the same one

Answer 1

You could try using os.killpg. This function sends the signal to the process group, it should work if your processes do not change process group.

import os
import signal

os.killpg(os.getpgid(pid), signal.SIGINT)

Note, that process group will be changed if you are creating process under shell (bash, zsh, etc.), in that case more complicated technique should be used.