Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Subprocess timeout failure

I want to use a timeout on a subprocess

 from subprocess32 import check_output
 output = check_output("sleep 30", shell=True, timeout=1)

Unfortunately, whilst this raises a timeout error, it does so after 30 seconds. It seems that check_output cannot interrupt the shell command.

What can I do on on the Python side to stop this? I suspect that subprocess32 fails to kill the timed out process.

like image 988
innisfree Avatar asked Apr 30 '16 07:04

innisfree


People also ask

Does Popen have a timeout?

The timeout argument is passed to Popen.communicate() . If the timeout expires, the child process will be killed and waited for. The TimeoutExpired exception will be re-raised after the child process has terminated.

Does subprocess call wait?

The subprocess module provides a function named call. This function allows you to call another program, wait for the command to complete and then return the return code.

What is Popen in subprocess?

The subprocess module defines one class, Popen and a few wrapper functions that use that class. The constructor for Popen takes arguments to set up the new process so the parent can communicate with it via pipes. It provides all of the functionality of the other modules and functions it replaces, and more.

Does subprocess Popen block?

Popen is nonblocking. call and check_call are blocking. You can make the Popen instance block by calling its wait or communicate method.


1 Answers

check_output() with timeout is essentially:

with Popen(*popenargs, stdout=PIPE, **kwargs) as process:
    try:
        output, unused_err = process.communicate(inputdata, timeout=timeout)
    except TimeoutExpired:
        process.kill()
        output, unused_err = process.communicate()
        raise TimeoutExpired(process.args, timeout, output=output)

There are two issues:

  • [the second] .communicate() may wait for descendant processes, not just for the immediate child, see Python subprocess .check_call vs .check_output
  • process.kill() might not kill the whole process tree, see How to terminate a python subprocess launched with shell=True

It leads to the behaviour that you observed: the TimeoutExpired happens in a second, the shell is killed, but check_output() returns only in 30 seconds after the grandchild sleep process exits.

To workaround the issues, kill the whole process tree (all subprocesses that belong to the same group):

#!/usr/bin/env python3
import os
import signal
from subprocess import Popen, PIPE, TimeoutExpired
from time import monotonic as timer

start = timer()
with Popen('sleep 30', shell=True, stdout=PIPE, preexec_fn=os.setsid) as process:
    try:
        output = process.communicate(timeout=1)[0]
    except TimeoutExpired:
        os.killpg(process.pid, signal.SIGINT) # send signal to the process group
        output = process.communicate()[0]
print('Elapsed seconds: {:.2f}'.format(timer() - start))

Output

Elapsed seconds: 1.00
like image 167
jfs Avatar answered Nov 16 '22 00:11

jfs