Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python subprocess check_output much slower then call

I was trying to understand why this is happening. I'm calling a command to restart networking on Ubuntu server 12.04.

Fast execution

When I call the command using one of following three ways it takes around 0.1 seconds to execute:

  1. directly in terminal
  2. python script using os.system
  3. python script using subprocess.call

terminal session:

root@ubuntu:~# time /etc/init.d/networking restart
 * Running /etc/init.d/networking restart
 * Reconfiguring network interfaces...
real    0m0.105s

root@ubuntu:~# time python -c "import os;
> os.system('/etc/init.d/networking restart')"
 * Running /etc/init.d/networking restart
 * Reconfiguring network interfaces...
real    0m0.111s

root@ubuntu:~# time python -c "import subprocess;
> subprocess.call(['/etc/init.d/networking', 'restart'])"
 * Running /etc/init.d/networking restart
 * Reconfiguring network interfaces...
real    0m0.111s

Slow execution

However if I use subprocess.check_output or Popen and try and read the output it takes 23 seconds. Way slower. It seems this dramatic difference only happens when I try and use a function that will return the commands output. I would like to understand why this is happening and find a solution to execute this command and get it's output without it taking so long.

terminal session:

root@ubuntu:~# time python -c "import subprocess;
> print subprocess.check_output(['/etc/init.d/networking', 'restart'])"
 * Running /etc/init.d/networking restart
 * Reconfiguring network interfaces...
real    0m23.201s

root@ubuntu:~# time python -c "from subprocess import Popen, PIPE;
> print Popen(['/etc/init.d/networking', 'restart'], stdout=PIPE).stdout.read()"
 * Running /etc/init.d/networking restart
 * Reconfiguring network interfaces...
real    0m23.201s

Update

One of the comments suggested trying out the tee command. The results where very interesting. In the terminal without any involvement of python if tee is used it takes the same 23 seconds. I am still curious why but at least this might give more of a clue as to what's going on.

root@ubuntu:~# time /etc/init.d/networking restart | tee out.txt
 * Running /etc/init.d/networking restart
 * Reconfiguring network interfaces...
real    0m23.181s
like image 301
Marwan Alsabbagh Avatar asked Dec 12 '12 07:12

Marwan Alsabbagh


People also ask

Does subprocess call wait?

The subprocess module provides a function named call. This function allows you to call another program, wait for the command to complete and then return the return code.

What does subprocess Check_output do in Python?

The subprocess. check_output() is used to get the output of the calling program in python. It has 5 arguments; args, stdin, stderr, shell, universal_newlines. The args argument holds the commands that are to be passed as a string.

Does subprocess call raise exception?

The Python subprocess call() function returns the executed code of the program. If there is no program output, the function will return the code that it executed successfully. It may also raise a CalledProcessError exception.

Should I use shell true in subprocess?

We should avoid using 'shell=true' in subprocess call to avoid shell injection vulnerabilities. In this call you have to pass a string as a command to the shell. If call_method is user controlled then it can be used to execute any arbitrary command which can affect system.


1 Answers

The code below is based on the excellent comment J.F. Sebastian made. The code below runs in 0.1 seconds as expected and returns the output of the command to a string.

from subprocess import check_call, STDOUT
from tempfile import NamedTemporaryFile

with NamedTemporaryFile() as f:
    check_call(['/etc/init.d/networking', 'restart'], stdout=f, stderr=STDOUT)
    f.seek(0)
    output = f.read()
like image 99
Marwan Alsabbagh Avatar answered Oct 13 '22 12:10

Marwan Alsabbagh