Live stdout output from Python subprocess in Jupyter notebook

Question

I'm using subprocess to run a command line program from a Python (3.5.2) script, which I am running in a Jupyter notebook. The subprocess takes a long time to run and so I would like its stdout to be printed live to the screen in the Jupyter notebook.

I can do this no problem in a normal Python script run from the terminal. I do this using:

def run_command(cmd): from subprocess import Popen, PIPE import shlex  with Popen(shlex.split(cmd), stdout=PIPE, bufsize=1, universal_newlines=True) as p:     for line in p.stdout:         print(line, end='')     exit_code = p.poll() return exit_code 

However, when I run the script in a Jupyter notebook, it does not print the stdout live to the screen. Instead, it prints everything after the subprocess has finished running.

Does anyone have any ideas on how to remedy this?

Many thanks, Johnny

Answer 1

The ipython notebook has it's own support for running shell commands. If you don't need to capture with subprocess stuff you can just do

cmd = 'ls -l' !{cmd} 

Output from commands executed with ! is automatically piped through the notebook.

Answer 2

If you set stdout = None (this is the default, so you can omit the stdout argument altogether), then your process should write its output to the terminal running your IPython notebook server.

This happens because the default behavior is for subprocess to inherit from the parent file handlers (see docs).

Your code would look like this:

from subprocess import Popen, PIPE import shlex  def run_command(cmd):     p = Popen(shlex.split(cmd), bufsize=1, universal_newlines=True)     return p.poll() 

This won't print to the notebook in browser, but at least you will be able to see the output from your subprocess asynchronously while other code is running.

Hope this helps.