Running interactive command line code from Jupyter notebook

Question

There is an interesting option in Ipython Jupyter Notebook to execute command line statements directly from the notebook. For example:

! mkdir ... ! python file.py 

Moreover - this code can be run using os:

import os os.system('cmd command') 

but how do I run interactive shell commands. For example:

!conda install package 

may require future input ([Y]/N) or folder location, but won't accept further input.

Answer 1

the !commandsyntax is an alternative syntax of the %system magic, which documentation can be found here.

As you guessed, it's invoking os.system and as far as os.system works there is no simple way to know whether the process you will be running will need input from the user. Thus when using the notebook or any multi-process frontend you have no way to dynamically provide input to your the program you are running. (unlike a call to input in Python we can intercept).

As you explicitly show interest in installing packages from the notebook, I suggest reading the following from Jake Van Der Plas which is a summary of a recent discussion on the subject, and explain some of the complications of doing so. You can of course go with --yes option of conda, but it does not guarantee that installing with conda will always work.

Note also that !command is an IPython feature, not a Jupyter one.

Answer 2

Assuming you are asking about interactivity, there is something you can try.

If you ever wondered how Jupyter knows when the output of a cell ends: well, it apparently does not know, it just dumps any captured output into the most recently active cell:

import threading,time a=5 threading.Thread(target=lambda:[print(a),time.sleep(20),print(a)]).start() 

(deliberately shorter-than-nice example, as this is just side-info. While the 20-second wait is running you have time to activate another cell, perhaps via issuing an a=6)

This can be used to get the output of some console code to the screen, while controlling it from the main thread:

import sys,threading,subprocess  proc=subprocess.Popen('/bin/sh',stdout=subprocess.PIPE,stdin=subprocess.PIPE,stderr=subprocess.STDOUT) pout=proc.stdout pin=proc.stdin  def outLoop():     running=True     while(running):         line=pout.readline().decode(sys.stdout.encoding)         print(line,end='')         running='\n' in line     print('Finished')  threading.Thread(target=outLoop).start() 

Then you can isssue commands, like

pin.write(b'ls -l\n') pin.flush() 

and

pin.write(b'exit\n') pin.flush() 

Even b'ls\nexit\n' works, that is why outLoop is so long (a simple while(proc.poll() is None)-print(...) loop would finish sooner than it has grabbed all output.

Then the whole thing can be automated as:

while(proc.poll() is None):     inp=bytearray(input('something: ')+'\n',sys.stdin.encoding)     if(proc.poll() is None):         pin.write(inp)         pin.flush() 

This works well on https://try.jupyter.org/, but obviously I did not want to try installing conda packages there, so I do not know what happens when conda asks a question.

A lucky thing is that the input field stays at the bottom of the cell (tested with ls;sleep 10;ls). An unlucky thing is that the input field needs an extra entry at the end to disappear (and that is already the 'nice' way, when it was a simple while(...)-write(bytearray(input()))-flush() 3-liner, it was exiting with an exception.

If someone wants to try this on Windows, it works with 'cmd', but I suggest using a hardcoded 'windows-1252' instead of sys.stdin/out.encoding: they say UTF-8, but a simple dir command already produces output which is neither UTF-8 nor ASCII (the non-breakable space between the 3-digit groups in sizes is a 0xA0 character). Or just remove the decode part (and use running=0xA in line)