A non-blocking read on a subprocess.PIPE in Python

Question

I'm using the subprocess module to start a subprocess and connect to its output stream (standard output). I want to be able to execute non-blocking reads on its standard output. Is there a way to make .readline non-blocking or to check if there is data on the stream before I invoke .readline? I'd like this to be portable or at least work under Windows and Linux.

Here is how I do it for now (it's blocking on the .readline if no data is available):

p = subprocess.Popen('myprogram.exe', stdout = subprocess.PIPE) output_str = p.stdout.readline() 

Answer 1

fcntl, select, asyncproc won't help in this case.

A reliable way to read a stream without blocking regardless of operating system is to use Queue.get_nowait():

import sys from subprocess import PIPE, Popen from threading  import Thread  try:     from queue import Queue, Empty except ImportError:     from Queue import Queue, Empty  # python 2.x  ON_POSIX = 'posix' in sys.builtin_module_names  def enqueue_output(out, queue):     for line in iter(out.readline, b''):         queue.put(line)     out.close()  p = Popen(['myprogram.exe'], stdout=PIPE, bufsize=1, close_fds=ON_POSIX) q = Queue() t = Thread(target=enqueue_output, args=(p.stdout, q)) t.daemon = True # thread dies with the program t.start()  # ... do other things here  # read line without blocking try:  line = q.get_nowait() # or q.get(timeout=.1) except Empty:     print('no output yet') else: # got line     # ... do something with line 

Answer 2

I have often had a similar problem; Python programs I write frequently need to have the ability to execute some primary functionality while simultaneously accepting user input from the command line (stdin). Simply putting the user input handling functionality in another thread doesn't solve the problem because readline() blocks and has no timeout. If the primary functionality is complete and there is no longer any need to wait for further user input I typically want my program to exit, but it can't because readline() is still blocking in the other thread waiting for a line. A solution I have found to this problem is to make stdin a non-blocking file using the fcntl module:

import fcntl import os import sys  # make stdin a non-blocking file fd = sys.stdin.fileno() fl = fcntl.fcntl(fd, fcntl.F_GETFL) fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)  # user input handling thread while mainThreadIsRunning:       try: input = sys.stdin.readline()       except: continue       handleInput(input) 

In my opinion this is a bit cleaner than using the select or signal modules to solve this problem but then again it only works on UNIX...