Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to read the first byte of a subprocess's stdout and then discard the rest in Python?

I'd like to read the first byte of a subprocess' stdout to know that it has started running. After that I'd like to discard all further output, so that I don't have to worry about the buffer.

What is the best way to do this?

Clarification: I'd like the subprocess to continue running alongside my program, I don't want to wait for it to terminate or anything like that. Ideally there would be some simple way to do this, without resorting to threading, forking or multiprocessing.

If I ignore the output stream, or .close() it, it causes errors if it is sent more data than it can fit in its buffer.

like image 931
Jeremy Avatar asked Mar 31 '11 03:03

Jeremy


2 Answers

If you're using Python 3.3+, you can use the DEVNULL special value for stdout and stderr to discard subprocess output.

from subprocess import Popen, DEVNULL

process = Popen(["mycmd", "myarg"], stdout=DEVNULL, stderr=DEVNULL)

Or if you're using Python 2.4+, you can simulate this with:

import os
from subprocess import Popen

DEVNULL = open(os.devnull, 'wb')
process = Popen(["mycmd", "myarg"], stdout=DEVNULL, stderr=DEVNULL)

However this doesn't give you the opportunity to read the first byte of stdout.

like image 61
David Foster Avatar answered Sep 24 '22 00:09

David Foster


This seems to work, but it doesn't feel idiomatic.

#!/usr/bin/env python3.1
import threading
import subprocess

def discard_stream_while_running(stream, process):
    while process.poll() is None:
        stream.read(1024)

def discard_subprocess_pipes(process, out=True, err=True, in_=True):
    if out and process.stdout is not None and not process.stdout.closed:
        t = threading.Thread(target=discard_stream_while_running, args=(process.stdout, process))
        t.start()

    if err and process.stderr is not None and not process.stderr.closed:
        u = threading.Thread(target=discard_stream_while_running, args=(process.stderr, process))
        u.start()

    if in_ and process.stdin is not None and not process.stdin.closed:
        process.stdin.close()

Example/test usage

if __name__ == "__main__":
    import tempfile
    import textwrap
    import time

    with tempfile.NamedTemporaryFile("w+t", prefix="example-", suffix=".py") as f:
        f.write(textwrap.dedent("""
            import sys
            import time

            sys.stderr.write("{} byte(s) read through stdin.\\n"
                             .format(len(sys.stdin.read())))

            # Push a couple of MB/s to stdout, messages to stderr.
            while True:
                sys.stdout.write("Hello Parent\\n" * 1000000)
                sys.stderr.write("Subprocess Writing Data\\n")
                time.sleep(0.5)
        """))
        f.flush()

        p = subprocess.Popen(["python3.1", f.name],
                             stdout=subprocess.PIPE,
                             stdin=subprocess.PIPE)

        p.stdin.write("Hello Child\n".encode())

        discard_subprocess_pipes(p) # <-- Here

        for s in range(16, 0, -1):
            print("Main Process Running For", s, "More Seconds")
            time.sleep(1)
like image 39
Jeremy Avatar answered Sep 23 '22 00:09

Jeremy