Dynamic output from python subprocess module

Question

how can i achieve output dynamically using subprocess module (while the external program keeps running) in python. The external program from which i want to get output dynamically is ngrok , ngrok keep running as long as my program is running but i need output while the process is running so that i can extract the newly generated "forwarding url"

when i try to do :

cmd = ['ngrok', 'http', '5000']
output = subprocess.Popen(cmd, stdout=subprocess.PIPE, buffersize=1)

it keep storing output in buffers

Answer 1

I know this is a duplicate, but I can't find any relevant threads about this now. All i get is output.communicate().

So here's a snippet that might be useful:

import subprocess
cmd = ['ngrok', 'http', '5000']
process = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

while process.poll() is None:
    print(process.stdout.readline())
print(process.stdout.read())
process.stdout.close()

This would output anything the process outputs, through your script into your output. It does so by looking for a newline character before outputting.

This piece of code would work, if it weren't for the fact that ngrok uses ncurses and/or hogs the output to it's own user/thread much like when SSH asks for a password when you do ssh user@host.

process.poll() checks if the process has an exit-code (if it's dead), if not, it continues to loop and print anything from the process's stdout.

There's other (better) ways to go about it, but this is the bare minimum I can give you without it being complicated really fast.

For instance, process.stdout.read() could be used in junction with select.select() to achieve better buffered output where new-lines are scares. Because if a \n never comes, the above example might hang your entire application.

There's a lot of buffer-traps here that you need to be aware of before you do manual things like this. Otherwise, use process.communicate() instead.

Edit: To get around the hogging/limitation of I/O used by ngrok, you could use pty.fork() and read the childs stdout via the os.read module:

#!/usr/bin/python

## Requires: Linux
## Does not require: Pexpect

import pty, os
from os import fork, waitpid, execv, read, write, kill

def pid_exists(pid):
    """Check whether pid exists in the current process table."""
    if pid < 0:
        return False
    try:
        kill(pid, 0)
    except (OSError, e):
        return e.errno == errno.EPERMRM
    else:
        return True

class exec():
    def __init__(self):
        self.run()

    def run(self):
        command = [
                '/usr/bin/ngrok',
                'http',
                '5000'
        ]

        # PID = 0 for child, and the PID of the child for the parent    
        pid, child_fd = pty.fork()

        if not pid: # Child process
            # Replace child process with our SSH process
            execv(command[0], command)

        while True:
            output = read(child_fd, 1024)
            print(output.decode('UTF-8'))
            lower = output.lower()

            # example input (if needed)
            if b'password:' in lower:
                write(child_fd, b'some response\n')
        waitpid(pid, 0)

exec()

There's still a problem here, and I'm not quite sure what or why that is.
I'm guessing the process is waiting for a signal/flush some how.
The problem is that it's only printing the first "setup data" of ncurses, meaning it wips the screen and sets up the background-color.

But this would give you the output of the process at least. replacing print(output.decode('UTF-8')) would show you what that output is.