Python pty.spawn stdin not echoed but redirected to master's stdout

Question

I want to call a program from Python and make it believe that its stdout is a tty even when Python's process stdout is attached to a pipe. So I used the pty.spawn function to achieve that, which can be verified from the following :

$ python -c "import sys; from subprocess import call; call(sys.argv[1:])" python -c "import sys; print sys.stdout.isatty()" | cat
False

$ python -c "import sys; import pty; pty.spawn(sys.argv[1:])" python -c "import sys; print sys.stdout.isatty()" | cat
True

(We see that in the second command we have achieved our goal, i.e. the spawned process is tricked into thinking that its stdout is a tty.)

But the problem is that if we use pty.spawn then its input is not echoed, rather it is being redirected to the master's stdout. This can be seen by the following command :

$ python -c "import sys; import pty; pty.spawn(sys.argv[1:])" cat > out.txt
$ # Typed "hello" in input, but that is not echoed (use ^D to exit). It is redirected to output.txt
$ cat out.txt
hello
hello

(But this problem does not exists when we use subprocess.call

$ python -c "import sys; from subprocess import call; call(sys.argv[1:])" cat > out1.txt
hello
$ cat out1.txt
hello

since its stdin and stdout are correctly attached to the master.)

I could not find a way so that a program is called by Python, where it sees its stdout as a tty (similar to pty.spawn) but its input is echoed correctly (similar to subprocess.call). Any ideas?

Answer 1

You are creating a terminal with stdout connected to a file so the normal echo-back that terminals do is being sent to the file rather than screen.

Im not sure that spawn is intended to be used directly like this: the pty library offers pty.fork() to create a child process and returns a file descriptor for the stdin/stdout. But you'll need a lot more code to use this.

To overcome the current problem you are having with spawn, heres two easy options:

Option 1: If all you care about is sending the output of the spawned command to a file, then you can do (i prefer named pipes and here files for python one-liners):

$ python <(cat << EOF
import sys
import pty
print 'start to stdout only'
pty.spawn(sys.argv[1:])
print 'complete to stdout only'
EOF
) bash -c 'cat > out.txt'

which will look like this when run:

start to stdout only
hello
complete to stdout only

that shows that the input (I typed hello) and the result of print statements are going to the screen. The contents of out.txt will be:

$ cat out.txt
hello

That is, only what you typed.

Option 2: If on the other hand you want the out file to contain the python output around the spawned command output, then you need something a bit more complicated, like:

python <(cat << EOF
import sys
import pty
import os
old_stdout = sys.stdout
sys.stdout = myfdout = os.fdopen(4,"w")
print 'start to out file only'
myfdout.flush()
pty.spawn(sys.argv[1:])
print 'complete to out file only'
sys.stdout = old_stdout
EOF
) bash -c 'cat >&4' 4>out.txt

which will only have this output to the terminal when run (ie whatever you type):

hello

but the out file will contain:

$ cat out.txt
start to out file only
hello
complete to out file only

Background: python pty library is powerful: its creating a terminal device attached to the python processes stdout and stdin. Id imagine most uses of this will use the pty.fork() call so that real stdin/stdout are not affected.

However in your case, at your shell, you redirected the stdout of the python process to a file. The resulting pty also therefore had its stdout attached to the file so the normal action of echoing stdin back to stdout was being redirected. The regular stdout (screen) was still in place but not being used by the new pty.

The key difference for Option 1 above is to move the redirection of stdout to occur somewhere inside the pty.spawn call, so that the pty created still has a clear connection to the actual terminal stdout (for when it tries to echo stdin as you type)

The difference for Option 2 is to create a second channel on an arbitrary file descriptor (ie file descriptor 4) and use this in place of stdout, once you are inside python and when you create your spawned process (ie redirect the stdout of your spawned process to the same file descriptor)

Both of these difference prevent the pty that pty.spawn creates from having its stdout changed or disconnected from the real terminal. This allows the echo-back of stdin to work properly.

There are packages that use the pty library and give you more control but you'll find most of these use pty.fork() (and interestingly I havent found one yet that actually uses pty.spawn)

EDIT Heres an example of using pty.fork():

import sys
import pty
import os
import select
import time
import tty
import termios

print 'start'
try:
    pid, fd = pty.fork()
    print 'forked'
except OSError as e:
    print e

if pid == pty.CHILD:
    cmd = sys.argv[1]
    args = sys.argv[1:]
    print cmd, args
    os.execvp(cmd,args)
else:
    tty.setraw(fd, termios.TCSANOW)
    try:
        child_file = os.fdopen(fd,'rw')
        read_list = [sys.stdin, child_file]
        while read_list:
            ready = select.select(read_list, [], [], 0.1)[0]
            if not ready and len(read_list) < 2:
                break
            elif not ready:
                time.sleep(1)
            else:
                for file in ready:
                    try:
                        line = file.readline()
                    except IOError as e:
                        print "Ignoring: ", e
                        line = None
                    if not line:
                        read_list.remove(file)
                    else:
                        if file == sys.stdin:
                            os.write(fd,line)
                        else:
                            print "from child:", line
    except KeyboardInterrupt:
        pass

EDIT This question has some good links for pty.fork()

UPDATE: should have put some comments in the code How the pty.fork() example works:

When the interpretor executes the call to pty.fork(), the processesing splits into two: there are now two threads that both appear to have just executed the pty.fork() call.

One thread is the thread you were originally in (the parent) and one is a new thread (the child).

In the parent, the pid and fd are set to the process id of the child and a file decriptor connnected to teh child's stdin and stdout: in the parent, when you read from fd you are reading what has been written to the childs stdout; when you write to fd you are writing to the childs stdin. So now, in the parent we have a way of communicating with the other thread over its stdout/stdin.

In the child, the pid is set to 0 and the fd is not set. If we want to talk to the parent thread, we can read and write over stdin/stdout knowing that the parent can and should do something with this.

The two threads are going to execute the same code from this point on, but we can tell if we are in the parent or the child thread based on the value in pid. If we want to do different things in the child and parent threads then we just need a conditional statement that sends the child down one code path and the parent down a different code path. Thats what this line does:

if pid == pty.CHILD:
  #child thread will execute this code
  ....
else
  #parent thread will execute this code
  ...

In the child, we simply want to spawn the new command in a new pty. The os.execvp is used becuase we will have more control over the pty as a terminal with this method but essentially its the same as pty.spawn()'. This means the child stdin/stdout are now connected to the command you wanted via a pty. IMmportantly, any input or output from the command (or the pty for that matter) will be available to the parent thread by reading fromfd. And the parent can write to the command via pty by writing tofd`

So now, in the parent, we need to connect the real stdin/stdout to the child stdin/stdout via reading and writing to fd. Thats what the parent code now does (the else part). Any data that turns up on the real stdin is written out to fd. Any data read from fd (by the parent) is written to stdout. So the only thing the parent thread is now doing is proxying between the real stdin/stdout and fd. If you wanted to do something with the input and output of you command programatically, this is where you would do it.

The only other thing that happens in the parent is this call:

tty.setraw(fd, termios.TCSANOW)

This is one way to tell the pty in the child to stop doing echo-back.

This solves the problem you were originally having: - your local terminal is connected only to the parent thread - normal echo-back is in place (ie before your input is passed into the process) - stdout of the process can be redirected - whatever you do with your terminal stdout has no impact on the stdin/stdout of the child process - the child process has been told to not do local echo-back of its stdin

That seems like a lot of explanation - if anyone has any edits for clarity?