Reading stdin line at a time in Python

Question

I am trying to test a particular chat program by sending a series of commands and capturing the output.

How can I pass:

sleep 2
echo test
sleep 2
echo test1

I have tried this:

(sleep 2; echo test; sleep 2; echo test1) | python3 test.py

but it just prints the first part while for the second I get nothing. It enters into an endless loop instead.

the python program's code is:

import sys, select

while True:
  socket_list = [sys.stdin]
  read_sockets, write_sockets, error_sockets = select.select(socket_list, [], [])
  for sock in read_sockets:
    message = sys.stdin.readline()
    sys.stdout.write("> %s: ")
    sys.stdout.flush()

I should mentioned that this is not the complete program but it is the part where it helps recreate the exact same effect.

Answer 1

Your example goes to loop because after echo test1 stdin is closed and read() always returns empty. Also you shouldn't use another blocking call with select. If you would use it on more than one object, you could still be blocked by readline from the previous event.

For reading a line at time from stdin you do not need to use select.select(). Possibly the simplest way to achieve that:

for line in sys.stdin:
    print(line)

If you want to use either stdin or a list of files as arguments to your program, Python's fileinput module is out of the box solution.

In case you want/need to use select.select() anyways:

import os, sys, select

buffer = ""
while True:
  select.select([sys.stdin.fileno()], [], [])
  read = os.read(sys.stdin.fileno(), 512)

  # empty read: EOF
  if len(read) == 0:
    # buffer might not be empty
    if len(buffer) > 0:
      sys.stdout.write(buffer + "\n")

    break

  # find newlines
  parts = read.split("\n")
  buffer += parts.pop(0)

  while len(parts) > 0:
    sys.stdout.write(buffer + "\n")
    buffer = parts.pop(0)

When an object becomes ready, select.select unblocks. The only object in this case is stdin. With more than one object you need to check the return value of select.select to find out which one is ready. os.read() is used to do non-blocking read from stdin (alternatively: it's possible to use stdin.read(1) reading a character at a time). read.split("\n") is used for finding newlines, note that it is possible a single read yields more than one line.

Answer 2

After running strace on cat, I created this:

import sys

while True:
    data = sys.stdin.read()
    if not len(data):
        break
    sys.stdout.write(data)

Command

(sleep 2; echo test; sleep 2; echo test1) | python3 test.py

Output

test
test1

This solution relies on this fact:

If the end of the file has been reached, f.read() will return an empty string ('').

See Methods of File Objects.

If you need a version that uses select:

import os
import select
import sys

while True:
    rlist = [sys.stdin.fileno()]
    wlist = []
    xlist = []
    rlist, wlist, xlist = select.select(rlist, wlist, xlist)
    if sys.stdin.fileno() in rlist:
        data = os.read(sys.stdin.fileno(), 4096)
        if not len(data):
            break
        sys.stdout.write(data)