Python sys.stdin.read(max) blocks until max is read (if max>=0), blocks until EOF else, but select indicates there is data to be read

Question

My problem is:

select indicates that there is data to be read, I want to read whatever is there, I do not want to wait for a max amount to be present. if max <= 0 then read waits until EOF is encountered, if max>0 read blocks until max bytes can be read.

I don't want this, I want to read whatever amount made select put it in the "ready for reading" list. read(1) isn't practical because this will involve A LOT of calls to read. BUT it mustn't block.

Is there a way to find out the amount present in the buffer when select returns (if it returns indicating something can be read, rather than timed out) and read that amount? Is there a way to use max as one would with sockets? Where it reads as much as it can instantly, then returns?

A solution might be to put the file in non-blocking mode for the read? I'm not sure, I didn't expect this "until EOF" behavior.

I will keep reading and trying but I've just spent 30 minutes or so and gotten no closer, that's why I appeal to you.

NOTE

There are a lot of questions asking how to make recv wait for an amount of input, and making things block until that max has arrived, I am not looking for this. My problem is it is blocking.

Addendum

setblocking(False) didn't work, I'm now reading on how to make it non-blocking for the duration of the read. I am given hope by the documentation:

stdin.read Found at: sys
read([size]) -> read at most size bytes, returned as a string.

If the size argument is negative or omitted, read until EOF is reached.
Notice that when in non-blocking mode, less data than what was 
 requested
may be returned, even if no size parameter was given.

Addendum II

It seems read(0) actually reads 0, that is nothing, this results in an endless loop, which is interesting!

I'm sorry I didn't actually try 0 it seems, I started with 4096 (thinking it'll read whatever is there....) tried with no parameter, but not with 0.

I wonder what use read(0) would be?

Addendum III

I'm now having problems with select (I've tried to take read(1) as a solution) Here's the actual code:

def getInput(self):
    log.log(log.INFO,"GetInput","Select")
    readsReady = select.select((sys.stdin,),(),(),1)[0]
    if len(readsReady) == 0:
        #timed out
        log.log(log.INFO,"GetInput","Select timed out")
        if not self.toClose:
            self.handler.post("GetInput")
        else:
            threads.getCurrentThread().removeAllHandlers()
    else:
        #OPTIMISED FOR READING 1
        #log.log(log.INFO,"GetInput","Reading")
        data = sys.stdin.read(1)
        log.log(log.INFO,"GetInput","Read: "+data)

        if data == "\n":
            self.onInputHandler.post("OnInput",self.buffer)
            self.buffer=""
        else:
            self.buffer+=data
        self.handler.post("GetInput")

There are few things in the following output unrelated to this, they are the "Hello world!" that comes through almost instantly, and the "TEST!" about 5 seconds in. The "hello" is something I typed, input, the "k" is something I typed later, after typing both I press enter once.

Output:

0.0147    Verbose        1   SocketReader                        Created reader
0.0156    Verbose        2   SocketWriter                        Created writer
0.0260    Information    0   SocketReadWriter                    Created and ready for: ('localhost', 8294)
0.0268    Information    3   GetInput                            Select
Hello World!
1.0281    Information    3   GetInput                            Select timed out
1.0584    Information    3   GetInput                            Select
2.0593    Information    3   GetInput                            Select timed out
2.0896    Information    3   GetInput                            Select
3.0900    Information    3   GetInput                            Select timed out
3.1203    Information    3   GetInput                            Select
4.1215    Information    3   GetInput                            Select timed out
4.1519    Information    3   GetInput                            Select
TEST!
5.1524    Information    3   GetInput                            Select timed out
5.1828    Information    3   GetInput                            Select
hello
6.1467    Information    3   GetInput                            Read: h
6.1770    Information    3   GetInput                            Select
7.1782    Information    3   GetInput                            Select timed out
7.2086    Information    3   GetInput                            Select
8.2098    Information    3   GetInput                            Select timed out
8.2401    Information    3   GetInput                            Select
9.2414    Information    3   GetInput                            Select timed out
9.2717    Information    3   GetInput                            Select
10.2723   Information    3   GetInput                            Select timed out
10.3026   Information    3   GetInput                            Select
k
10.7939   Information    3   GetInput                            Read: e
10.8243   Information    3   GetInput                            Select
10.8245   Information    3   GetInput                            Read: l
10.8547   Information    3   GetInput                            Select
10.8549   Information    3   GetInput                            Read: l
10.8851   Information    3   GetInput                            Select
10.8853   Information    3   GetInput                            Read: o
10.9155   Information    3   GetInput                            Select
10.9157   Information    3   GetInput                            Read: 

10.9459   Information    3   GetInput                            Select
10.9461   Information    3   GetInput                            Read: k
10.9763   Information    3   GetInput                            Select
You said: hello
11.9775   Information    3   GetInput                            Select timed out
12.0123   Information    3   GetInput                            Select
13.0133   Information    3   GetInput                            Select timed out
13.0437   Information    3   GetInput                            Select
^C13.3985   Verbose        2   Threads                             Thread: 2 has ended
14.0442   Information    3   GetInput                            Select timed out
14.0746   Information    3   GetInput                            Select
14.3622   Verbose        1   Threads                             Thread: 1 has ended
15.0758   Information    3   GetInput                            Select timed out
15.1363   Information    3   GetInput                            Select
16.1373   Information    3   GetInput                            Select timed out
16.1677   Verbose        3   Threads                             Thread: 3 has ended

This is easier to see here: http://pastebin.com/raw.php?i=H6UHHmy8

What's weird?

It reads the "h" of hello, but doesn't read the "hello\n" until k happens, it is always 1 letter ahead of 1 newline behind if that makes sense.

Would multiple calls to select cause a problem? (in another thread the socket reader also uses select)

The format for the log is:

*Time since the program start

*Log level

*Thread ID (Unique among running threads)

*Log tag

*Log message

What the Handler class does

is allow threads to post messages safely to each other, the handler checks a queue (and any booked events to happen at a certain time, like the TEST, which happens on a different thread don't worry), the post of "GetInput" scheduals another call to GetInput, putting it at the back of the queue. The "OnInput" message is passed to the handler of a different thread, the one I want to deal with input.

I've done it this way because it provides a nice way to handle threading, and means I have nice re-usable code (like the SocketReadWriter), I hope this doesn't result in criticism of my threading model, but it actually works. The problem is with my attempt to get user input.

You can also see when I do ctrl+c that the thing shuts down, this is what the toClose thing does. When select times out, if it is supposed to close, it will. A thread is ended when it has no handlers left (handlers only take over after the function the thread is to run has returned, this function might create a class which has a handler for a member, thus when the constructor returns, and the function returns, there's a handler keeping the class alive)

Work around

def getInput(self):
    log.log(log.INFO,"GetInput","Select")
    if sys.stdin.closed:
        readsReady = []
    else:
        readsReady = select.select((sys.stdin,),(),(),1)[0]
    if len(readsReady) == 0:
        #timed out
        log.log(log.INFO,"GetInput","Select timed out")
        if not self.toClose:
            self.handler.post("GetInput")
        else:
            threads.getCurrentThread().removeAllHandlers()
    else:
        data = sys.stdin.readline()
        if len(data) == 0:
            log.log(log.WARN,"GetInput","No data was returned indicating the file was closed")
            self.handler.post("GetInput") #if this is a close event, the next
            #timeout will deal with it
            return
        if data[-1] == "\n":
            data = data[:-1]
        log.log(log.INFO,"GetInput","Read: "+data)
        self.onInputHandler.post("OnInput",data)
        #if data == "\n":
        #   self.onInputHandler.post("OnInput",self.buffer)
        #   self.buffer=""
        #else:
        #   self.buffer+=data
        self.handler.post("GetInput")

def onClose(self):
    #log.log(log.WARN,"Input: OnClose","Called")
    self.toClose = True
    sys.stdin.close()

Answer 1

In os module there is os.read function that allows lower level control over reading from file descriptor. It is nonblocking as long as there is at least a byte ready to read.

os.read(fd, n)

Read at most n bytes from file descriptor fd. Return a string containing the bytes read. If the end of the file referred to by fd has been reached, an empty string is returned.

Availability: Unix, Windows.

Note: This function is intended for low-level I/O and must be applied to a file descriptor as returned by os.open() or pipe(). To read a “file object” returned by the built-in function open() or by popen() or fdopen(), or sys.stdin, use its read() or readline() methods.