Python: Read whitespace separated strings from file similar to readline

Question

In Python, f.readline() returns the next line from the file f. That is, it starts at the current position of f, reads till it encounters a line break, returns everything in between and updates the position of f.

Now I want to do the exactly the same, but for whitespace separated files (not only newlines). For example, consider a file f with the content

token1 token2

token3                            token4


         token5

So I'm looking for some function readtoken() such that after opening f, the first call of f.readtoken() returns token1, the second call retuns token2 etc.

For efficiency and to avoid problems with very long lines or very large files, there should be no buffering.

I was almost sure that this should be possible "out of the box" with the standard library. However, I didn't find any suitable function or a way to redefine the delimiters for readline().

Answer 1

You'd need to create a wrapper function; this is easy enough:

def read_by_tokens(fileobj):
    for line in fileobj:
        for token in line.split():
            yield token

Note that .readline() doesn't just read a file character by character until a newline is encountered; the file is read in blocks (a buffer) to improve performance.

The above method reads the file by lines but yields the result split on whitespace. Use it like:

with open('somefilename') as f:
    for token in read_by_tokens(f):
        print(token)

Because read_by_tokens() is a generator, you either need to loop directly over the function result, or use the next() function to get tokens one by one:

with open('somefilename') as f:
    tokenized = read_by_tokens(f)

    # read first two tokens separately
    first_token = next(tokenized)
    second_token = next(tokenized)

    for token in tokenized:
        # loops over all tokens *except the first two*
        print(token)