Python parsing log file to extract events in real time

Question

I've a process that is logging messages to a file.

I want to implement another process (in Python) that parses these logs (as they are written to the file), filters the lines that I'm interested in and then performs certain actions based on the state of the first process.

I was wondering before I go ahead and write something on my own if there is a library in Python that does something like this.

Also, ideas regarding how implement something like this Python would be appreciated.

Thanks.

Answer 1

C programs usually seek to the current position to clear any “end of file” flags. But as @9000 correctly pointed out, python apparently takes care of this, so you can read from the same file repeatedly even if it has reached end of file.

You might have to take care of incomplete lines, though. If your application writes its log in pieces, then you want to make sure that you handle whole lines, and not those pieces. The following code will accomplish that:

f = open('some.log', 'r')
while True:
    line = ''
    while len(line) == 0 or line[-1] != '\n':
        tail = f.readline()
        if tail == '':
            time.sleep(0.1)          # avoid busy waiting
            # f.seek(0, io.SEEK_CUR) # appears to be unneccessary
            continue
        line += tail
    process(line)