Tail read a growing dynamic file and extract two columns and then print a graph

Question

What is the best way to read a 1 GB file that gets time series data logged in it and generate a real time graph with two of its columns (one time and other a number)? I see that you have different ways of tailign the file.

Answer 1

Sounds like a good job for RRDTool.

But if you want to stick with Python, I would use tail to stream the data into my program (this is assuming the file is continuously written to, otherwise a straight open() in Python will work).

tail -F data.log | python myprogram.py

myprogram.py could look something like:

import sys

p = ... # create a pylab plot instance 
for line in sys.stdin:
    elements = line.split(',') # or whatever separator your file has in it
    p.add(element[0], element[1]) # add data to the pylab plot instance

Answer 2

Here's the unix pipe which has 3 parts: the tail'er, the filter (gawk), and the plotter (python).

tail -f yourfile.log | gawk '/PCM1/{print $21; fflush();}' | python -u tailplot.py

and here is the python script. You can feed it 1 (y) or 2 (x y) columns of data. If you don't use gawk, be sure to figure out how to disable buffering. sed -u for example.

pa-poca$ cat ~/tailplot.py

import math
import time
import sys
import pylab

pylab.ion()
pylab.xlabel("X")
pylab.ylabel("Y")

x = []
y = []
counter = 1
while True :
    line = sys.stdin.readline()
    a = line.split()
    if len(a) == 2:
      x.append(a[0])
      y.append(a[1])
    elif len(a) == 1:
      x.append(counter)
      y.append(a[0])
      counter = counter + 1
    pylab.plot(x, y, 'b')
    pylab.draw()