How do I read a file which is constantly updating?

Question

I am getting a stream of data (text format) from an external server and like to pass it on to a script line-by-line. The file is getting appended in a continuous manner. Which is the ideal method to perform this operation. Is IO::Socket method using Perl will do? Eventually this data has to pass through a PHP program (reusable) and eventually land onto a MySQL database.

The question is how to open the file, which is continuously getting updated?

Answer 1

In Perl, you can make use of seek and tell to read from a continuously growing file. It might look something like this (borrowed liberally from perldoc -f seek)

open(FH,'<',$the_file) || handle_error();  # typical open call
for (;;) {
    while (<FH>) {
        # ... process $_ and do something with it ...
    }
    # eof reached on FH, but wait a second and maybe there will be more output
    sleep 1;
    seek FH, 0, 1;      # this clears the eof flag on FH
}

Answer 2

In perl there are a couple of modules that make tailing a file easier. IO::Tail and File::Tail one uses a callback the other uses a blocking read so it just depends on which suits your needs better. There are likely other tailing modules as well but these are the two that came to mind.

IO::Tail - follow the tail of files/stream

 use IO::Tail;
 my $tail = IO::Tail->new();
 $tail->add('test.log', \&callback);
 $tail->check();
 $tail->loop();

File::Tail - Perl extension for reading from continously updated files

use File::Tail;
my $file = File::Tail->new("/some/log/file");
while (defined(my $line= $file->read)) {
    print $line;
}