Tail Sampling Logs

Question

I have a process that needs to do periodic processing on an ever-growing logfile. Right now, the way I do this is pretty simple (I'll include the bash script if you're truly curious).

• Start up tail -n0 -f $FILE
• Each iteration:
  • Kill the tail
  • Move the old sample
  • Start up a new tail

This solves for the problem of not having any overlap, but I'm worried about the 1 or 2 lines I might miss. Is there a better way of doing this to that avoids overlap (and "under"lap)?

Answer 1

By "move the old sample", I assume you mean rotating the file by moving the current one away and replacing it with a new file.

If so, then you could use the --follow=name option for tail instead of -f. This follows the file by name rather than the file descriptor, which allows it to continue even if the files is replaced. You can then leave your tail running while you replace the files and not miss any entries.

For a more robust approach also include --retry, or simply use -F which implies --follow=name --retry.

From the man page:

-f, --follow[={name|descriptor}]

output appended data as the file grows; -f, --follow, and --follow=descriptor are equivalent

--retry

keep trying to open a file even if it is inaccessible when tail starts or if it becomes inaccessible later - useful only with -f

-F

same as --follow=name --retry