Grep in reverse order without reading whole file

Question

I have a log file that may be very large (10+ GB). I'd like to find the last occurrence of an expression. Is it possible to do this with standard posix commands?

Here are some potential answers, from similar questions, that aren't quite suitable.

• Use tail -n <x> <file> | grep -m 1 <expression>: I don't know how far back the expression is, so I don't know what <x> would be. It could be several GB previous, so then you'd be tailing the entire file. I suppose you could loop and increment <x> until it's found, but then you'd be repeatedly reading the last part of the file.
• Use tac <file> | grep -m 1 <expression>: tac reads the entire source file. It might be possible to chain something on to sigkill tac as soon as some output is found? Would that be efficient?
• Use awk/sed: I'm fairly sure these both always start from the top of the file (although I may be wrong, my sed-fu is not strong).
• "There'd be no speed up so why bother": I think that's incorrect, since file systems can seek to the end of a file without reading the whole thing. There'd be a little trial and error/buffering to find each new line, but that shouldn't slow things down much, compared to reading (e.g.) 10 GB that are never used.
• Write a python/perl script to do it: this is my fall-back if no one can suggest anything better. I'd rather stick to something that can be done straight through the command line, since I'm executing it straight through ssh, and I'd rather not have to upload a script file as well. Using mmap's rfind() in python, I think we can do it in a few lines, provided the expression to find is static (which mine, unfortunately, is not). A regex requires a bit more work, something like this.

If it helps, the expression is anchored at the start of a line, eg: "^foo \d+$".

Answer 1

Whatever script you write will almost certainly be slower than:

tac file | grep -m 1 '^foo [0-9][0-9]*$'