java - analyzing big text files

Question

I need to analyze a log file at runtime with Java.

What I need is, to be able to take a big text file, and search for a certain string or regex within a certain range of lines.

The range itself is deduced by another search.

For example, I want to search the string "operation ended with failure" in the file, but not the whole file, only starting with the line which says "starting operation".

Of course I can do this with plain InputStream and file reading, but is there a library or a tool that will help do it more conveniently?

Answer 1

If the file is really huge, then in your case either good written java or any *nix tool solution will be almost equally slow (it will be bound to IO). In such a case you won't avoid reading the whole file line-by-line.... And in this case few lines of java code would do the job ... But rather than once-off search I'd think about splitting the file at generation time, which might be much more efficient. You could redirect the log file to another program/script (either awk or python would be perfect for it) and split the file on-line/when generated rather than post-factum.