What's the best tool to parse log files? [closed]

Question

I use grep to parse through my trading apps logs, but it's limited in the sense that I need to visually trawl through the output to see what happened etc.

I'm wondering if Perl is a better option? Any good resources to learn log and string parsing with Perl?

I'd also believe that Python would be good for this. Perl vs Python vs 'grep on linux'?

Answer 1

In the end, it really depends on how much semantics you want to identify, whether your logs fit common patterns, and what you want to do with the parsed data.

If you can use regular expressions to find what you need, you have tons of options. Perl is a popular language and has very convenient native RE facilities. I personally feel a lot more comfortable with Python and find that the little added hassle for doing REs is not significant.

If you want to do something smarter than RE matching, or want to have a lot of logic, you may be more comfortable with Python or even with Java/C++/etc. For instance, it is easy to read line-by-line in Python and then apply various predicate functions and reactions to matches, which is great if you have a ruleset you would like to apply.

Answer 2

All scripting languages are good candidates: Perl, Python, Ruby, PHP, and AWK are all fine for this. Using any one of these languages are better than peering at the logs starting from a (small) size.

Wearing Ruby Slippers to Work is an example of doing this in Ruby, written in Why's inimitable style. Here's a basic example in Perl. I suggest you choose one of these languages and start cracking.