How do you go about batch processing poorly formatted text files?

Question

People complain a lot about XML but, when compared to EDI and some of the proprietary file formats I've dealt with in my career, I think XML is bliss. The work I did on importing data files from Automotive Comparative Raters, each with it's own creative and nightmarish file format, still gives me nightmares.

That being said I'm curious how other programmers approach automated parsing of poorly formatted text files. Do you have a language preference? Are there any automation tools that you find invaluable? How do you make your code reusable?

Answer 1

A solution I learned about quite recently is using a standalone lexer. You get to use structured regular expressions and you avoid the constraints of a full blown parser generator.

Here are some examples with ocamllex (the lexer generator provided with OCaml):

• an ocamllex tutorial with some examples.
• processing of genbank loosely formatted text files (other link which better illustrates the point but hindered by a javascript dialog).

Obviously lexer generators are also available in other languages if using OCaml is an issue for you.

Answer 2

Perl / Python, build up functionality slowly, keep the worse ones as test case, lots of coffee