Are there any existing batch log file aggregation solutions?

Question

I wish to export from multiple nodes log files (in my case apache access and error logs) and aggregate that data in batch, as a scheduled job. I have seen multiple solutions that work with streaming data (i.e think scribe). I would like a tool that gives me the flexibility to define the destination. This requirement comes from the fact that I want to use HDFS as the destination.

I have not been able to find a tool that supports this in batch. Before re-creating the wheel I wanted to ask the StackOverflow community for their input.

If a solution exists already in python that would be even better.

Answer 1

we use http://mergelog.sourceforge.net/ to merge all our apache logs..