Parsing Log4j Layouts from Log Files [closed]

Question

Is there any open source tool out there that can read all Layout implementations of Apache Log4j 1.2.x into something meaningful (e.g., Log4j's own LogEvent objects)? So far I haven't found an application (Chainsaw included) that can do that.

I am looking for something straightforward, that converts from a text file (plain or XML), into a Java object, given the file name and the pattern parameters (e.g., the format pattern in PatternLayout). If it has a clear API that makes it easy to integrate into an application, even better, but this is not necessary.

Answer 1

Here is the code Chainsaw uses to convert patternlayouts into something useful, using only log4j APIs. In the case of the latest developer snapshot of Chainsaw, it is used to build a Chainsaw config directly from log4j xml or properties file fileappender entries.

Use the converters and fields populated by PatternParser#parse to do what you want. The PatternParser class is included in the log4j 'extras' companion.

From http://svn.apache.org/repos/asf/logging/chainsaw/trunk/src/main/java/org/apache/log4j/chainsaw/LogFilePatternLayoutBuilder.java

public static String getLogFormatFromPatternLayout(String patternLayout) {
    String input = OptionConverter.convertSpecialChars(patternLayout);
    List converters = new ArrayList();
    List fields = new ArrayList();
    Map converterRegistry = null;

    PatternParser.parse(input, converters, fields, converterRegistry, PatternParser.getPatternLayoutRules());
    return getFormatFromConverters(converters);
}

If you actually want something that will read a log file conforming to a specific PatternLayout and generate log4j LoggingEvents, see LogFilePatternReceiver. Chainsaw uses the getLogFormatFromPatternLayout method to convert the patternLayout into the LogFormat required by the receiver.

By the way, the developer snapshot of Chainsaw contains a lot of additional new features - if you want to try it out, you can get it here:

http://people.apache.org/~sdeboy

Answer 2

I'm developing open source log viewer OtrosLogViewer. You can use it as library in your application. Take a look at this example.

//Define log4j properties
Properties p = new Properties();
p.put("type", "log4j");
p.put("pattern", "TIMESTAMP LEVEL [THREAD]  MESSAGE");
p.put("dateFormat", "yyyy-MM-dd HH:mm:ss,SSS");
Log4jPatternMultilineLogParser logParser = new Log4jPatternMultilineLogParser();    
LogImporterUsingParser importerUsingParser = new LogImporterUsingParser(logParser);
importerUsingParser.init(p);

//
ParsingContext context = new ParsingContext();

//Create log collector, it capture all parsed log events
ProxyLogDataCollector dataCollector = new ProxyLogDataCollector();

//Create input stream from file
InputStream in = new FileInputStream("log4j/log4j.txt");

//parse log file
importerUsingParser.importLogs(in, dataCollector, context);

Answer 3

I have come across your project and it is a nice job, however ideally I would like something that takes a native Log4j conversion pattern (e.g., %-5p [%t]: %m%n) and a file formatted using that pattern, and reads the corresponding information.

Essentially, I am looking for the reverse of the Layout.format(LoggingEvent) method of the base Layout class of Log4j. This method produces a String that is stored in the log file, so the reverse operation would be able to parse that file and create LoggingEvent objects, or something similar (something like a deserialization procedure).

Thanks!

EDIT: The conversion is straightforward, by just doing

LogFilePatternLayoutBuilder.getLogFormatFromPatternLayout(conversionPattern);

via the Chainsaw class org.apache.log4j.chainsaw.LogFilePatternLayoutBuilder.