Minimal overhead distributed event logging library/framework?

Question

We'd like to keep records of all major events in our systems. For example, where the database might store the current user status, the event log should record all changes to that status along with when the changes occurred.

The event logging facility should be as close to zero overhead for the event thrower as possible, should accommodate structured information (as opposed to text log messages), and should support distributed deployment (many boxes throwing many events).

In a past life we had a UDP based system that worked well because we had great control over the system (minimized packet loss). The even throwers would fire off UDP packets that would be caught and journaled on other boxes. I'm looking for something similar, hopefully open source, off the shelf, and deployable in more general networks. Alternatively I'm open to suggestions for how to build something like this.

This should work across multiple languages, but will be primarily targeted for Java and Python. The pariticipating (event throwing) applications will vary; some will be web apps, others batch oriented apps. The results will likely live in Hadoop/HDFS/HBase.

Answer 1

If you want to go down the UDP route (as you seem happy with that), and Java is an option, then check out Log4j and its support for UDP transmission via the Log4j UDPAppender.

LoggingEvent will take a java.lang.Object as a message, so it's pretty generic and you can throw whatever data you want into that. If you're going across the network it should (most likely) be serialisable, and given that you want UDP, should be of a comensurate size - 64k or less, and then dependent on the transport layer). You'll simply have to intercept the LoggingEvent on the server side and then process it however you want.

Note that the UDP appender comes as a companion component to Log4j and you'll have to build it yourself. But that's trivial.

Answer 2

You may consider using old good *nix Syslog. It has very small overhead and is mostly used over UDP or local UNIX sockets, but may use TCP if you need reliable logging. Works for my (Python/Perl, mostly, but it is completely language/platform-agnostic) like a charm.

Sorry, I'm not familiar with Java, but feature-wise, this seems to be some good library I've googed: http://syslog4j.org/

Edit: Quick googling discovered an article called "Robust event logging with Syslog", which seems to be pretty detailed on the subject. Sorry, I've misread it when posted and thought it is a *nix syslog library, but it isn't.