Which logging library to use for cross-language (Java, C++, Python) system

Question

I have a system where a central Java controller launches analysis processes, which may be written in C++, Java, or Python (mostly they are C++). All these processes currently run on the same server. What are you suggestions to

• Create a central log to which all processes can write to
• What if in the future I push some processes to another server. How can I support distributed logging?

Thanks!

Answer 1

I'd recommend using the platform's native logger which is syslog on Posix and Event Log on Windows.

For C++, you can use the native calls on the platform.

I know Python comes with syscall wrapper on Posix and there are wrappers for Event Log in the PyWin32 extension. I assume that someone has created Java wrappers by now.

Update

Regarding syslog and multiple files. syslog support the concept of facilities - via facilities you can have different logs go to different files. Unfortunatley, facilities are predefined; while there are 8 generic ones LOG_LOCAL0 through LOG_LOCAL7 you cannot define arbitrary facilities.

Also note that it's up to the syslog daemon to decide where to route log messages for each facility / level. You may need to adjust your syslog daemon configuration to have each facility get sent to a different file.

Answer 2

Apache has cross-platform logging libraries, which allow you to log from various programming languages using similar APIs. Unfortunately they don't have a Python API, though you should be able to whip one up with log4cpp and Boost.Python.

A project I work on uses one of these libraries to log to a database, which allows us "distributed logging" with a centralized place for the log messages. I have to admit I'm not a fan of this. Another project I work on uses one of these libraries to log to the native logging facility. The Windows Event Log has some features for distributed logging, but AFAIK syslog does not.

Although I don't have any experience with it, a better fit may be Facebook's Scribe project. The feature set meets your requirements, including a Python API. Unfortunately it uses Thrift which doesn't work for C++ on Windows (that is, the Thrift compiler generates C++ code that only works on UNIX). You may be able to get around this problem using Cygwin, but I can't promise that approach will work.