Distributed Logging with flume [closed]

Question

I have a mobile service distributed over 7 servers each of them doing a specific task. I want to log information from them and later derive business intelligence from them. I have rounded it to Flume. How can I use it to gather information? My system is written in PHP. Does flume work on PHP?

Answer 1

It depends on your needs and what your server environment is like. One thing I can tell you is that Flume has no direct integration with PHP. However, there are other ways around this.

I'm running servers hosted in Amazon EC2 running a combination of rsyslog + flume. In my setup, I collect web logs from my servers running nginx running on linux servers. The nginx servers emit web request logs as syslog messages into rsyslog; rsyslog makes a tcp request to my central flume collector; the flume collector listens to these messages with the syslogTcp sink; the flume collector forwards the messages into Amazon S3. I then analyze the logs files with Amazon EMR at some later point in time.

In your situation, PHP can also be configured to write to syslog (http://php.net/manual/en/function.syslog.php); therefore, you can have a similar setup and have the syslog forward the logs to a central flume collector node.

If you don't want to rely on syslog messages, you can also have flume clients running on your servers. The flume clients can be configured to tail local log files with flume's tail sink or you can tail all log files in a specified directory with flume's tailDir sink and have them streamed to a flume collector.

A nice benefit of flume is that you can configure it to have messages delivered with a very good probability that it'll reach it's destination for important messages, while other messages can be sent with lower delivery requirements.

The flume user guide is your best place to get more detailed information: http://flume.apache.org/FlumeUserGuide.html

Another good place to look is to jump on freenode and join the #flume channel.