I'm interested in sending all Rails application logging to a database (MySQL or MongoDB) either in addition to or instead of to a log file. There are a few reasons, most of which are concerned about log file analysis. We already use Google Analytics, but there are a variety of things we want to do that aren't as workable in Analytics.
Furthermore, I'd like to do "real time" investigation of issues by looking at logs. Sifting through a log file is a tedious way to do that, and I'd like to do better searching and filtering than a log file (easily) allows for.
Finally, I often want to examine something closer to site visitor behavior: tracing the path through the site for example, so that I can see what the last page was that a user was looking at before an error occurred. Given we have multiple app servers, the separate log files make this a real pain. If all the data were in a database, I could then easily see the proper sequence of pages for a given visitor. I know that Syslog would be one way to solve this particular thing (single log file/repository), but I want to combine that with better searching abilities that I associate with database searches.
I'm wondering what folks recommend to solve this. Do you directly log to a database, or do you dump log files into a DB (but what's your approach for that so that it's essentially realtime/as up to date as the logfile itself)?
I am currently determining at what level I'd like this logging, because another thing I looked at is writing a small Rack filter that would log all requests. This would miss all the extra output that the normal Rails logging dumps out (all the SQL and output on cache hits and misses, etc.), but it would achieve a big part of my goal, and seems to have the advantage of not disturbing anything else in the system.
Anyway, I am not looking for one right answer, more of a discussion and information on what anyone else might be doing in this same light.
In hindsight, a better answer is to log to BOTH file system (first, immediately) and then to a centralized database (even if delayed). Most modern logging frameworks follow a publish-subscribe model (often called logging sources and sinks) which will allow multiple logging sinks (targets) to be defined.
Database logging is an important part of your highly available database solution design because database logs make it possible to recover from a failure, and they make it possible to synchronize primary and secondary databases. All databases have logs associated with them. These logs keep records of database changes.
The bottom line is that DB logging will slowly kill your operational DB performance; your DB response times will grow, and may become unreliable since they will have to wait for resources — most frequently, disk I/O. Mixing your read heavy operational DB with a write heavy logging is not a good idea.
My company have been logging some structured traffic info straight into a MySQL log database. This database is replicated downstream to another database. All analytics run off the final database replication. Our site sustain quite a bit of traffic. So far, it doesn't seem to have any major problems. However, our IT department have some growing concerns regarding to the scalability of the current setup and is suggesting that we offload the log info onto "proper" log-files. The log-files will then be reinserted back into the same downstream database tables. Which brings me to this question. :)
Here are some of pros and cons that I see regarding to the subject of log-files vs log-db (relational):
I think some stress tests on the log database are needed in my situation. This way at least I know how much headroom I have.
Recently, I've been looking into some key-value / document-based databases like Redis, Tokyo Cabinet, and MongoDB. These fast inserting databases can potentially be the sweet spot since they provide persistence, high (write) throughputs, and querying capabilities to varying degrees. They can make the data-extraction process much simpler than parsing and map-reducing through gigs of log files.
In the long run, I believe it is crucial to have a robust analytics data warehouse. Freeing application data from analytic data and vice versa can be a big WIN.
Lastly, I would just like to point out there are many similar / closely related questions here on StackOverflow in case you want to broaden your discussion.
Edit:
rsyslog looks very interesting. It gives you the ability to write directly to MySQL. If you are using Ruby, you should have a look at the logging gem. It provides multi-target logging capabilities. It's really nice.
If you want to change the default logging behavior, simply create a custom logger object that respond to all the Rails logger method:
http://github.com/rails/rails/blob/9d7aae710384fb5f04129c35b86c5ea5fb9d83a9/activesupport/lib/active_support/buffered_logger.rb
Because it's your logger, you can decide to implement your personal logic. You can write to the database, to the standard output of whenever you want.
Then, replace the default logger for every base class you want to customize.
ActiveRecord::Base.logger = YouLogger.new
You can easily create an initializer file called logger.rb and write there all your custom configurations. In this way, the logger will be immediately replaced on Rails startup.
Chris,
I think Dima's comment is important here. Are you satisfied with (1) having an access log in a DB (in real time), or (2) are you more interested in Rails/app-specific logging?
For (1), with Apache (at least), you can log to a database using piped logging.
http://httpd.apache.org/docs/1.3/logs.html#piped
I wrote a program that runs in the background waiting for input, which it parses and logs to a Postgres DB. My httpd.conf file pipes to this program with a CustomLog directive.
This is relatively simple to set up, and gives you all the obvious advantages of being able to analyze your logs in a DB. It works very well for me, especially for tracing what a user was doing just before an error. However, you have to protect against sql injection, buffer overflows, and other security issues in the logging program.
For (2), I am not a Rails developer so I can only talk about general approaches. If you want to log environment vars, or application data, or very selective bits of information, you could consider writing a web server module. Depending on your exact needs, you could also get by with some combination of conditional logging directives and filtering in the logging program.
It really comes down to whether you need a Rails-specific solution or a more general web-server-wide solution.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With