Logging to MongoDB from python

Question

i want to log some information into mongodb using python . i found 2 libraries mongodblog and log4mongo for python. any idea which one is better ? or any other library which is better than these ?

Answer 1

When you use MongoDB for logging, the concern is the lock contention by high write throughputs. Although MongoDB's insert is fire-and-forget style by default, calling a lot of insert() causes a heavy write lock contention. This could affect the application performance, and prevent the readers to aggregate / filter the stored logs.

One solution might be using the log collector framework such as Fluentd, Logstash, or Flume. These daemons are supposed to be launched at every application nodes, and takes the logs from app processes.

They buffer the logs and asynchronously writes out the data to other systems like MongoDB / PostgreSQL / etc. The write is done by batches, so it's a lot more efficient than writing directly from apps. This link describes how to put the logs into Fluentd from Python program.

• Fluentd: Data Import from Python Applications

Here's some tutorials about MongoDB + Fluentd.

• Fluentd + MongoDB: The Easiest Way to Log Your Data Effectively on 10gen blog
• Fluentd: Store Apache Logs into MongoDB