Anynchronous Logging strategy for Python & PHP

Question

Here's the situation: We have a bunch of python scripts continuously doing stuff and ultimately writing data in mysql, and we need a log to analyse the error rate and script performance.

We also have php front-end that interacts with the mysql data and we also need to log the user actions so that we can analyse their behaviour, and compute some scoring functions. So we thought of having a mysql table table for each case (one for "python scripts" log and one for "user actions" log).

Ideally, we would be writing to thsese log tables asynchronously, for performance and low-latency reasons. Is there a way to do so in Python (we are using django ORM) and in PHP (we are using Yii Framework) ?

Are there any better approaches for solving this problem ?

Update : for the user actions, (Web UI), we are now considering loading the Apache Log into mysql with relevant session info automatically through simple Apache configuration

Answer 1

There are (AFAIK) only two ways to do anything a-synchronously in PHP:

• Fork the process (requires pcntl_fork)
• exec() a process and release it by (assuming *nix) appending > /dev/null & to the end of the command string.

Both of these approaches result in a new process being created, albeit temporarily, so whether this would afford any performance increase is debatable and depends highly on your server environment - I suspect it would make things worse, not better. If your database is very heavily loaded (and therefore the thing that is slowing you down) you might get a faster result from dumping the log messages to file, and having a daemon script that crawls for thing to enter into the DB - but again, whether this would help is debatable.

Python supports multi-threading which makes life a lot easier.