Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

What's more efficient - storing logs in sql database or files?

I have few scripts loaded by cron quite often. Right now I don't store any logs, so if any script fails to load, I won't know it till I see results - and even when I notice that results are not correct, I can't do anything since I don't know which script failed.

I've decided to store logs, but I am still not sure how to do it. So, my question is - what's more efficient - storing logs in sql database or files?

I can create 'logs' table in my mysql database and store each log in separate row, or I can just use php's file_put_contents or fopen/fwrite to store logs in separate files.

My scripts would approximately add 5 logs (in total) per minute while working. I've done few tests to determine what's faster - fopen/fwrite or mysql's insert. I looped an "insert" statement 3000 times to make 3000 rows and looped fopen/fwrite 3000 times to make 3000 files with sample text. Fwrite executed 4-5 times faster than sql's insert. I made a second loop - I looped a 'select' statement and assigned it to a string 3000 times - I also opened 3000 files using 'fopen' and assigned the results to the string. Result was the same - fopen/fwrite finished the task 4-5 times faster.

So, to all experienced programmers - what's your experience with storing logs? Any advice?

// 04.09.2011 EDIT - Thank you all for your answers, they helped ma a lot. Each post were valuable, so it was quite hard to accept only one answer ;-)

like image 210
biphobe Avatar asked Aug 31 '11 13:08

biphobe


People also ask

Should I log to database or file?

In hindsight, a better answer is to log to BOTH file system (first, immediately) and then to a centralized database (even if delayed). Most modern logging frameworks follow a publish-subscribe model (often called logging sources and sinks) which will allow multiple logging sinks (targets) to be defined.

Is it better to store files in database or filesystem?

Database provides a proper data recovery process while file system did not. In terms of security the database is more secure then the file system (usually). The migration process is very easy in File system just copy and paste into the target while for database this task is not as simple.

Which database is best for storing logs?

If you are only logging lots and lots of simple log messages, MongoDB is a very good choice as it scales so good.

Are files faster than databases?

As a general rule, databases are slower than files. If you require indexing of your files, a hard-coded access path on customised indexing structures will always have the potential to be faster if you do it correctly. But 'performance' is not the goal when choosing a database over a file based solution.


1 Answers

Logs using files are more efficient, however logs stored in the database are easier to read, even remotely (you can write a web frontend if required, for example).

Note however that connecting and inserting rows into the database is error prone (database server down, password wrong, out-of-resources) so where would you log those errors if you decided to use the database?

like image 162
trojanfoe Avatar answered Oct 18 '22 08:10

trojanfoe