Blackbox type data logging

Question

In a Linux embedded application I'm developing, there is the need to record some events that happen from time to time. These records are saved on a MTD flash device and once written there is no need to change them or do efficient searches, but read access is required to show the data back to the user. A big problem is that power can go away at any time, without a proper shutdown sequence. The frequency these events occur can be very slow (days/weeks), but several of them will occur at once. The data to be saved for each event is strongly typed: date, time, a couple of short text strings and several integers.

Currently I inherited a solution based on jffs2 and SQLite that is far from optimal because the DB file sometimes get corrupted. When this happens the whole file gets unreadable and there is no way to understand if it was caused by a bug in jffs2, or in SQLite or if the flash sector was bad, or if the power was cut at the wrong time.

Is there a library or a combination of filesystem/library that can better help me solve this kind of problem ? Or should I just use a text file with a CSV-like format ?

Answer 1

I'm not expert on embeded systems, but I would think that a CSV would probably be best. It basically can't be corrupted, or if it does, then you can easily see the error and fix it manually (new line or just removing a line). I have been working on receiving the data from an embeded system where they have a lot of corruption problems (partially on the system and partially during the phone line transfer). It would be very helpful if it were in a CSV type format so we could find the errors and remove or fix them instead of corrupting the entire data set.

If you aren't needing to search within the system, then a CSV works perfectly.

Answer 2

We are using plain old syslogd to a YAFFS2 partition on NAND flash, it appears to work well: when messages are sent to the logger and power is removed immediately after (<100ms) the message is there and the log never appears to corrupt.

This is based on observation rather than my explicitly knowing that everything will always be consistent by design, mind.