Grand Unified Theory of logging

Question

Is their a Grand Unified Theory of logging? Shall we develop one? Question (just to show this is not a discussion :), how can I improve on the following? (note that I live mainly in the embedded world, but non-embedded suggestions are also welcome)

How do you log, when do you log, what do you log, what do you do with log files?

How do you log - I generally have macros, #ifdef TESTING, sort of thing. They write to RAM and a low priority process writes them out when the system is idle (using UDP, since I do embedded systems)

When do you log - same as voting, early and often. At every (in)significant program event, I log at varying levels. Events received, transaction succeed/fail, data updated, etc

What do you log - Fatal/Error/Warning/Info/Debug/Trace is covered in When to use the different log levels?

What do you do with log files - 1) keep them (in CVS), both pass and fail 2) capture everything and filter later in case I can't repeat a problem. I have tools to filter the log by "level" (Fatal/Error/etc), process, file, etc. And to draw message sequence charts, dump data structures, draw histograms of memory usage - what am I missing?

Hmmm, binary or ascii log file format? Ascii is bulkier, but binary requires more processing. I have done both, currently I use ascii

Question - did I miss anything, and how can I improve on this?

Answer 1

You could "instrument" your code in many different ways, everything from start-up/shut-down events to individual machine instruction execution (using a processor emulator). Of all the possibilities, what's worth doing? Don't just do it for the sake of completeness; have a specific goal in mind. A business case if you like, with a benefit you expect to receive. E.g.:

• Insight into CPU task execution times/patterns to enable optimisation (if you need to improve performance).
• Insight into other systems to resolve system integration issues (e.g. what messages is your VoIP box sending and receiving when it connects to a particular peer?)
• Insight into the nature of errors (for field diagnostics)
• Aid in development
• Aid in validation testing

I imagine that there's no grand unified theory of logging, because what you do would depend on many details:

• Quantity of data
• Type of data
  • Events
  • Streamed audio/video
• Available storage
  • Storage speed
  • Storage capacity
• Available channels to extract data
  • Bandwidth
  • Cost
  • Availability
    • Internet connected 24×7
    • Site visit required
    • Need to unlock a rusty gate, climb a ladder onto a roof, to plug in a cable, after filling out OHS documentation
    • Need to wait until the Antarctic winter is over and the ice sheets thaw
• Random access vs linear access (e.g. if you compress it, do you need to read from the start to decompress and access some random point?)
• Need to survive error conditions
  • Watchdog reboots
  • Possible data corruption
    • Due to failing power supply
    • Due to unreliable storage media
    • Need to survive a plane crash

As for ASCII vs binary, I usually prefer to keep the logging simple, and put any nice presentation in a PC application that decodes the data. It's usually easier to create a user-friendly presentation in PC software (written in e.g. Python) rather than in the embedded system itself.