C# high speed data logging data handling

Question

I've written an application that logs trace data from an embedded system via UDP. Currently I receive datagrams and parse out the variable length records and store them in a list. The front end can access the list and present the data (graphs and text lists etc).

The problem I'm running into is that sometimes I need to log an exceptional amount of data. So much that my list implementation causes an out of memory exception.

My requirements are:

• Allow multithreaded reading and writing of the data (can't just post process)
• Handle large amounts of data (worst case ~2MB/s ... 7.2GB/hr of logging)
• Allow storage of data set
• Random read, index based, access

Does anyone have some suggestions about how to attack this? Here were a few thoughts that I had:

• I'd like a nifty disk backed, memory cached List. It seems like something that would exist but I haven't found one.
• Local database? I don't know too much about databases but it seems like overkill.
• Store the data to a file right away. Keep a list in memory that holds the byte offset for each record index. Can my reader thread access this simultaneously?

Answer 1

A local database really would be a nice way to handle this - especially because queries would aid with your investigation of logs. Plus, then your UDP receiving program could just be a separate thread that spits information at the database (if your data's REALLY fast-paced, you could have two buffers and alternate between them; flush the full buffer to the database while the other one is filling up). It really depends on the scale of your project though.

You could always use your third option (storing to a file right away), and have a separate "Log Investigation" tool that reads that file in without running into OOM exceptions.