What allocating method to use for a high volume logger application?

Question

I'm developing a logger/sniffer using Delphi. During operation I get hugh amounts of data, that can accumulate during stress operations to around 3 GB of data. On certain computers when we get to those levels the application stops functioning and sometimes throws exceptions.

Currently I'm using GetMem function to allocate the pointer to each message.

Is there a better way to allocate the memory so I could minimize the chances for failure? Keep in mind that I can't limit the size to a hard limit.

What do you think about using HeapAlloc, VirtualAlloc or maybe even mapped files? Which would be better?

Thank you.

Answer 1

Your fundamental problem is the hard address space limit of 4GB for 32 bit processes. Since you are hitting problems at 3GB I can only presume that you are using /LARGEADDRESSAWARE running on 64 bit Windows or 32 bit Windows with the /3GB boot switch.

I think you have a few options, including but not limited to the following:

1. Use less memory. Perhaps you can process in smaller chunks or push some of the memory to disk.
2. Use 64 bit Delphi (just released) or FreePascal. This relieves you of the address space constraint but constrains you to 64 bit versions of Windows.
3. Use memory mapped files. On a machine with a lot of memory this is a way of getting access to the OS memory cache. Memory mapped files are not for the faint hearted.

I can't advise definitively on a solution since I don't know your architecture but in my experience, reducing your memory footprint is often the best solution.

Using a different allocator is likely to make little difference. Yes it is true that there are low-fragmentation allocators but they surely won't really solve your problem. All they could do would be make it slightly less likely to arise.