Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is there an open source MemoryStream implementation that does not use contiguous memory?

Like GuyFawkes, I would like to use MemoryStream to store a large amount of data, but keep encountering the 'out of memory' exceptions.
TomTom's answer is what I would like to do - use an implementation that does not require a contiguous block - but I'm wondering if there is already a free implementation available, to save me writing one myself?

Does anyone know of a good, free re-implementation of MemoryStream that can work with large streams?

EDIT:

The MemoryMappedFile solution is very interesting and I will be remembering it for other projects, however as Henk says, it strays too far from the abstraction that MemoryStream is aiming for. Specifically, the requirement of a known capacity.
The data that the replacement shall handle will in some cases be very large, but in others relatively small (and no we don't know which it will be until its too late ;)); further, many instances of the class will be in existence at the same time. Ultimately the work required to use MemoryMappedFiles (to determine an appropriate size for each one) would be equivalent to that of implementing TomTom's solution.

like image 737
sebf Avatar asked Mar 14 '12 16:03

sebf


2 Answers

Here is my implementation in case anyone needs it; I will leave this question open for a bit in case someone still responds with anything better.

http://www.codeproject.com/Articles/348590/A-replacement-for-MemoryStream

like image 74
sebf Avatar answered Sep 23 '22 02:09

sebf


You yould create a MemoryMappedFile without a file, i.e. one that lives in system memory. The DelayAllocatePages option delays allocations until the memory is actually needed. You need to specify a maximum capacity in advance though. Use the CreateViewStream Method to create a stream.

like image 21
dtb Avatar answered Sep 23 '22 02:09

dtb