C# System.Buffer.BlockCopy Memory Issue?

Question

I have a problem that has been bothering me now for a few days. I have tried to Google the problem, but so far have not been able to find any solutions, not even a single person with the same problem.

It seems that the C# method System.Buffer.BlockCopy leaves you with some sort of memory ghosts. I have for example this method:

private float[,] readFloatArray2 (byte[] b) {
   int floatSize = sizeof(float);
   float[,] v = new float[2, (b.Length / 2) / floatSize];
   System.Buffer.BlockCopy(b, 0, v, 0, b.Length);

   return v;
}

to convert a byte array to a 2D float array. The data is previously read from a stream. I have located the problem to be the System.Buffer.BlockCopy method.

If I remove the BlockCopy command, the memory used by the application will be half the size. this means that its not my fault that the byte array is still alive. because without the BlockCopy command, the byte array dies properly. the float array gets created anyway (with or without the copied information).

I am not quite sure if this is a problem of the BlockCopy command or the GC because I have also tried to call System.GC.Collect(); after the BlockCopy and then it also works perfectly (I know you should not do this... thats why I am asking for advise here).

I also wouldn't bother asking, but the issue involves several hundred meg.

Besides the memory issues, the method works perfectly fine. Does anyone know what causes the memory problem?

greetings and thanks in advance oli

ps: I am using .NET4.0 with Visual Studio 2010 PRO and WIN7... don't know whether this is relevant or not.

Answer 1

I have located the problem to be the System.Buffer.BlockCopy method. If I remove the BlockCopy command, the memory used by the application will be half the size. this means that its not my fault that the byte array is still alive. Because without the BlockCopy command, the byte array dies properly.

I disagree with that conclusion. I see several phases:

1. The byte array exists and is filled with data
2. You allocate the float array, but it isn't filled with data yet.
3. You fill the float array with data
4. The byte array isn't referenced anymore, but hasn't been collected yet
5. The byte array has been collected.

The live of the byte array isn't influenced by BlockCopy.

Step 2 reserves and commits virtual memory. So the commit size grows in this step. But since the content of the array has never been written to, and consists entirely of 00 bytes, the windows memory manager doesn't allocate any physical memory for it. It just notes that these pages consist entirely of 00s.

The physical memory for the float array get only allocated in step 3. You'd get the same effect if you'd add a loop that initialized every field in the array.

---

Apart from the actual problem, I also have some design suggestions:

1. Reuse buffers. The GC is good for small shortlived objects, but very bad for large shortlived objects. This means you should not use functions that allocate and return large arrays. Instead take them as a parameter, so an existing array can be reused.

2. If you're using working with audio data(Seems likely), I wouldn't use a solid 2D array. I'd instead use an array of arrays. Where the inner array represents the samples in a single buffer, and the outer array represents the buffers. This has two advantages:

   • You can easily write code that only operates on a single channel.
   • Solid 2D arrays are slow to index, so it's often faster too.
3. Do you really want to read all data at once? I'd read in chunks of a few kilobytes.

Answer 2

BlockCopy doesn't have managed .NET implementation. Internally, it invokes external win api.

[SecuritySafeCritical]
public static extern void BlockCopy(Array src, int srcOffset, Array dst, int dstOffset, int count);