OutOfMemoryException when creating multiple byte arrays

Question

I'm constantly hitting an OutOfMemoryException inside a method that creates and processes some byte arrays. The code looks like this:

1. Create MemoryStream to get some data (about 60MB).
2. Create byte array (same size as MemoryStream, about 60MB)
3. Fill the array with bytes from memory stream
4. Close MemoryStream
5. Process data from byte array
6. Leave method

When this method is called like 20-30 times I get OutOfMemoryException right where the byte array is allocated. But I don't think it's the system memory issue. Application memory usage is around 500MB (private working set) and the test machine is 64-bit with 4GB of RAM.

Is it possible that memory used by the byte array or MemoryStream is not released after method finishes? But then, it does not look like this memory is allocated for the process as private working set is only 500MB or so.

What may cause the OutOfMemoryException when creating large byte array (60MB) beside physical memory shortage?

[Edited to add code sample] Source comes from PdfSharp lib

Exception is thrown at line byte[] imageBits = new byte[streamLength]; It indeed looks like LOH fragmentation issue.

/// <summary>
/// Reads images that are returned from GDI+ without color palette.
/// </summary>
/// <param name="components">4 (32bpp RGB), 3 (24bpp RGB, 32bpp ARGB)</param>
/// <param name="bits">8</param>
/// <param name="hasAlpha">true (ARGB), false (RGB)</param>
private void ReadTrueColorMemoryBitmap(int components, int bits, bool hasAlpha)
{
  int pdfVersion = Owner.Version;
  MemoryStream memory = new MemoryStream();
  image.gdiImage.Save(memory, ImageFormat.Bmp);
  int streamLength = (int)memory.Length;

  if (streamLength > 0)
  {
    byte[] imageBits = new byte[streamLength];
    memory.Seek(0, SeekOrigin.Begin);
    memory.Read(imageBits, 0, streamLength);
    memory.Close();

    int height = image.PixelHeight;
    int width = image.PixelWidth;

    if (ReadWord(imageBits, 0) != 0x4d42 || // "BM"
        ReadDWord(imageBits, 2) != streamLength ||
        ReadDWord(imageBits, 14) != 40 || // sizeof BITMAPINFOHEADER
        ReadDWord(imageBits, 18) != width ||
        ReadDWord(imageBits, 22) != height)
    {
      throw new NotImplementedException("ReadTrueColorMemoryBitmap: unsupported format");
    }
    if (ReadWord(imageBits, 26) != 1 ||
      (!hasAlpha && ReadWord(imageBits, 28) != components * bits ||
       hasAlpha && ReadWord(imageBits, 28) != (components + 1) * bits) ||
      ReadDWord(imageBits, 30) != 0)
    {
      throw new NotImplementedException("ReadTrueColorMemoryBitmap: unsupported format #2");
    }

    int nFileOffset = ReadDWord(imageBits, 10);
    int logicalComponents = components;
    if (components == 4)
      logicalComponents = 3;

    byte[] imageData = new byte[components * width * height];

    bool hasMask = false;
    bool hasAlphaMask = false;
    byte[] alphaMask = hasAlpha ? new byte[width * height] : null;
    MonochromeMask mask = hasAlpha ?
      new MonochromeMask(width, height) : null;

    int nOffsetRead = 0;
    if (logicalComponents == 3)
    {
      for (int y = 0; y < height; ++y)
      {
        int nOffsetWrite = 3 * (height - 1 - y) * width;
        int nOffsetWriteAlpha = 0;
        if (hasAlpha)
        {
          mask.StartLine(y);
          nOffsetWriteAlpha = (height - 1 - y) * width;
        }

        for (int x = 0; x < width; ++x)
        {
          imageData[nOffsetWrite] = imageBits[nFileOffset + nOffsetRead + 2];
          imageData[nOffsetWrite + 1] = imageBits[nFileOffset + nOffsetRead + 1];
          imageData[nOffsetWrite + 2] = imageBits[nFileOffset + nOffsetRead];
          if (hasAlpha)
          {
            mask.AddPel(imageBits[nFileOffset + nOffsetRead + 3]);
            alphaMask[nOffsetWriteAlpha] = imageBits[nFileOffset + nOffsetRead + 3];
            if (!hasMask || !hasAlphaMask)
            {
              if (imageBits[nFileOffset + nOffsetRead + 3] != 255)
              {
                hasMask = true;
                if (imageBits[nFileOffset + nOffsetRead + 3] != 0)
                  hasAlphaMask = true;
              }
            }
            ++nOffsetWriteAlpha;
          }
          nOffsetRead += hasAlpha ? 4 : components;
          nOffsetWrite += 3;
        }
        nOffsetRead = 4 * ((nOffsetRead + 3) / 4); // Align to 32 bit boundary
      }
    }
    else if (components == 1)
    {
      // Grayscale
      throw new NotImplementedException("Image format not supported (grayscales).");
    }

    FlateDecode fd = new FlateDecode();
    if (hasMask)
    {
      // monochrome mask is either sufficient or
      // provided for compatibility with older reader versions
      byte[] maskDataCompressed = fd.Encode(mask.MaskData);
      PdfDictionary pdfMask = new PdfDictionary(document);
      pdfMask.Elements.SetName(Keys.Type, "/XObject");
      pdfMask.Elements.SetName(Keys.Subtype, "/Image");

      Owner.irefTable.Add(pdfMask);
      pdfMask.Stream = new PdfStream(maskDataCompressed, pdfMask);
      pdfMask.Elements[Keys.Length] = new PdfInteger(maskDataCompressed.Length);
      pdfMask.Elements[Keys.Filter] = new PdfName("/FlateDecode");
      pdfMask.Elements[Keys.Width] = new PdfInteger(width);
      pdfMask.Elements[Keys.Height] = new PdfInteger(height);
      pdfMask.Elements[Keys.BitsPerComponent] = new PdfInteger(1);
      pdfMask.Elements[Keys.ImageMask] = new PdfBoolean(true);
      Elements[Keys.Mask] = pdfMask.Reference;
    }
    if (hasMask && hasAlphaMask && pdfVersion >= 14)
    {
      // The image provides an alpha mask (requires Arcrobat 5.0 or higher)
      byte[] alphaMaskCompressed = fd.Encode(alphaMask);
      PdfDictionary smask = new PdfDictionary(document);
      smask.Elements.SetName(Keys.Type, "/XObject");
      smask.Elements.SetName(Keys.Subtype, "/Image");

      Owner.irefTable.Add(smask);
      smask.Stream = new PdfStream(alphaMaskCompressed, smask);
      smask.Elements[Keys.Length] = new PdfInteger(alphaMaskCompressed.Length);
      smask.Elements[Keys.Filter] = new PdfName("/FlateDecode");
      smask.Elements[Keys.Width] = new PdfInteger(width);
      smask.Elements[Keys.Height] = new PdfInteger(height);
      smask.Elements[Keys.BitsPerComponent] = new PdfInteger(8);
      smask.Elements[Keys.ColorSpace] = new PdfName("/DeviceGray");
      Elements[Keys.SMask] = smask.Reference;
    }

    byte[] imageDataCompressed = fd.Encode(imageData);

    Stream = new PdfStream(imageDataCompressed, this);
    Elements[Keys.Length] = new PdfInteger(imageDataCompressed.Length);
    Elements[Keys.Filter] = new PdfName("/FlateDecode");
    Elements[Keys.Width] = new PdfInteger(width);
    Elements[Keys.Height] = new PdfInteger(height);
    Elements[Keys.BitsPerComponent] = new PdfInteger(8);
    // TODO: CMYK
    Elements[Keys.ColorSpace] = new PdfName("/DeviceRGB");
    if (image.Interpolate)
      Elements[Keys.Interpolate] = PdfBoolean.True;
  }
}

Answer 1

I hope you are using MemoryStream.GetBuffer() and that you are not copying to a new array.

Your main problem is not a direct lack of memory but fragmentation of the LOH. That can be a tricky problem, the main issue is allocating large buffers of varying size. Items on the LOH are collected but not compacted.

Solutions might be:

• first make sure you are not blocking anything from being collected. use a profiler.
• try to reuse your buffer(s).
• round allocations up to a set of fixed numbers.

The last 2 both require you to work with oversized arrays, may take some work.

Answer 2

Disposing has been suggested, but only for the MemoryStream, this will do you nothing. Also, GC.Collect has been suggested. This may not help, as it seems your memory is not substantially growing. Be careful, calling GC.Collect can be an expensive operation.

Fragmentation

It really looks like you're hitting the infamous Large Object Heap fragmentation issue. This may well be caused by often allocating and freeing chunks of 60MB of memory. If the LOH gets fragmented, it stays fragmented. This is a major issue with long-running .NET applications and a reason why ASP.NET is often configured to restart at intervals.

• Large object fragmentation (Stackoverflow)
• Excellent explanation and easy to follow code on Codeproject.
• Reported to Microsoft, but nothing will be done yet.

Prevent OutOfMemoryException

See the above CodeProject article on how to do this. The trick is, use MemoryFailPoint and catch the InsufficientMemoryException. This way, you can gracefully degrade and your application will not become unstable.

Possible general solution

Make sure your large objects live as long as possible. Reuse the buffer. Allocate it once with sufficient size and zero the buffer when you need it again. This way you won't run into any other memory issues. When your objects stay below 85k in size, they generally don't run into the LOH and won't clutter.

64 bit machines should not have this issue

EDIT: According to this post (workaround tab) and this post (see open comment), this issue should not appear on 64 bit machines. Since you say you run your code on 64 bit machines, perhaps you compiled with the configuration set to x86?