I'm constantly hitting an OutOfMemoryException
inside a method that creates and processes some byte arrays. The code looks like this:
When this method is called like 20-30 times I get OutOfMemoryException
right where the byte array is allocated. But I don't think it's the system memory issue. Application memory usage is around 500MB (private working set) and the test machine is 64-bit with 4GB of RAM.
Is it possible that memory used by the byte array or MemoryStream
is not released after method finishes? But then, it does not look like this memory is allocated for the process as private working set is only 500MB or so.
What may cause the OutOfMemoryException
when creating large byte array (60MB) beside physical memory shortage?
[Edited to add code sample] Source comes from PdfSharp lib
Exception is thrown at line byte[] imageBits = new byte[streamLength];
It indeed looks like LOH fragmentation issue.
/// <summary>
/// Reads images that are returned from GDI+ without color palette.
/// </summary>
/// <param name="components">4 (32bpp RGB), 3 (24bpp RGB, 32bpp ARGB)</param>
/// <param name="bits">8</param>
/// <param name="hasAlpha">true (ARGB), false (RGB)</param>
private void ReadTrueColorMemoryBitmap(int components, int bits, bool hasAlpha)
{
int pdfVersion = Owner.Version;
MemoryStream memory = new MemoryStream();
image.gdiImage.Save(memory, ImageFormat.Bmp);
int streamLength = (int)memory.Length;
if (streamLength > 0)
{
byte[] imageBits = new byte[streamLength];
memory.Seek(0, SeekOrigin.Begin);
memory.Read(imageBits, 0, streamLength);
memory.Close();
int height = image.PixelHeight;
int width = image.PixelWidth;
if (ReadWord(imageBits, 0) != 0x4d42 || // "BM"
ReadDWord(imageBits, 2) != streamLength ||
ReadDWord(imageBits, 14) != 40 || // sizeof BITMAPINFOHEADER
ReadDWord(imageBits, 18) != width ||
ReadDWord(imageBits, 22) != height)
{
throw new NotImplementedException("ReadTrueColorMemoryBitmap: unsupported format");
}
if (ReadWord(imageBits, 26) != 1 ||
(!hasAlpha && ReadWord(imageBits, 28) != components * bits ||
hasAlpha && ReadWord(imageBits, 28) != (components + 1) * bits) ||
ReadDWord(imageBits, 30) != 0)
{
throw new NotImplementedException("ReadTrueColorMemoryBitmap: unsupported format #2");
}
int nFileOffset = ReadDWord(imageBits, 10);
int logicalComponents = components;
if (components == 4)
logicalComponents = 3;
byte[] imageData = new byte[components * width * height];
bool hasMask = false;
bool hasAlphaMask = false;
byte[] alphaMask = hasAlpha ? new byte[width * height] : null;
MonochromeMask mask = hasAlpha ?
new MonochromeMask(width, height) : null;
int nOffsetRead = 0;
if (logicalComponents == 3)
{
for (int y = 0; y < height; ++y)
{
int nOffsetWrite = 3 * (height - 1 - y) * width;
int nOffsetWriteAlpha = 0;
if (hasAlpha)
{
mask.StartLine(y);
nOffsetWriteAlpha = (height - 1 - y) * width;
}
for (int x = 0; x < width; ++x)
{
imageData[nOffsetWrite] = imageBits[nFileOffset + nOffsetRead + 2];
imageData[nOffsetWrite + 1] = imageBits[nFileOffset + nOffsetRead + 1];
imageData[nOffsetWrite + 2] = imageBits[nFileOffset + nOffsetRead];
if (hasAlpha)
{
mask.AddPel(imageBits[nFileOffset + nOffsetRead + 3]);
alphaMask[nOffsetWriteAlpha] = imageBits[nFileOffset + nOffsetRead + 3];
if (!hasMask || !hasAlphaMask)
{
if (imageBits[nFileOffset + nOffsetRead + 3] != 255)
{
hasMask = true;
if (imageBits[nFileOffset + nOffsetRead + 3] != 0)
hasAlphaMask = true;
}
}
++nOffsetWriteAlpha;
}
nOffsetRead += hasAlpha ? 4 : components;
nOffsetWrite += 3;
}
nOffsetRead = 4 * ((nOffsetRead + 3) / 4); // Align to 32 bit boundary
}
}
else if (components == 1)
{
// Grayscale
throw new NotImplementedException("Image format not supported (grayscales).");
}
FlateDecode fd = new FlateDecode();
if (hasMask)
{
// monochrome mask is either sufficient or
// provided for compatibility with older reader versions
byte[] maskDataCompressed = fd.Encode(mask.MaskData);
PdfDictionary pdfMask = new PdfDictionary(document);
pdfMask.Elements.SetName(Keys.Type, "/XObject");
pdfMask.Elements.SetName(Keys.Subtype, "/Image");
Owner.irefTable.Add(pdfMask);
pdfMask.Stream = new PdfStream(maskDataCompressed, pdfMask);
pdfMask.Elements[Keys.Length] = new PdfInteger(maskDataCompressed.Length);
pdfMask.Elements[Keys.Filter] = new PdfName("/FlateDecode");
pdfMask.Elements[Keys.Width] = new PdfInteger(width);
pdfMask.Elements[Keys.Height] = new PdfInteger(height);
pdfMask.Elements[Keys.BitsPerComponent] = new PdfInteger(1);
pdfMask.Elements[Keys.ImageMask] = new PdfBoolean(true);
Elements[Keys.Mask] = pdfMask.Reference;
}
if (hasMask && hasAlphaMask && pdfVersion >= 14)
{
// The image provides an alpha mask (requires Arcrobat 5.0 or higher)
byte[] alphaMaskCompressed = fd.Encode(alphaMask);
PdfDictionary smask = new PdfDictionary(document);
smask.Elements.SetName(Keys.Type, "/XObject");
smask.Elements.SetName(Keys.Subtype, "/Image");
Owner.irefTable.Add(smask);
smask.Stream = new PdfStream(alphaMaskCompressed, smask);
smask.Elements[Keys.Length] = new PdfInteger(alphaMaskCompressed.Length);
smask.Elements[Keys.Filter] = new PdfName("/FlateDecode");
smask.Elements[Keys.Width] = new PdfInteger(width);
smask.Elements[Keys.Height] = new PdfInteger(height);
smask.Elements[Keys.BitsPerComponent] = new PdfInteger(8);
smask.Elements[Keys.ColorSpace] = new PdfName("/DeviceGray");
Elements[Keys.SMask] = smask.Reference;
}
byte[] imageDataCompressed = fd.Encode(imageData);
Stream = new PdfStream(imageDataCompressed, this);
Elements[Keys.Length] = new PdfInteger(imageDataCompressed.Length);
Elements[Keys.Filter] = new PdfName("/FlateDecode");
Elements[Keys.Width] = new PdfInteger(width);
Elements[Keys.Height] = new PdfInteger(height);
Elements[Keys.BitsPerComponent] = new PdfInteger(8);
// TODO: CMYK
Elements[Keys.ColorSpace] = new PdfName("/DeviceRGB");
if (image.Interpolate)
Elements[Keys.Interpolate] = PdfBoolean.True;
}
}
I hope you are using MemoryStream.GetBuffer()
and that you are not copying to a new array.
Your main problem is not a direct lack of memory but fragmentation of the LOH. That can be a tricky problem, the main issue is allocating large buffers of varying size. Items on the LOH are collected but not compacted.
Solutions might be:
The last 2 both require you to work with oversized arrays, may take some work.
Disposing has been suggested, but only for the MemoryStream
, this will do you nothing. Also, GC.Collect
has been suggested. This may not help, as it seems your memory is not substantially growing. Be careful, calling GC.Collect can be an expensive operation.
It really looks like you're hitting the infamous Large Object Heap fragmentation issue. This may well be caused by often allocating and freeing chunks of 60MB of memory. If the LOH gets fragmented, it stays fragmented. This is a major issue with long-running .NET applications and a reason why ASP.NET is often configured to restart at intervals.
See the above CodeProject article on how to do this. The trick is, use MemoryFailPoint
and catch the InsufficientMemoryException
. This way, you can gracefully degrade and your application will not become unstable.
Make sure your large objects live as long as possible. Reuse the buffer. Allocate it once with sufficient size and zero the buffer when you need it again. This way you won't run into any other memory issues. When your objects stay below 85k in size, they generally don't run into the LOH and won't clutter.
EDIT: According to this post (workaround tab) and this post (see open comment), this issue should not appear on 64 bit machines. Since you say you run your code on 64 bit machines, perhaps you compiled with the configuration set to x86?
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With