StringBuilder.ToString() throws OutOfMemoryException

Question

I have a created a StringBuilder of length "132370292", when I try to get the string using the ToString() method it throws OutOfMemoryException.

StringBuilder SB = new StringBuilder();

for(int i =0; i<=5000; i++)
{
    SB.Append("Some Junk Data for testing. My Actual Data is created from different sources by Appending to the String Builder.");
}

try
{
    string str = SB.ToString(); // Throws OOM mostly
    Console.WriteLine("String Created Successfully");
}
catch(OutOfMemoryException ex)
{
    StreamWriter sw = new StreamWriter(@"c:\memo.txt", true);
    sw.Write(SB.ToString()); //Always writes to the file without any error
    Console.WriteLine("Written to File Successfully");
}

What is the reason for the OOM while creating a new string and why it doesn't throw OOM while writing to a file?

Machine Details: 64-bit, Windows-7, 2GB RAM, .NET version 2.0

Answer 1

What is the reason for the OOM while creating a new string

Because you're running out of memory - or at least, the CLR can't allocate an object with the size you've requested. It's really that simple. If you want to avoid the errors, don't try to create strings that don't fit into memory. Note that even if you have a lot of memory, and even if you're running a 64-bit CLR, there are limits to the size of objects that can be created.

and why it doesn't throw OOM while writing to a file ?

Because you have more disk space than memory.

I'm pretty sure the code isn't exactly as you're describing though. This line would fail to compile:

sw.write(SB.ToString());

... because the method is Write rather than write. And if you're actually calling SB.ToString(), then that's just as likely to fail as str = SB.ToString().

It seems more likely that you're actually writing to the file in a streaming fashion, e.g.

using (var writer = File.CreateText(...))
{
    for (int i = 0; i < 5000; i++)
    {
        writer.Write(mytext);
    }
}

That way you never need to have huge amounts of text in memory - it just writes it to disk as it goes, possibly with some buffering, but not enough to cause memory issues.

Answer 2

Workaround: Suppose you would want to write a big string stored in StringBuilder to a StreamWriter, I would do a write this way to avoid SB.ToString's OOM exception. But if your OOM exception is due to StringBuilder's content add itself, you should work on that.

public const int CHUNK_STRING_LENGTH = 30000;
while (SB.Length > CHUNK_STRING_LENGTH )
{
    sw.Write(SB.ToString(0, CHUNK_STRING_LENGTH ));
    SB.Remove(0, CHUNK_STRING_LENGTH );
}
sw.Write(SB);

Answer 3

You have to remember that strings in .NET are stored in memory in 16-bit unicode. This means string of length 132370292 will reqire 260MB of RAM.

Furthermore, while executing

string str = SB.ToString();

you are creating a COPY of your string (another 260MB).

Keep in mind that each process have its own RAM limit so OutOfMemoryException can be thrown even if you have some free RAM left.