Very large String into Byte Array

Question

I have an XML file which result from the export of a database (Oracle 11g Unicode) table. This table has a BLOB field which represent a file. The file could be a very large one.

So in the case where I have a very large file a get in the XML a very large string representation of that file.

I have to get the bytes of this string in order to insert the file in another database instance.

A this point the XML is charged and i have then a string representing the file.

What I've done is this:

Encoding.Unicode.GetBytes(stringFileRepresentation);

But I'm getting an OutOfMemoryException.

If I do this:

Encoding.Unicode.GetBytes(stringFileRepresentation.ToCharArray());

I get also an OutOfMemoryException.

I tried too to do this before decoding the string:

var chars = stringFileRepresentation.ToCharArray();
Encoding.Unicode.GetBytes(chars);

And I getting the OutOfMemoryException when calling ToCharArray().

So I guess is a problem when processing the string.

Then I'm trying the following method that I found here event if I'm not sure I have to conservate the encoding of the string:

byte[] bytes = new byte[str.Length * sizeof(char)];
Buffer.BlockCopy(str.ToCharArray(), 0, bytes, 0, bytes.Length);

But I'm getting also an OutOfMemoryException in instantiating the bytes variable.

Now, I ran OutOfOptions and I don't know what to do.

Answer 1

Since you've already got the original full string in memory, you can use a StringReader to buffer through it:

This will get the text into a file. You can use a similar technique to write to a different stream instead of a file.

using (var sr = new StringReader(fileContents))
{
    using (var outputWriter = new StreamWriter(@"C:\temp\output.txt"))
    {
        char[] buffer = new char[10];
        int numChars;
        while ((numChars = sr.ReadBlock(buffer, 0, buffer.Length)) > 0)
        {
            outputWriter.Write(buffer, 0, numChars);
        }
    }
}

EDIT

Writing to something other than a file is pretty similar - for example, suppose you wanted to write directly to a stream (doesn't matter what kind of stream - could be a MemoryStream, HttpResponse stream, FileStream, etc.):

using (var sr = new StringReader(fileContents))
{
    using (var outputStream = GetSomeStreamFromSomewhere())
    {
        char[] buffer = new char[10];
        int numChars;
        while ((numChars = sr.ReadBlock(buffer, 0, buffer.Length)) > 0)
        {
            char[] temp = new char[numChars];
            Array.Copy(buffer, 0, temp, 0, numChars);
            byte[] byteBuffer = Encoding.UTF8.GetBytes(temp);
            outputStream.Write(byteBuffer, 0, byteBuffer.Length);
        }
    }
}