XSLT transformation on Large XML files with C#

Question

I have some very large XML files (800 MB to 1.5 GB). I need to apply XSLT on that. I am able to read it XMLTextReader. When i applied XSLT transformation, get SystemOutOfMemory Exception.

My code looks like;

static void Main(string[] args)
{
    XDocument newTree = new XDocument();
    XmlTextReader oReader = new XmlTextReader(@"C:\Projects\myxml.xml");


    using (XmlWriter writer = newTree.CreateWriter())
    {
        XslCompiledTransform oTransform = new XslCompiledTransform();
        oTransform.Load(@"C:\Projects\myXSLT.xsl");
        oTransform.Transform(oReader, writer);
    }
    Console.WriteLine(newTree);
}

Thanks in advance. It is very urgent. If I don't get any solution, I need to split XML into smaller XML and do transformation.

Answer 1

XSLT uses XPath and this requires that the whole XML document be maintained in memory. Thus the problem of insufficient memory is by definition.

There are simle rules to approximate how much memory is needed and one of them says 5 * text-size.

So, for a "typical 1.5GB XML file" 8GB RAM may be sufficient.

Either split the document into smaller parts or wait for an implementation of XSLT 2.1, which defines special streaming instructions. In the meantime one may use the latest (commercial) version of Saxon, which implements extensions for streaming and successful processing of 64GB document has been reported on twitter.

Answer 2

we are facing a similar problem. The solution we came uo with was to not use xslt for this case, and instead use Linq to Xml transformations while stteaming the data. You can leverage the c# yield keyword to iterate through an xml stream and tackle the file piecemeal this way. See streaming with linq to xml

the nature of xslt requires the xml to be loaded into memory. what needs to occur is you need to break down the large file into more managable pieces. if you use the xml streaming technique, you can break the document up into sub elements which you can then individually apply the xslt to. you may have to rewrite the xslt to accomodate this behavior.

Aside from this, the only other option is to throw more hardware at it, but this might even require an operating system upgrade depending on RAM limitations...