Reading large XML documents in .net

Question

I need to read large xml using .net files which can easily be several GB of size.

I tried to use XDocument, but it just throws an System.OutOfMemoryException when I try to load the document.

What is the most performant way to read XML files of large size?

Answer 1

You basically have to use the "pull" model here - XmlReader and friends. That will allow you to stream the document rather than loading it all into memory in one go.

Note that if you know that you're at the start of a "small enough" element, you can create an XElement from an XmlReader, deal with that using the glory of LINQ to XML, and then move onto the next element.

Answer 2

The following page makes an interesting read, providing a means to mine data from XML file without loading it in memory. It allows you to combine the speed of XmlReader with the flexibility of Linq:

http://msdn.microsoft.com/en-us/library/bb387035.aspx

And quite an interesting article based on this technique:

http://blogs.msdn.com/b/xmlteam/archive/2007/03/24/streaming-with-linq-to-xml-part-2.aspx