Processing large XML files using .NET 3.5

Question

What is the "recommended" approach for processing very large XML files in .NET 3.5?

For writing, I want to generate an element at a time then append to a file.

For reading, I would likewise want to read an element at a time (in the same order as written).

I have a few ideas how to do it using strings and File.Append, but does .NET 3.5 provide XML Api's for dealing with arbitrarily large XML files?

Answer 1

Without going into specifics this isn't easy to answer. .NET offers different methods to process XML files:

• XmlDocument creates a DOM, supports XPath queries but loads the entire XML file into memory.
• XElement/XDocument has support for LINQ and also reads the entire XML file into memory.
• XmlReader is a forward-only reader. It does not read the entire file into memory.
• XmlWriter is just like the XmlReader, except for writing

Based on what you say an XmlReader/XmlWriter combination seems like the best approach.

Answer 2

As Dirk said, using an XmlWriter/XmlReader combo sounds like the best approach. It can be very lengthy and if your XML file is fairly complex it gets very unwieldy. I had to do something similar recently with some strict memory constraints. My SO question might come in handy.

But personally, I found this method here on MSDN blogs to be very easy to implement and it neatly handles appending to the end of the XML file without fragments.