C# optimize memory usage: How to free memory claimed by DataTable

Question

Currently I'm optimizing memory-usage of an huge batch-program. The most memory is used by different DataTables. For examle, my DataTable dataTable is using approx 260MB.

Like suggestet in the accepted answer of the thread "What is the memory overhead of storing data in a .NET DataTable?" I'm trying to move relevant data out of the DataTable. This is my code:

GC.Collect(); // force the garbage collector to free memory
// First stop point - Total process memory (taskmanager) = 900 MB
List<ExpandoObject> expandoList = new List<ExpandoObject>();
foreach (DataRow dataRow in dataTable.Rows)
{
    dynamic expandoItem = new ExpandoObject();
    expandoItem.FieldName = dataRow["FieldName"].ToString();
    expandoList.Add(expandoItem);
}
// Second stop point - Total process memory (taskmanager) = 1055 MB
dataTable.Clear();
dataTable.Dispose();
dataTable = null;
GC.Collect(); // force the garbage collector to free memory
// Third stop point - Total process memory (taskmanager) = 1081 MB (wtf? even more!)

I'm using Clear, Dispose and setting to null because it is suggested in following thread: Datatable.Dispose() will make it remove from memory?

See the stop points comments to see memory usage at that points. I also tried it with using (DataTable dataTable = ...) but the results were the same.

What am I doing wrong? And maybe is there a better way to minify data from DataTable?

Answer 1

I finally found this thread about the memory-usage: .NET EXE memory footprint

The accepted answer says following:

TaskManager should not be used to measure the memory footprint of a .NET application.

When a .NET application starts, it asks the OS for a chunk of memory which it then segments to become the managed heap, stack, and large object heap. It is this total chunk of memory that TaskManager is reporting, which may or may not be completely used by .NET. Once a .NET application is given a chunk of memory, it will not release it until asked by the OS, which will only happen with the OS determines a need for more memory resources.

If you want to measure memory allocations, you need to look at the various performance > monitor (PerfMon) counters.

In short: the task manager shows the reserved memory instead of the really used. This means setting the DataTable to null works fine.

It is possible to use the GarbageCollector to get the really used memory with following code:

long memoryInMB = GC.GetTotalMemory(forceFullCollection: true) / 1024 / 1024;

I tried this with my code and the removal of the datatable reduced the used total memory by 28MB. Not as much to be worth the effort to extract data from DataTable to another container :-/

I hope this can help some other people who have the same problem.

For more info's about GC, Dispose and Finalize you should absolutely check out this answer.