List<object[]> using more memory than DataTable?

Question

I've always believed that DataTable would consume more memory than a generic List. I am testing loading a DataTable and loading a List from a SQL Server query. In this case, the DataTable is consuming less memory. I'm getting the top 2000 rows and there are 134 fields per row. One binary field and the rest are standard varchar, int, bit and so on.

How in the world could a DataTable with all it's overhead consume less memory than List? The GC is reporting approx 4mb with DataTable and 5mb with list.

I did test with a few of the NorthWind tables and the list was marginally smaller in those cases.

private void GetMemory()
    {
        label1.Text = string.Format("{0:0.00} MB", GC.GetTotalMemory(true) / 1024.0 / 1024.0);            
    }
    private void DataTableButton_Click(object sender, EventArgs e)
    {
        var conn = new SqlConnection(@"Initial Catalog=ADatabase;Data Source=AServer;Integrated Security=True");
        conn.Open();

        var cmd = new SqlCommand("SELECT TOP 2000 * FROM AManyColumnedTable", conn);

        var r = cmd.ExecuteReader();
        _rows = new List<object[]>();

        //uses more memory
        object[] a = null;
        while (r.Read())
        {
            a = new object[r.FieldCount];
            r.GetValues(a);
            _rows.Add(a);
        }
        //uses less memory
        //_table = new DataTable("TheTable");
        //_table.Load(r);

        r.Close();
        conn.Close();
        GetMemory();
    }

Answer 1

It's an illusion. Somehow the GC doesn't give you the right value, perhaps because there are objects that are not collected yet. Maybe loading a data table causes more intermediate objects, which causes another garbage collection.

Your list of arrays store the data in the same manner as the data table, but without the overhead of the extra information that the data table and each data row contains.

If you want it to be more memory efficient, you should create a class for a data record, that way you can store the value types as plain values instead of boxed in objects. An int for example uses 20 bytes when it's boxed but only 4 as a plain member variable.

Answer 2

Just out of curiosity, try calling TrimExcess on the List and then check the memory. I doubt this will make a huge difference, but I suspect it will make some difference.

Basically, a List grows in batches. It doesn't grow by one every time you add an object, rather it grows by a certain amount, and then if it needs more, it allocates more. I'm curious if there's some extra space in your list that you're not using.