I hit an OutOfMemoryException with List<string> - is this the limit or am I missing something?

Question

Given the opportunity to rewrite, I would, but anyway, the code as it stands:

List<string> foobar;

Then we add a bunch of strings to foobar.

At count=16777216, we hit an out of memory limit.

My understanding is that each string would be a different size. Indeed looking at the data (not my data), most are 2 or 3 characters.

what is the max limit of data into list in c#? indicates that the max limit is:

The maximum number of elements that can be stored in the current implementation of List is, theoretically, Int32.MaxValue - just over 2 billion.

However:

In the current Microsoft implementation of the CLR there's a 2GB maximum object size limit. (It's possible that other implementations, for example Mono, don't have this restriction.)

In my example, I have, what, 16 million results * a few bytes? Task manager shows about a gig being used, but I have 8 gigs of RAM.

16777216 (2^24) seems like a fairly specific value - suspiciously like a limit, but I can't find any documentation anywhere to a) back this up or b) find a way around it?

Any help would be appreciated.

---

Some code:

List<string> returnList = new List<string>();
SqlDataReader dr; //  executes a read on a database, have removed that part as that bit works fine

  if (dr.HasRows)
  {
      while (dr.Read())
      {
          returnList.Add(dr.GetString(0).Trim());
      }
  }

That's the simplified form, I now have some try/catch for the OOM Exception, but this is the actual code that's giving me grief.

Answer 1

If you're trying to use very large lists in 64 bit environments you need to enable large objects in the application configuration.

http://msdn.microsoft.com/en-us/library/hh285054.aspx

The OOM is likely due to the way Lists/ArrayLists allocate memory, which I believe is each time their boundary is reached, they attempt to double in size. The list cannot double from 2^24. You could theoretically maximize your list size by pre-specifying a size. (I.e. 2GB)

Answer 2

I've posted what I exactly did here, worth giving it a go. Again steps are:

1. On each iteration query portion of data using an stored proc
2. Transfer them

3. Move to the next portion

   List<string> returnList;
   int index = 0;
   SqlCommand cmd = new SqlCommand("ExampleStoredProc", conn);
   cmd.CommandType = CommandType.StoredProcedure;
   while (true)
   {
       cmd.Parameters.Add(
           new SqlParameter("@index", index));
       SqlDataReader dr = cmd.ExecuteReader();
       if (dr.HasRows)
       {
           returnList = new List<string>();
           returnList.Add(dr.GetString(0).Trim());
           //transfer data here
       }
       else
       {
           break;
       }
       index++;
   }
   

and the stored proc should be something like this:

CREATE PROCEDURE ExampleStoredProc
    @index INT
AS
BEGIN
    SELECT * 
    FROM  veryBigTable
    WHERE Id >= (@index *1000) AND Id < ((@index + 1) * 1000)
END
GO

I'll definitely work no matter how many records you have, just the more data you have, longer it'll take to finish.