StackOverflowException on initialising large list

Question

I'm doing the following:

public static class DataHelper
{
  public static List<Seller> Sellers = new List<Seller> {
    {new Seller {Name = "Foo", Location = new LatLng {Lat = 12, Lng = 2}, Address = new Address {Line1 = "blah", City = "cokesville"}, Country = "UK"},
    //plus 3500 more Sellers
  };
}

When I access DataHelper.Sellers from inside my MVC website, I get a StackOverflowException. When I debug with Visual Studio, the stack has only half a dozen frames and there is not the usual obvious sign of a stack overflow (i.e. no repeated method calls).

The app call can be as simple as this to provoke the exception:

public ActionResult GetSellers()
{
  var sellers = DataHelper.Sellers;
  return Content("done");
}

Extra info:

• when I run the same code from within a unit test it is fine
• if I remove half the sellers (either top half or bottom half), it is fine in the web app, so it is not a problem with any specific seller
• I have tried changing Sellers to a property and initialising list on first call - no help
• I also tried adding half to one list then half to another and combining the 2 - again no help

I will be very impressed by the correct answer to this one!

Answer 1

This is caused by the fact that, effectively, your inline list initialisation is too large for the stack - see this very related question over on the msdn forums where the scenario is nigh-on identical.

A method in .Net has both a stack depth and size. A StackOverflowException is caused not just by the number of calls on the stack, but the overall size of each method's allocation of memory in the stack. In this case, your method is just far too large - and that's caused by the number of local variables.

By way of example, consider the following code:

    public class Foo
    {
            public int Bar { get; set;}
    }
    public Foo[] GetInts()
    {
        return new Foo[] { new Foo() { Bar = 1 }, new Foo() { Bar = 2 }, 
          new Foo() { Bar = 3 }, new Foo() { Bar = 4 }, new Foo() { Bar = 5 } };
    }

Now look at the lead-in IL of that method when compiled (this is a release build, too):

.maxstack 4
.locals init (
    [0] class SomeExample/Foo '<>g__initLocal0',
    [1] class SomeExample/Foo '<>g__initLocal1',
    [2] class SomeExample/Foo '<>g__initLocal2',
    [3] class SomeExample/Foo '<>g__initLocal3',
    [4] class SomeExample/Foo '<>g__initLocal4',
    [5] class SomeExample/Foo[] CS$0$0000
)

Note - the actual bit before the /, i.e. SomeExample will depend on the namespace and class in which the method and nested class are defined - I've had to edit a few times to remove the typenames from some in progress code that I'm writing at work!

Why all those locals? Because of the way that inline initialisation is performed. Each object is newed and stored in a 'hidden' local, this is required so that the property assignments can be performed on the inline initialisations of each Foo (the object instance is required to generate the property set for Bar). This also demonstrates how inline initialisation is just some C# syntactic sugar.

In your case, it's these locals that's causing the stack to blow (there are at least a few thousand of them just for the top-level objects - but you also have nested initialisers as well).

The C# compiler could alternatively pre-load the number of references required for each on to the stack (popping each one off for each property assignment) but then that is abusing the stack where the use of locals will perform much better.

It could also use a single local, since each is merely written-too and then stored in the list by array index, the local is never needed again. That might be one for the C# team to consider - Eric Lippert may have a few thoughts about this if he stumbles on this thread.

Now, this examination also gives us a potential route around this use of locals for your very massive method: use an iterator:

public Foo[] GetInts()
{
    return GetIntsHelper().ToArray();
}

private IEnumerable<Foo> GetIntsHelper()
{
    yield return new Foo() { Bar = 1 };
    yield return new Foo() { Bar = 2 };
    yield return new Foo() { Bar = 3 };
    yield return new Foo() { Bar = 4 };
    yield return new Foo() { Bar = 5 };
}

Now the IL for GetInts() now simply has .maxstack 8 at the head, and no locals. Looking at the iterator function GetIntsHelper() we have:

.maxstack 2
.locals init (
    [0] class SomeExample/'<GetIntsHelper>d__5'
)

So now we've stopped using all those locals in those methods...

But...

Look at the class SomeExample/'<GetIntsHelper>d__5', which has been automatically generated by the compiler - and we see that the locals are still there - they've just been promoted to fields on that class:

.field public class SomeExample/Foo '<>g__initLocal0'
.field public class SomeExample/Foo '<>g__initLocal1'
.field public class SomeExample/Foo '<>g__initLocal2'
.field public class SomeExample/Foo '<>g__initLocal3'
.field public class SomeExample/Foo '<>g__initLocal4'

So the question is - will the creation of that object also blow the stack if applied to your scenario? Probably not, because in memory it should be like trying to initialise large array - where million-element arrays are quite acceptable (assuming enough memory in practise).

So - you might be able to fix your code quite simply by moving over to using an IEnumerable method that yields each element.

But best practise says that if you absolutely have to have this statically defined - consider adding the data to an embedded resource or file on disk (XML and Linq to XML might be a good choice) and then loading it from there on demand.

Better still - stick it in a database :)