Weird exceptions compiled dynamically built expression

Question

I am creating and compiling an expression with the System.Ling.Expressions API. Compilation works fine, but in some cases I get unexplained NullReferenceExceptions or even System.Security.Verification exceptions when running the compiled lambda. For reference, the purpose of this project is to create and compile a custom serializer function for a .NET type.

The following is the DebugInfo for an expression that throws a NullReferenceException:

.Lambda #Lambda1<System.Action`2[IO.IWriter,<>f__AnonymousType1`2[System.Int32[],System.Int32]]>(
    IO.IWriter $writer,
    <>f__AnonymousType1`2[System.Int32[],System.Int32] $t) {
    .Block() {
        .Invoke (.Lambda #Lambda2<System.Action`2[IO.IWriter,System.Int32[]]>)(
            $writer,
            $t.a);
        .Invoke (.Lambda #Lambda3<System.Action`2[IO.IWriter,System.Int32]>)(
            $writer,
            $t.b)
    }
}

.Lambda #Lambda2<System.Action`2[IO.IWriter,System.Int32[]]>(
    IO.IWriter $writer,
    System.Int32[] $t) {
    .Block() {
        .Invoke (.Lambda #Lambda4<System.Action`2[IO.IWriter,System.Int32]>)(
            $writer,
            .Call System.Linq.Enumerable.Count((System.Collections.Generic.IEnumerable`1[System.Int32])$t));
        .Call IO.SerializerHelpers.WriteCollectionElements(
            (System.Collections.Generic.IEnumerable`1[System.Int32])$t,
            $writer,
            .Lambda #Lambda3<System.Action`2[IO.IWriter,System.Int32]>)
    }
}

.Lambda #Lambda3<System.Action`2[IO.IWriter,System.Int32]>(
    IO.IWriter $writer,
    System.Int32 $t) {
    .Call $writer.WriteInt($t)
}

.Lambda #Lambda4<System.Action`2[IO.IWriter,System.Int32]>(
    IO.IWriter $w,
    System.Int32 $count) {
    .Call $w.BeginWritingCollection($count)
}

The exception is thrown within the call to #Lambda3, which is called repeatedly from WriteCollectionElements. The implementation of WriteCollectionElements is as follows:

static void WriteCollectionElements<T>(IEnumerable<T> collection, IWriter writer, Action<IWriter, T> writeAction)
        {
            foreach (var element in collection)
            {
                writeAction(writer, element);
            }
        }

From debugging inside this function, I have determined that collection, writer, writeAction, and element are all non-null when the exception is thrown. The argument that I am passing to the compiled lambda is:

new { a = new[] { 20, 10 }, b = 2 }

Also strange is that if I remove the b property and re-generate my serializer function, everything works fine. In this case the DebugInfo for the serializer is:

.Lambda #Lambda1<System.Action`2[IO.IWriter,<>f__AnonymousType5`1[System.Int32[]]]>(
    IO.IWriter $writer,
    <>f__AnonymousType5`1[System.Int32[]] $t) {
    .Block() {
        .Invoke (.Lambda #Lambda2<System.Action`2[IO.IWriter,System.Int32[]]>)(
            $writer,
            $t.a)
    }
}

.Lambda #Lambda2<System.Action`2[IO.IWriter,System.Int32[]]>(
    IO.IWriter $writer,
    System.Int32[] $t) {
    .Block() {
        .Invoke (.Lambda #Lambda3<System.Action`2[IO.IWriter,System.Int32]>)(
            $writer,
            .Call System.Linq.Enumerable.Count((System.Collections.Generic.IEnumerable`1[System.Int32])$t));
        .Call IO.SerializerHelpers.WriteCollectionElements(
            (System.Collections.Generic.IEnumerable`1[System.Int32])$t,
            $writer,
            .Lambda #Lambda4<System.Action`2[IO.IWriter,System.Int32]>)
    }
}

.Lambda #Lambda3<System.Action`2[IO.IWriter,System.Int32]>(
    IO.IWriter $w,
    System.Int32 $count) {
    .Call $w.BeginWritingCollection($count)
}

.Lambda #Lambda4<System.Action`2[IO.IWriter,System.Int32]>(
    IO.IWriter $writer,
    System.Int32 $t) {
    .Call $writer.WriteInt($t)
}

I am running .NET Framework 4 (at least that's my build target) on Windows 7, VS Express C# 2010.

Does anyone have any idea what might be going wrong or next steps for trying to debug? I'm happy to post more information if it will help.

EDIT: I have since (to my knowledge) found my way around this bug, although I am no closer to understanding why it happens. In the code that generates the Expressions I've posted above, I had the following:

MethodInfo writeCollectionElementsMethod = // the methodInfo for WriteCollectionElements with .MakeGenericMethod() called with typeof(T)
Expression<Action<IWriter, T> writeActionExpression = // I created this expression separately
ParameterExpression writerParameter, enumerableTParameter = // parameters of type IWriter and IEnumerable<T>, respectively

// make an expression to invoke the method
var methodCallExpression = Expression.Call(
    instance: null, // static
    method: writeCollectionElementsMethod,
    arguments: new[] {
        enumerableTParameter,
        writerParameter,
        // passing in this expression correctly would produce the weird error in some cases as described above
        writeActionExpression
    }
);

// make an expression to invoke the method
var methodCallExpressionV2 = Expression.Call(
    instance: null, // static
    method: writeCollectionElementsMethod,
    arguments: new[] {
        enumerableTParameter,
        writerParameter,
        // this did not cause the bug
        Expression.Constant(writeActionExpression.Compile())
    }
);

However, I didn't like compiling every expression separately, so I ended up doing away with the WriteCollectionElements function altogether and just creating the foreach loop dynamically via Expression.Loop, Expression.Break, etc.

Thus, I am no longer blocked, but still very curious.

Answer 1

If you build the Actions manually in C# resharper complains about Lambda1 and Lambda2 implicitly capturing variables in the clousure

Action<IWriter, int> lambda4 = ( (IWriter writer, int length) => writer.BeginWritingCollection(length));
Action<IWriter, int> lambda3 = ( (IWriter writer, int value) => writer.WriteInt(value));
Action<IWriter, int[]> lambda2 = ( (IWriter writer, int[] value) =>
   {
      lambda4(writer, ((IEnumerable<int>) value).Count());
      WriteCollectionElements((IEnumerable<int>)value, writer, lambda3);
   });
Action<IWriter, TheData> lambda1 = ((writer, data) =>
   {
      lambda2(writer, data.a);
      lambda3(writer, data.b);
   });
class TheData { int[] a; int b; }

In this case resharper states :
"Implicitly captured closure : lambda2" on the lambda2 expression
"Implicitly captured closure : lambda4" on the lambda1 expression

The explanation for this is here and here. If the line to WriteCollectionElements is removed, the warning disappears. Essentially the JIT compile creates a wrapper class for the inner expression calls, capturing the VALUES of the writer and anonymous type for the purpose of handing the action for BeginWritingCollection to the WriteCollectionElements static method.

The solution would be to inline the statements from lambda2 into lambda1

Action<IWriter, int> lambda4 = ( (IWriter writer, int length) => writer.BeginWritingCollection(length));
Action<IWriter, int> lambda3 = ( (IWriter writer, int value) => writer.WriteInt(value));
Action<IWriter, TheData> lambda1 = ((writer, data) =>
   {
      lambda4(writer, ((IEnumerable<int>) value.a).Count());
      WriteCollectionElements((IEnumerable<int>)value.a, writer, lambda3);
      lambda3(writer, data.b);
   });
class TheData { int[] a; int b; }