.NET4 ExpandoObject usage leaking memory

Question

I have an inherited .NET 4.0 application that runs as a Windows service. I'm not a .NET expert by any stretch, but after writing code for 30+ years I know how to find my way around.

When the service first starts it clocks in at around 70MB private working set. The longer the service runs, the more memory it takes. The increase isn't so dramatic that you notice while just sitting and watching, but we have seen instances where after the application has run for a long time (100+ days) it's up to multiple GB (5GB is the current record). I attached ANTS Memory Profiler to a running instance and found that usage of ExpandoObject seems to account for multiple megabytes of strings which do not get cleaned up by GC. There are likely other leaks, but this was the most noticeable so it got attacked first.

I've learned from other SO posts that "normal" usage of ExpandoObject generates an internal RuntimeBinderException when reading (but not writing) the dynamically assigned attributes.

dynamic foo = new ExpandoObject();
var s;
foo.NewProp = "bar"; // no exception
s = foo.NewProp;     // RuntimeBinderException, but handled by .NET, s now == "bar"

You can see the exception happen in VisualStudio, but ultimately it's handled in the .NET internals and all you get back is the value you want.

Except... The string in the exception's Message property appears to stay on the heap and never gets Garbage Collected, even long after the ExpandoObject that generated it goes out of scope.

Simple example:

using System;
using System.Dynamic;

namespace ConsoleApplication2
{
   class Program
   {
      public static string foocall()
      {
         string str = "", str2 = "", str3 = "";
         object bar = new ExpandoObject();
         dynamic foo = bar;
         foo.SomePropName = "a test value";
         // each of the following references to SomePropName causes a RuntimeBinderException - caught and handled by .NET
         // Attach an ANTS Memory profiler here and look at string instances
         Console.Write("step 1?");
         var s2 = Console.ReadLine();
         str = foo.SomePropName;
         // Take another snapshot here and you'll see an instance of the string:
         // 'System.Dynamic.ExpandoObject' does not contain a definition for 'SomePropName'
         Console.Write("step 2?");
         s2 = Console.ReadLine();
         str2 = foo.SomePropName;
         // Take another snapshot here and you'll see 2nd instance of the identical string
         Console.Write("step 3?");
         s2 = Console.ReadLine();
         str3 = foo.SomePropName;

         return str;
      }
      static void Main(string[] args)
      {
         var s = foocall();
         Console.Write("Post call, pre-GC prompt?");
         var s2 = Console.ReadLine();
         // At this point, ANTS Memory Profiler shows 3 identical strings in memory
         // generated by the RuntimeBinderExceptions in foocall. Even though the variable
         // that caused them is no longer in scope the strings are still present.

         // Force a GC, just for S&G
         GC.Collect();
         GC.WaitForPendingFinalizers();
         GC.Collect();
         Console.Write("Post GC prompt?");
         s2 = Console.ReadLine();
         // Look again in ANTS.  Strings still there.
         Console.WriteLine("foocall=" + s);
      }
   }
}

"bug" is in the eye of the beholders, I suppose (my eyes say bug). Am I missing something? Is this normal and expected by the .NET masters in the group? Is there some way to tell it to clear the things out? Is the best way to just not use dynamic/ExpandoObject in the first place?

Answer 1

This appears to be due to caching performed by compiler-generated code for the dynamic property access. (Analysis performed with the output from VS2015 and .NET 4.6; other compiler versions may produce different output.)

The call str = foo.SomePropName; is rewritten by the compiler into something like this (according to dotPeek; note that <>o__0 etc. are tokens that aren't legal C# but are created by the C# compiler):

if (Program.<>o__0.<>p__2 == null)
{
    Program.<>o__0.<>p__2 = CallSite<Func<CallSite, object, string>>.Create(Binder.Convert(CSharpBinderFlags.None, typeof (string), typeof (Program)));
}
Func<CallSite, object, string> target1 = Program.<>o__0.<>p__2.Target;
CallSite<Func<CallSite, object, string>> p2 = Program.<>o__0.<>p__2;
if (Program.<>o__0.<>p__1 == null)
{
    Program.<>o__0.<>p__1 = CallSite<Func<CallSite, object, object>>.Create(Binder.GetMember(CSharpBinderFlags.None, "SomePropName", typeof (Program), (IEnumerable<CSharpArgumentInfo>) new CSharpArgumentInfo[1]
    {
        CSharpArgumentInfo.Create(CSharpArgumentInfoFlags.None, (string) null)
    }));
}
object obj3 = Program.<>o__0.<>p__1.Target((CallSite) Program.<>o__0.<>p__1, obj1);
string str1 = target1((CallSite) p2, obj3);

Program.<>o__0.<>p__1 is a static field (on a private nested class) of type CallSite<Func<CallSite,object,object>>. It holds a reference to the dynamic method that is compiled on demand the first time foo.SomePropName is accessed. (Presumably this is because creating the binding is slow, so caching it provides a significant speed increase on subsequent accesses.)

This DynamicMethod holds a reference to a DynamicILGenerator which references a DynamicScope which eventually holds a list of tokens. One of these tokens is the dynamically-generated string 'System.Dynamic.ExpandoObject' does not contain a definition for 'SomePropName'. This string exists in memory so that the dynamically-generated code can throw (and catch) a RuntimeBinderException with the "right" message.

Overall, the <>p__1 field keeps about 2K of data alive (including 172 bytes for this string). There is no supported way to free this data, because it's rooted by a static field on a compiler-generated type. (You could of course use reflection to set that static field to null, but this would be extremely dependent on the implementation details of the current compiler, and very likely to break in the future.)

From what I've seen so far, it appears that using dynamic allocates about 2K of memory per property access in C# code; you probably just have to consider this the price of using dynamic code. However (at least in this simplified example), that memory is only allocated the first time the code is executed, so it shouldn't keep using more memory the longer the program runs; there may be a different leak that pushes the working set up to 5GB. (There are three instances of the string because there are three separate lines of code that execute foo.SomePropName; however, there would still only be three instances if you invoked foocall 100 times.)

To improve performance and reduce memory usage, you may want to consider using Dictionary<string, string> or Dictionary<string, object> as a simpler key/value store (if that's possible with the way the code is written). Note that ExpandoObject implements IDictionary<string, object> so the following small rewrite produces the same output but avoids the overhead of dynamic code:

public static string foocall()
{
    string str = "", str2 = "", str3 = "";

    // use IDictionary instead of dynamic to access properties by name
    IDictionary<string, object> foo = new ExpandoObject();

    foo["SomePropName"] = "a test value";
    Console.Write("step 1?");
    var s2 = Console.ReadLine();

    // have to explicitly cast the result here instead of having the compiler do it for you (with dynamic)
    str = (string) foo["SomePropName"];

    Console.Write("step 2?");
    s2 = Console.ReadLine();
    str2 = (string) foo["SomePropName"];
    Console.Write("step 3?");
    s2 = Console.ReadLine();
    str3 = (string) foo["SomePropName"];

    return str;
}