Why is Entity Framework significantly slower when running in a different AppDomain?

Question

We have a Windows service that loads a bunch of plugins (assemblies) in to their own AppDomain. Each plugin is aligned to a "service boundary" in the SOA sense, and so is responsible for accessing its own database. We have noticed that EF is 3 to 5 times slower when in a separate AppDomain.

I know that the first time EF creates a DbContext and hits the database, it has to do some setup work which has to be repeated per AppDomain (i.e. not cached across AppDomains). Considering that the EF code is entirely self-contained to the plugin (and hence self-contained to the AppDomain), I would have expected the timings to be comparable to the timings from the parent AppDomain. Why are they different?

Have tried targeting both .NET 4/EF 4.4 and .NET 4.5/EF 5.

Sample code

EF.csproj

Program.cs

class Program {     static void Main(string[] args)     {         var watch = Stopwatch.StartNew();         var context = new Plugin.MyContext();         watch.Stop();         Console.WriteLine("outside plugin - new MyContext() : " + watch.ElapsedMilliseconds);          watch = Stopwatch.StartNew();         var posts = context.Posts.FirstOrDefault();         watch.Stop();         Console.WriteLine("outside plugin - FirstOrDefault(): " + watch.ElapsedMilliseconds);          var pluginDll = Path.GetFullPath(AppDomain.CurrentDomain.BaseDirectory + @"..\..\..\EF.Plugin\bin\Debug\EF.Plugin.dll");         var domain = AppDomain.CreateDomain("other");         var plugin = (IPlugin) domain.CreateInstanceFromAndUnwrap(pluginDll, "EF.Plugin.SamplePlugin");          plugin.FirstPost();          Console.ReadLine();     } } 

EF.Interfaces.csproj

IPlugin.cs

public interface IPlugin {     void FirstPost(); } 

EF.Plugin.csproj

MyContext.cs

public class MyContext : DbContext {     public IDbSet<Post> Posts { get; set; } } 

Post.cs

public class Post {     public int Id { get; set; } } 

SamplePlugin.cs

public class SamplePlugin : MarshalByRefObject, IPlugin {     public void FirstPost()     {         var watch = Stopwatch.StartNew();         var context = new MyContext();         watch.Stop();         Console.WriteLine(" inside plugin - new MyContext() : " + watch.ElapsedMilliseconds);          watch = Stopwatch.StartNew();         var posts = context.Posts.FirstOrDefault();         watch.Stop();         Console.WriteLine(" inside plugin - FirstOrDefault(): " + watch.ElapsedMilliseconds);     } } 

Sample timings

Notes:

• This is querying against an empty database table - 0 rows.
• Timings are intentionally looking at just first calls. Subsequent calls are much faster, but still relatively 3 to 5 times slower in the child AppDomain versus the parent AppDomain.

Run 1

      outside plugin - new MyContext() : 55     outside plugin - FirstOrDefault(): 783      inside plugin - new MyContext() : 352      inside plugin - FirstOrDefault(): 2675  

Run 2

      outside plugin - new MyContext() : 53     outside plugin - FirstOrDefault(): 798      inside plugin - new MyContext() : 355      inside plugin - FirstOrDefault(): 2687  

Run 3

      outside plugin - new MyContext() : 45     outside plugin - FirstOrDefault(): 778      inside plugin - new MyContext() : 355      inside plugin - FirstOrDefault(): 2683  

AppDomain research

After some further research in to the cost of AppDomains, there seems to be a suggestion that subsequent AppDomains have to re-JIT system DLLs and so there is an inherent start-up cost in creating an AppDomain. Is that what is happening here? I would have expected that the JIT-ing would have been on AppDomain creation, but perhaps it is EF JIT-ing when it is called?

Reference for re-JIT: http://msdn.microsoft.com/en-us/magazine/cc163655.aspx#S8

Timings sounds similar, but not sure if related: First WCF connection made in new AppDomain is very slow

Update 1

Based on @Yasser's suggestion that there is EF communication across the AppDomains, I tried to isolate this further. I don't believe this to be the case.

I have completely removed any EF reference from EF.csproj. I now have enough rep to post images, so this is the solution structure:

As you can see, only the plugin has a reference to Entity Framework. I have also verified that only the plugin has a bin folder with an EntityFramework.dll.

I have added a helper to verify if the EF assembly has been loaded in the AppDomain. I have also verified (not shown) that after the call to the database, additional EF assemblies (e.g. dynamic proxy) are also loaded.

So, checking if EF has loaded at various points:

1. In Main before calling the plugin
2. In Plugin before hitting the database
3. In Plugin after hitting the database
4. In Main after calling the plugin

... produces:

 Main - IsEFLoaded: False Plugin - IsEFLoaded: True Plugin - new MyContext() : 367 Plugin - FirstOrDefault(): 2693 Plugin - IsEFLoaded: True Main - IsEFLoaded: False 

So it seems that the AppDomains are fully isolated (as expected) and the timings are the same inside the plugin.

Updated Sample code

Program.cs

class Program {     static void Main(string[] args)     {         var dir = Path.GetFullPath(AppDomain.CurrentDomain.BaseDirectory + @"..\..\..\EF.Plugin\bin\Debug");         var evidence = new Evidence();         var setup = new AppDomainSetup { ApplicationBase = dir };         var domain = AppDomain.CreateDomain("other", evidence, setup);         var pluginDll = Path.Combine(dir, "EF.Plugin.dll");         var plugin = (IPlugin) domain.CreateInstanceFromAndUnwrap(pluginDll, "EF.Plugin.SamplePlugin");          Console.WriteLine("Main - IsEFLoaded: " + Helper.IsEFLoaded());         plugin.FirstPost();         Console.WriteLine("Main - IsEFLoaded: " + Helper.IsEFLoaded());          Console.ReadLine();     } } 

Helper.cs

(Yeah, I wasn’t going to add another project for this…)

public static class Helper {     public static bool IsEFLoaded()     {         return AppDomain.CurrentDomain             .GetAssemblies()             .Any(a => a.FullName.StartsWith("EntityFramework"));     } } 

SamplePlugin.cs

public class SamplePlugin : MarshalByRefObject, IPlugin {     public void FirstPost()     {         Console.WriteLine("Plugin - IsEFLoaded: " + Helper.IsEFLoaded());          var watch = Stopwatch.StartNew();         var context = new MyContext();         watch.Stop();         Console.WriteLine("Plugin - new MyContext() : " + watch.ElapsedMilliseconds);          watch = Stopwatch.StartNew();         var posts = context.Posts.FirstOrDefault();         watch.Stop();         Console.WriteLine("Plugin - FirstOrDefault(): " + watch.ElapsedMilliseconds);          Console.WriteLine("Plugin - IsEFLoaded: " + Helper.IsEFLoaded());     } } 

Update 2

@Yasser: System.Data.Entity is loaded in to the plugin only after hitting the database. Initially only the EntityFramework.dll is loaded in the plugin, but post-database other EF assemblies are loaded too:

Zipped solution. The site only keeps files for 30 days. Feel free to suggest a better file sharing site.

Also, I am interested to know if you can verify my findings by referencing EF in the main project and seeing if the timings pattern from the original sample are reproducible.

Update 3

To be clear, it is first call timings that I am interested in analyzing which includes EF startup. On first call, going from ~800ms in a parent AppDomain to ~2700ms in a child AppDomain is very noticeable. On subsequent calls, going from ~1ms to ~3ms is hardly noticeable at all. Why is the first call (including EF startup) so much more expensive inside child AppDomains?

I’ve updated the sample to focus just on a FirstOrDefault() call to reduce the noise. Some timings for running in the parent AppDomain and running in 3 child AppDomains:

 EF.vshost.exe|0|FirstOrDefault(): 768 EF.vshost.exe|1|FirstOrDefault(): 1 EF.vshost.exe|2|FirstOrDefault(): 1  AppDomain0|0|FirstOrDefault(): 2623 AppDomain0|1|FirstOrDefault(): 2 AppDomain0|2|FirstOrDefault(): 1  AppDomain1|0|FirstOrDefault(): 2669 AppDomain1|1|FirstOrDefault(): 2 AppDomain1|2|FirstOrDefault(): 1  AppDomain2|0|FirstOrDefault(): 2760 AppDomain2|1|FirstOrDefault(): 3 AppDomain2|2|FirstOrDefault(): 1 

Updated Sample Code

    static void Main(string[] args)     {         var mainPlugin = new SamplePlugin();          for (var i = 0; i < 3; i++)             mainPlugin.Do(i);          Console.WriteLine();          for (var i = 0; i < 3; i++)         {             var plugin = CreatePluginForAppDomain("AppDomain" + i);              for (var j = 0; j < 3; j++)                 plugin.Do(j);              Console.WriteLine();         }          Console.ReadLine();     }      private static IPlugin CreatePluginForAppDomain(string appDomainName)     {         var dir = Path.GetFullPath(AppDomain.CurrentDomain.BaseDirectory + @"..\..\..\EF.Plugin\bin\Debug");         var evidence = new Evidence();         var setup = new AppDomainSetup { ApplicationBase = dir };         var domain = AppDomain.CreateDomain(appDomainName, evidence, setup);         var pluginDll = Path.Combine(dir, "EF.Plugin.dll");         return (IPlugin) domain.CreateInstanceFromAndUnwrap(pluginDll, "EF.Plugin.SamplePlugin");     }  public class SamplePlugin : MarshalByRefObject, IPlugin {     public void Do(int i)     {         var context = new MyContext();          var watch = Stopwatch.StartNew();         var posts = context.Posts.FirstOrDefault();         watch.Stop();         Console.WriteLine(AppDomain.CurrentDomain.FriendlyName + "|" + i + "|FirstOrDefault(): " + watch.ElapsedMilliseconds);     } } 

Zipped solution. The site only keeps files for 30 days. Feel free to suggest a better file sharing site.

Answer 1

This seems to be just the cost of child AppDomains. A rather ancient post (which may no longer be relevant) suggests that there could be other considerations outside of just having to JIT-compile each child AppDomain, e.g. evaluating security policies.

Entity Framework does have a relatively high startup cost so the effects are magnified, but for comparision calling other parts of System.Data (e.g. a straight SqlDataReader) is just as horrible:

 EF.vshost.exe|0|SqlDataReader: 67 EF.vshost.exe|1|SqlDataReader: 0 EF.vshost.exe|2|SqlDataReader: 0  AppDomain0|0|SqlDataReader: 313 AppDomain0|1|SqlDataReader: 2 AppDomain0|2|SqlDataReader: 0  AppDomain1|0|SqlDataReader: 290 AppDomain1|1|SqlDataReader: 3 AppDomain1|2|SqlDataReader: 0  AppDomain2|0|SqlDataReader: 316 AppDomain2|1|SqlDataReader: 2 AppDomain2|2|SqlDataReader: 0 

public class SamplePlugin : MarshalByRefObject, IPlugin {     public void Do(int i)     {         var watch = Stopwatch.StartNew();         using (var connection = new SqlConnection("Data Source=.\\sqlexpress;Initial Catalog=EF.Plugin.MyContext;Integrated Security=true"))         {             var command = new SqlCommand("SELECT * from Posts;", connection);             connection.Open();             var reader = command.ExecuteReader();             reader.Close();         }         watch.Stop();          Console.WriteLine(AppDomain.CurrentDomain.FriendlyName + "|" + i + "|SqlDataReader: " + watch.ElapsedMilliseconds);     } } 

Even newing up a humble DataTable is inflated:

 EF.vshost.exe|0|DataTable: 0 EF.vshost.exe|1|DataTable: 0 EF.vshost.exe|2|DataTable: 0  AppDomain0|0|DataTable: 12 AppDomain0|1|DataTable: 0 AppDomain0|2|DataTable: 0  AppDomain1|0|DataTable: 11 AppDomain1|1|DataTable: 0 AppDomain1|2|DataTable: 0  AppDomain2|0|DataTable: 10 AppDomain2|1|DataTable: 0 AppDomain2|2|DataTable: 0 

public class SamplePlugin : MarshalByRefObject, IPlugin {     public void Do(int i)     {         var watch = Stopwatch.StartNew();         var table = new DataTable("");         watch.Stop();          Console.WriteLine(AppDomain.CurrentDomain.FriendlyName + "|" + i + "|DataTable: " + watch.ElapsedMilliseconds);     } }