EF, ASP MVC + dependency injection. Issues with multiple concurrent requests and DB connectivity

Question

I'm working on a NopCommerce-based project which uses ASP MVC, Autofac and Entity Framework. I'm having exceptions which happen when calling a method on a service from inside an MVC Route which will make a call to the DB using EF.

During dev, everything works fine - however during load testing, when there are concurrent users, 1 or 2 requests will crash out, and one of the following errors are logged to ELMAH.

System.InvalidOperationException ExecuteReader requires an open and available Connection. The connection's current state is open.

System.InvalidOperationException ExecuteReader requires an open and available Connection. The connection's current state is closed.

System.InvalidOperationException The connection was not closed. The connection's current state is connecting.

System.ObjectDisposedException The ObjectContext instance has been disposed and can no longer be used for operations that require a connection.

System.ObjectDisposedException The operation cannot be completed because the DbContext has been disposed.

System.ArgumentException An item with the same key has already been added.

I test this by opening many links on the site and then using a Chrome plug-in to refresh all the tabs, simulating ~25 requests hitting the site at the same time.

The service has 2 methods which are called from inside the route, and then one of these same methods can get called 50+ times from the controller action. Sometimes the exception is triggered from inside the Route, and sometimes it comes from inside the controller. Meaning that the route's GetRouteData has completed, passed over 'flow' to the controller and then failed inside there. But, most of the time, the exception occurs within the Route. When the exception does happen from inside the controller, it's at different lines where the exception occurs.

Sometimes one method will fail, and another time that method will run fine, and then next method in the call stack fails. It's different each time, but these method calls use a common method for retrieving from the DB.

There are 2 other routes which are registered before this one that are mapped to *{url} that perform database lookups of the incoming URL, and the exception never happens there. So, it's not like this route is the first operation to perform any DB work.

The service is dependency registered :-

builder.RegisterControllers(typeFinder.GetAssemblies().ToArray());

builder.Register<IDbContext>(c => new NopObjectContext(DataSettings.DataConnectionString)).InstancePerLifetimeScope();

builder.RegisterGeneric(typeof(EfRepository<>)).As(typeof(IRepository<>)).InstancePerLifetimeScope();

builder.RegisterType<MemoryCacheManager>().As<ICacheManager>().Named<ICacheManager>("nop_cache_static").SingleInstance();

builder.RegisterType<MyService>().As<IMySerivce>()    
               .WithParameter(ResolvedParameter.ForNamed<ICacheManager>("nop_cache_static"))    
               .InstancePerLifetimeScope(); 

The Controller receives the service via Constructor Injection:-

protected readonly IMyService _myService;

public MyController(IMyService myService)
{
    _myService = myService;
}

And the route resolves the service as followed:

public override RouteData GetRouteData(HttpContextBase httpContext)
{
    var _myService = EngineContext.Current.Resolve<IMyService>();
    myService.databaseOperation(); <--- falls over here w/ concurrency
} 

Any ideas why these errors are occurring? and how to resolve them?

From my understanding, it seems that our DBContext is being shared across 2 requests, however in my dependency registration I've told it to resolve as a lifetime score - which should be unique to every request. I've read a lot into causes of this exception and how it's down to the dependency injection framework to control the lifetime of a dependency and to manage disposing of its resources - it's here that things are falling down, it seems.

In regards to the service itself, it works like all other services in the application - there's nothing standout and different about it.

I've pasted the full exception stacktraces http://pastebin.com/XYEwRQsv as well as the offending line of code.

EDIT: I am using MultipleActiveResultSets=True in the connection string. The entity that this service deals with is stand-alone, i.e. it has no relations with other entities, so there should be no multiple iterations of child entities once the query's been executed, as other answers in regards to these exceptions point out.

EDIT: This is the line that throws the exception.

public string GetFriendlyUrlString(string originalUrl)
{
    var friendly =  _cacheManager.Get(originalUrl.ToLower(), () =>
            (from f in _friendlyUrlRepository.ReadOnlyTable      <--------- Here
             where f.OriginalUrl == originalUrl
             select f.Url).ToList().SingleOrDefault());

    return friendly ?? originalUrl;
}

And exception is

ExecuteReader requires an open and available Connection. The connection's current state is closed.

This is so strange. In my route, there are 4 places where a DB call can happen. My exceptions are 95% of the time from one of these 4 calls - usually the first fails, but sometimes the first DB call will be ok and the others fall through. Very infrequently I see the exception come from inside the controller that this route uh...routes to. Again, with that controller exception then the actual DB connection problem happens on one of 5 lines of code - again showing that it's made x many DB calls then fallen over.

Answer 1

Autofac's InstancePerLifetimeScope does NOT guarantee a unique scope per http request. What it guarantees is that there will only be one instance of your component within the lifetime scope from which it was resolved. So, for example, if you resolve an InstancePerLifetimeScope component from the root container, that component will essentially act as a singleton over the course of the application.

If you're resolving your dependency inside a singly-registered service (e.g., a global action filter), then your dependency (dbContext or whatever) will not get disposed on each request -- it'll just stay open, leaking memory and bleeding connections until it either gets disposed or takes your app down.

Assuming you're using the Autofac Mvc integration, as an experiment, perhaps try to TEMPORARILY register your dbContext as InstancePerMatchingLifetimeScope("AutofacWebRequest") -- this is actually the equivalent of doing exactly what Will Appleby said, but without affinity to a newer version of Autofac than you may be using.

If that solves your problem, then basically, @Will Appleby was right. You'll need to register all your InstancePerLifetimeScope components as InstancePerRequest (newly added to Autofac core as of version 3.4.0), or InstancePerHttpRequest (available in Autofac MVC integration prior to 3.4.0).

Otherwise, if I'm guessing correctly, you'll probably get an exception something like this:

DependencyResolutionException: No scope with a Tag matching 'AutofacWebRequest' is visible from the scope in which the instance was requested.

If that happens, you'll still want to change your registrations to InstancePerRequest/InstancePerHttpRequest, but now you have a deeper task at hand. You probably have a singleton (or other object that lives longer than the httpRequest) holding your dbContext hostage. You'll need to identify that longer-lived component, and figure out how to free that captive dependency -- that will likely require refactoring the component to better comply with Autofac's scoping behavior.

You'll want to refer to the troubleshooting section here: http://autofac.readthedocs.org/en/latest/faq/per-request-scope.html#troubleshooting-per-request-dependencies

Here's an excerpt from there:

Common causes for this include:

• Application registrations are being shared across application types.
• A unit test is running with real application registrations but isn’t simulating per-request lifetimes.
• You have a component that lives longer than one request but it takes a dependency that only lives for one request. For example, a singleton component that takes a service registered as per-request.
• Code is running during application startup (e.g., in an ASP.NET Global.asax) that uses dependency resolution when there isn’t an active request yet.
• Code is running in a “background thread” (where there’s no request semantics) but is trying to call the ASP.NET MVC DependencyResolver to do service location.

Answer 2

Are you sure your IDbContext implementation is being injected with the right scope? I don't use Autofac myself but a quick check of their website documentation suggests that InstancePerRequest would be a more suitable scope for your needs:

InstancePerRequest

Also have you got MARS set to true in your connection string?

Multiple Active Result Sets