High memory usage by ASP.NET applications

Question

We have an issue with some of our ASP.Net applications. Some of our apps claim a large amount of memory from start as their working set.

On our 2 webfarm-servers (4gb of RAM each) run multiple applications. We have a stable environment with about 1.2gb of memory free.

Then we add an MVC5 + WebApi v2 + Entity Framework app that instantly claims 1+gb as working set memory, while only actually using about 300mb. This causes other applications to complain that there is not enough memory left.

We already tried setting limit for virtual memory and the limit for private memory, without any avail. If we set this to about 500mb, the app still uses more or less the same amount of memory (way over 500) and does not seem to respect the limits put in place.

For reference, I tested this with an empty MVC5 project (VS2013 template) and this already claims 300mb of memory, while only using about 10mb.

Setting the app as a 32-bit app seems to have some impact in reducing the size of the working set.

Is there any way to reduce the size of the working set, or to enforce a hard limit on the size of it?

Edit: In the case of the huge memory use for the project using Web Api v2 and Entity Framework, my API controllers look like this:

namespace Foo.Api
{
public class BarController : ApiController
{
    private FooContext db = new FooContext(); 

    public IQueryable<Bar> GetBar(string bla)
    {
        return db.Bar.Where(f => f.Category.Equals(bla)).OrderBy(f => f.Year);
    }
}

as they look in most tutorials I could find (including the ones from microsoft). Using using here does not work because of LINQ deferred loading. It could work if I added a ToList (not tested) everywhere, but does this have any other impact?

edit2: It works if I do

namespace Foo.Api
{
public class BarController : ApiController
{
    public List<Bar> GetBar(string bla)
    {
        using(FooContext db = new FooContext){
           return db.Bar.Where(f => f.Category.Equals(bla)).OrderBy(f => f.Year).ToList();
        }
    }
}

Does the ToList() have any implications on the performance of the api? (I know I can not continue querying cheaply as with an IQueryable)

Edit3: I notice that its the private working set of the app that is quite high. Is there a way to limit this? (Without causing constant recycles)

Edit4: As far as I know I have a Dispose on each and every APIController. My front-end is just some simple MVC controllers but for the large part .cshtml and javascript (angular) files.

We have another app, just regular mvc, with two models and some simple views (no database, or other external resources that could be leaked) and this also consumes up to 4-500mb of memory. If I profile it, I can't see anything that indicates memory leaks, I do see that only 10 or 20 mb is actually used, the rest is unmanaged memory that is unassigned (but part of the private memory working set, so claimed by this app and unusable by any other).

Answer 1

I had a similar problem with some of my applications. I was able to solve the problem by properly closing the disposable database resources by wrapping them in using clauses.

For Entity Framework, that would mean to ensure you always close your context after each request. Connections should be disposed between requests.

using (var db = new MyEFContext())
{
   // Execute queries here
   var query = from u as db.User
               where u.UserId = 1234
               select u.Name;

   // Execute the query.
   return query.ToList();

   // This bracket will dispose the context properly.
}

You may need to wrap the context into a service that request-caches your context in order to keep it alive throughout the request, and disposes of it when complete.

Or, if using the pattern of having a single context for the entire controller like in the MSDN examples, make sure you override the Dispose(bool) method, like the example here.

protected override void Dispose(bool disposing)
{
    if (disposing)
    {
        db.Dispose();
    }
    base.Dispose(disposing);
}

So your controller (from above) should look like this:

namespace Foo.Api
{
    public class BarController : ApiController
    {
        private FooContext db = new FooContext(); 

        public IQueryable<Bar> GetBar(string bla)
        {
             return db.Bar.Where(f => f.Category.Equals(bla)).OrderBy(f => f.Year);
        }

        // WebApi 2 will call this automatically after each 
        // request. You need this to ensure your context is disposed
        // and the memory it is using is freed when your app does garbage 
        // collection.
        protected override void Dispose(bool disposing)
        {
            if (disposing)
            {
                db.Dispose();
            }
            base.Dispose(disposing);
        }
    }
}

The behavior I saw was that the application would consume a lot of memory, but it could garbage collect enough memory to keep it from ever getting an OutOfMemoryException. This made it difficult to find the problem, but disposing the database resources solved it. One of the applications used to hover at around 600 MB of RAM usage, and now it hovers around 75 MB.

But this advice doesn't just apply to database connections. Any class that implements IDisposable should be suspect if you are running into memory leaks. But since you mentioned you are using EntityFramework, it is the most likely suspect.

Answer 2

Removing all Telerik Kendo MVC references (dll and such) fixed our problems. If we run the application without, all our memory problems are gone and we see normal memory use.

Basically: it was an external library causing high memory use.