C# EntityFramework IQueryable Memory Leak

Question

We're seeing memory resources not be released:

With the following code using .NET Core:

class Program
{
    static void Main(string[] args)
    {
        while (true) {          
            var testRunner = new TestRunner();
            testRunner.RunTest();
        }
    }
}

public class TestRunner {
    public void RunTest() {
        using (var context = new EasyMwsContext()) {
            var result = context.FeedSubmissionEntries.Where(fse => TestPredicate(fse)).ToList();
        }
    }

    public bool TestPredicate(FeedSubmissionEntry e) {
        return e.AmazonRegion == AmazonRegion.Europe && e.MerchantId == "1234";
    }
}

If I remove the test predicate .Where I get a straight line as expected, with the predicate the memory will continue to rise indefinitely.

So while I can fix the problem I'd like to understand what is happening?

EDIT:

Altering the line to:

public void RunTest() {
    using (var context = new EasyMwsContext()) {
        var result = context.FeedSubmissionEntries.ToList();
    }
}

Gives the graph:

So I don't believe this is due to client side evaluation either?

EDIT 2:

Using EF Core 2.1.4

And the object heap:

Edit 3:

Added a retention graph, seems to be an issue with EF Core?

Answer 1

I suspect the culprit isn't a memory leak but a rather unfortunate addition to EF Core, Client Evaluation. Like LINQ-to-SQL, when faced with a lambda/function that can't be translated to SQL, EF Core will create a simpler query that reads more data and evaluate the function on the client.

In your case, EF Core can't know what TestPredicate is so it will read every record in memory and try to filter the data afterwards.

BTW that's what happened when SO moved to EF Core last Thursday, October 4, 2018. Instead of returning a few dozen lines, the query returned ... 52 million lines :

var answers = db.Posts
                .Where(p => grp.Select(g=>g.PostId).Contains(p.Id))
                ...
                .ToList();

Client evaluation is optional but on by default. EF Core logs a warning each time client evaluation is performed, but that won't help if you haven't configured EF Core logging.

The safe solution is to disable client-side evaluation as shown in the Optional behavior: throw an exception for client evaluation section of the docs, either in each context's OnConfiguring method or globally in the Startup.cs configuration :

protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
    optionsBuilder
        .UseSqlServer(...)
        .ConfigureWarnings(warnings => 
                           warnings.Throw(RelationalEventId.QueryClientEvaluationWarning));
}

UPDATE

A quick way to find out what's leaking is to take two memory snapshots in the Diagnostics window and check what new objects were created and how much memory they use. It's quite possible there's a bug in client evaluation.

Answer 2

I ended up running into the same issue. Once I knew what the problem was I was able to find a bug report for it here in the EntityFrameworkCore repository.

The short summary is that when you include an instance method in an IQueryable it gets cached, and the methods do not get released even after your context is disposed of.

At this time it doesn't look like much progress has been made towards resolving the issue. I'll be keeping an eye on it, but for now I believe the best options for avoiding the memory leak are:

1. Rewrite your methods so no instance methods are included in your IQueryable
2. Convert the IQueryableto a list with ToList() before using LINQ methods that contain instance methods (not ideal if you're trying to limit the results of a database query)
3. Make the method you're calling static to limit how much memory piles up