Removing select N+1 without .Include

Question

Consider these contrived entity objects:

public class Consumer
{
    public int Id { get; set; }
    public string Name { get; set; }
    public bool NeedsProcessed { get; set; }
    public virtual IList<Purchase> Purchases { get; set; }  //virtual so EF can lazy-load
}

public class Purchase
{
    public int Id { get; set; }
    public decimal TotalCost { get; set; }
    public int ConsumerId { get; set; }
}

Now let's say I want to run this code:

var consumers = Consumers.Where(consumer => consumer.NeedsProcessed);

//assume that ProcessConsumers accesses the Consumer.Purchases property
SomeExternalServiceICannotModify.ProcessConsumers(consumers);

By default this will suffer from Select N+1 inside the ProcessConsumers method. It will trigger a query when it enumerates the consumers, then it'll grab each purchases collection 1 by 1. The standard solution to this problem would be to add an include:

var consumers = Consumers.Include("Purchases").Where(consumer => consumer.NeedsProcessed);

//assume that ProcessConsumers accesses the Consumer.Purchases property
SomeExternalServiceICannotModify.ProcessConsumers(consumers);

That works fine in many cases, but in some complex cases, an include can utterly destroy performance by orders of magnitude. Is it possible to do something like this:

1. Grab my consumers, var consumers = _entityContext.Consumers.Where(...).ToList()
2. Grab my purchases, var purchases = _entityContext.Purchases.Where(...).ToList()
3. Hydrate the consumer.Purchases collections manually from the purchases I already loaded into memory. Then when I pass it to ProcessConsumers it won't trigger more db queries.

I'm not sure how to do #3. If you try to access any consumer.Purchases collection that'll trigger the lazy load (and thus the Select N+1). Perhaps I need to cast the Consumers to the proper type (instead of the EF proxy type) and then load the collection? Something like this:

foreach (var consumer in Consumers)
{
     //since the EF proxy overrides the Purchases property, this doesn't really work, I'm trying to figure out what would
     ((Consumer)consumer).Purchases = purchases.Where(x => x.ConsumerId = consumer.ConsumerId).ToList();
}

EDIT: I have re-written the example a bit to hopefully reveal the issue more clearly.

Answer 1

If I'm understanding correctly, you would like to load both a filtered subset of Consumers each with a filtered subset of their Purchases in 1 query. If that's not correct, please forgive my understanding of your intent. If that is correct, you could do something like:

var consumersAndPurchases = db.Consumers.Where(...)
    .Select(c => new {
        Consumer = c,
        RelevantPurchases = c.Purchases.Where(...)
    })
    .AsNoTracking()
    .ToList(); // loads in 1 query

// this should be OK because we did AsNoTracking()
consumersAndPurchases.ForEach(t => t.Consumer.Purchases = t.RelevantPurchases);

CannotModify.Process(consumersAndPurchases.Select(t => t.Consumer));

Note that this WON'T work if the Process function is expecting to modify the consumer object and then commit those changes back to the database.