Custom IQueryProvider that falls back on LinqToObjects

Question

I have written a custom IQueryProvider class that takes an expression and analyses it against a SQL database (I know I could use Linq2Sql but there are some modifications and tweaks that I need that unfortunately make Linq2Sql unsuitable). The class will identify and do something with the properties that are marked (using attributes) but any that aren't I would like to be able to pass the expression on to a LinqToObject provider and allow it to filter the results after.

For example, suppose I have the following linq expression:

var parents=Context.Parents
    .Where(parent=>parent.Name.Contains("T") && parent.Age>18);

The Parents class is a custom class that implements IQueryProvider and IQueryable interfaces, but only the Age property is marked for retrieval, so the Age property will be processed, but the Name property is ignored because it is not marked. After I've finished processing the Age property, I'd like to pass the whole expression to LinqToObjects to process and filter, but I don't know how.

N.B. It doesn't need to remove the Age clause of the expression because the result will be the same even after I've processed it so I will always be able to send the whole expression on to LinqToObjects.

I've tried the following code but it doesn't seem to work:

IEnumerator IEnumerable.GetEnumerator() {       
    if(this.expression != null && !this.isEnumerating) {
        this.isEnumerating = true;
        var queryable=this.ToList().AsQueryable();
        var query = queryable.Provider.CreateQuery(this.expression);
        return query.GetEnumerator();
    }
    return this;
}

this.isEnumerating is just a boolean flag set to prevent recursion.

this.expression contains the following:

{value(namespace.Parents`1[namespace.Child]).Where(parent => ((parent.Name.EndsWith("T") AndAlso parent.Name.StartsWith("M")) AndAlso (parent.Test > 0)))}

When I step through the code, despite converting the results to a list, it still uses my custom class for the query. So I figured that because the class Parent was at the beginning of the expression, it was still routing the query back to my provider, so I tried setting this.expression to Argument[1] of the method call so it was as such:

{parent => ((parent.Name.EndsWith("T") AndAlso parent.Name.StartsWith("M")) AndAlso (parent.Test > 0))}

Which to me looks more like it, however, whenever I pass this into the CreateQuery function, I get this error 'Argument expression is not valid'.

The node type of the expression is now 'Quote' though and not 'Call' and the method is null. I suspect that I just need to make this expression a call expression somehow and it will work, but I'm not sure how to.

Please bear in mind that this expression is a where clause, but it may be any kind of expression and I'd prefer not to be trying to analyse the expression to see what type it is before passing it in to the List query provider.

Perhaps there is a way of stripping off or replacing the Parent class of the original expression with the list provider class but still leaving it in a state that can just be passed in as expression into the List provider regardless of the type of expression?

Any help on this would be greatly appreciated!

Answer 1

You were so close!

My goal was to avoid having to "replicate" the full mind-numbingly convoluted SQL-to-Object expressions feature set. And you put me on the right track (thanks!) here's how to piggy-back SQL-to-Object in a custom IQueryable:

public IEnumerator<T> GetEnumerator() {

    // For my case (a custom object-oriented database engine) I still 
    // have an IQueryProvider which builds a "subset" of objects each populated 
    // with only "required" fields, as extracted from the expression. IDs, 
    // dates, particular strings, what have you. This is "cheap" because it 
    // has an indexing system as well.

    var en = ((IEnumerable<T>)this.provider.Execute(this.expression));

    // Copy your internal objects into a list.

    var ar = new List<T>(en);
    var queryable = ar.AsQueryable<T>();

    // This is where we went wrong:
    // queryable.Provider.CreateQuery(this.expression);
    // We can't re-reference the original expression because it will loop 
    // right back on our custom IQueryable<>. Instead, swap out the first 
    // argument with the List's queryable:

    var mc = (MethodCallExpression)this.expression;
    var exp = Expression.Call(mc.Method, 
                    Expression.Constant(queryable), 
                    mc.Arguments[1]);


    // Now the CLR can do all of the heavy lifting
    var query = queryable.Provider.CreateQuery<T>(exp);
    return query.GetEnumerator();
}

Can't believe this took me 3 days to figure out how to avoid reinventing wheel on LINQ-to-Object queries.