How do you compose query expressions in F#?

Question

I've been looking at query expressions here http://msdn.microsoft.com/en-us/library/vstudio/hh225374.aspx

And I've been wondering why the following is legitimate

let testQuery = query {
        for number in netflix.Titles do
        where (number.Name.Contains("Test"))
    }

But you can't really do something like this

let christmasPredicate = fun (x:Catalog.ServiceTypes.Title) -> x.Name.Contains("Christmas")
let testQuery = query {
        for number in netflix.Titles do
        where christmasPredicate 
    }

Surely F# allows composability like this so you can reuse a predicate?? What if I wanted Christmas titles combined with another predicate like before a specific date? I have to copy and paste my entire query? C# is completely unlike this and has several ways to build and combine predicates

Answer 1

This was quite easy to do with the F# 2.0 version of queries which required explicit quotations (I wrote a blog post about it). There is a way to achieve similar thing in C# (another blog post) and I think similar tricks could be played with F# 3.0.

If you do not mind uglier syntax, then you can use explicit quotations in F# 3.0 too. When you write
query { .. } the compiler actually generates something like:

query.Run(<@ ... @>) 

where the code inside <@ .. @> is quoted F# code - that is, code stored in an Expr type that represents the source code and can be translated to LINQ expressions and thus to SQL.

Here is an example that I tested with the SqlDataConnection type provider:

let db = Nwind.GetDataContext()  let predicate = <@ fun (p:Nwind.ServiceTypes.Products) ->    p.UnitPrice.Value > 50.0M @>  let test () =   <@ query.Select       ( query.Where(query.Source(db.Products), %predicate),          fun p -> p.ProductName) @>   |> query.Run   |> Seq.iter (printfn "%s") 

The key trick is that, when you use explicit quotations (using <@ .. @>) you can use the % operator for quotation slicing. This means that the quotation of predicate is put into the quotation of the query (in test) in place where you write %predicate.

The code is quite ugly compared to the nice query expression, but I suspect you could make it nicer by writing some DSL on top of this or by pre-processing the quotation.

EDIT: With a bit more effort, it is actually possible to use the query { .. } syntax again. You can quote the entire query expression and write <@ query { .. } @> - this will not directly work, but you can then take the quotation and extract the actual body of the query and pass it to query.Run directly. Here is a sample that works for the above example:

open System.Linq open Microsoft.FSharp.Quotations open Microsoft.FSharp.Quotations.Patterns  let runQuery (q:Expr<IQueryable<'T>>) =    match q with   | Application(Lambda(builder, Call(Some builder2, miRun, [Quote body])), queryObj) ->       query.Run(Expr.Cast<Microsoft.FSharp.Linq.QuerySource<'T, IQueryable>>(body))   | _ -> failwith "Wrong argument"  let test () =   <@ query { for p in db.Products do              where ((%predicate) p)              select p.ProductName } @>   |> runQuery   |> Seq.iter (printfn "%s") 

Answer 2

Naively, in the original example one could try to quote the predicate, and then splice it in:

let christmasPredicate = <@ fun (x:Catalog.ServiceTypes.Title) -> 
                             x.Name.Contains("Christmas") @>
let testQuery = query {
        for number in netflix.Titles do
        where ((%christmasPredicate) number) 
        select number
    }

(I've taken the liberty of cleaning up the original example slightly)

Examples such as this (with simple, first-order lambda-abstractions) often do work in F# anyway, but in general, there is no guarantee that F#'s default QueryBuilder will normalize away the resulting applications of lambda-abstractions in the quoted term. This can result in strange error messages, or in queries with poor performance (e.g. querying one table and then generating one query on another table per row of the first table, instead of doing a single query join).

We recently developed a library called FSharpComposableQuery that (if opened) overloads the query operator to perform normalization (and to do some other helpful things). It provides a strong guarantee to generate a single query for a nontrivial subset of F# query expressions. Using FSharpComposableQuery's version of query, the above naive composition works. We have also tested extensively to try to ensure that FSharpComposableQuery doesn't break existing query code.

Similarly, for example, using FSharpComposableQuery, Tomas's example does not require the special RunQuery function. Instead, one can simply do:

open FSharpComposableQuery

let predicate = <@ fun (p:Nwind.ServiceTypes.Product) -> 
                     p.UnitPrice.Value > 50.0M @>
let test () =
  query { for p in db.Products do
          where ((%predicate) p)
          select p.ProductName }
  |> Seq.iter (printfn "%s")

(Caveat: I have only tested the above code only with the OData version of Northwind, not the SQL type provider, but we have tested a large number of similar and more complex examples. The OData version fails with a mysterious error from OData, but this seems orthogonal to the matter at hand.)

FSharpComposableQuery is now available from NuGet here: https://www.nuget.org/packages/FSharpComposableQuery

and more information (including examples and a small tutorial, demonstrating more complex forms of composition) can be found here:

http://fsprojects.github.io/FSharp.Linq.ComposableQuery/

[EDIT: Changed the above links to remove the word "Experimental", since the project name has changed.]