How do I change the Rx Builder implementation to fix the stack overflow exception?

Question

I'm trying to come up with an Rx Builder to use Reactive Extension within the F# Computation Expression syntax. How do I fix it so that it doesnt blow the stack? Like the Seq example below. And is there any plans to provide an implementation of the RxBuilder as part of the Reactive Extensions or as part of future versions of the .NET Framework ?

open System
open System.Linq
open System.Reactive.Linq

type rxBuilder() =    
    member this.Delay f = Observable.Defer f
    member this.Combine (xs: IObservable<_>, ys : IObservable<_>) = 
        Observable.merge xs ys      
    member this.Yield x = Observable.Return x
    member this.YieldFrom (xs:IObservable<_>) = xs

let rx = rxBuilder()

let rec f x = seq { yield x 
                    yield! f (x + 1) }

let rec g x = rx { yield x 
                    yield! g (x + 1) }


//do f 5 |> Seq.iter (printfn "%A")

do g 5 |> Observable.subscribe (printfn "%A") |> ignore

do System.Console.ReadLine() |> ignore

Answer 1

A short answer is that Rx Framework doesn't support generating observables using a recursive pattern like this, so it cannot be easily done. The Combine operation that is used for F# sequences needs some special handling that observables do not provide. The Rx Framework probably expects that you'll generate observables using Observable.Generate and then use LINQ queries/F# computation builder to process them.

Anyway, here are some thoughts -

First of all, you need to replace Observable.merge with Observable.Concat. The first one runs both observables in parallel, while the second first yields all values from the first observable and then produces values from the second observable. After this change, the snippet will at least print ~800 numbers before the stack overflow.

The reason for the stack overflow is that Concat creates an observable that calls Concat to create another observable that calls Concat etc. One way to solve this is to add some synchronization. If you're using Windows Forms, then you can modify Delay so that it schedules the observable on the GUI thread (which discards the current stack). Here is a sketch:

type RxBuilder() =   
  member this.Delay f = 
      let sync = System.Threading.SynchronizationContext.Current 
      let res = Observable.Defer f
      { new IObservable<_> with
          member x.Subscribe(a) = 
            sync.Post( (fun _ -> res.Subscribe(a) |> ignore), null)
            // Note: This is wrong, but we cannot easily get the IDisposable here
            null }
  member this.Combine (xs, ys) = Observable.Concat(xs, ys)
  member this.Yield x = Observable.Return x
  member this.YieldFrom (xs:IObservable<_>) = xs

To implement this properly, you would have to write your own Concat method, which is quite complicated. The idea would be that:

• Concat returns some special type e.g. IConcatenatedObservable
• When the method is called recursively you'll create a chain of IConcatenatedObservable that reference each other
• The Concat method will look for this chain and when there are e.g. three objects, it will drop the middle one (to always keep chain of length at most 2).

That's a bit too complex for a StackOverflow answer, but it may be a useful feedback for the Rx team.

Answer 2

Notice this has been fixed in Rx v2.0 (as mentioned here already), more generally for all of the sequencing operators (Concat, Catch, OnErrorResumeNext), as well as the imperative operators (If, While, etc.).

Basically, you can think of this class of operators as doing a subscribe to another sequence in a terminal observer message (e.g. Concat subscribes to the next sequence upon receiving the current one's OnCompleted message), which is where the tail recursion analogy comes in.

In Rx v2.0, all of the tail-recursive subscriptions are flattened into a queue-like data structure for processing one at a time, talking to the downstream observer. This avoids the unbounded growth of observers talking to each other for successive sequence subscriptions.