How to design a pluggable system in functional style?

Question

Disclaimer:
Although I accept the gospel of immutable state and higher order functions, my real-world experience is still 95% object-oriented. I'd love to change that, but whatchagonnado. So my brain is very much wired to OO.

Question:
I have this situation very frequently: a piece of business functionality implemented as a small "core" plus multiple "plugins", working together to present a seemingly solid surface to the user. I found that this "microkernel" architecture works extremely well in a lot of circumstances. Plus, very conveniently, it nicely combines with a DI container, which can be used for plugin discovery.

So, how do I do this in a functional way?

I do not think that the basic idea in this technique is inherently object-oriented, because I've just described it without using any OO terms or concepts. However, I can't quite wrap my head around the functional way to go about it. Sure, I can represent plugins as functions (or buckets of functions), but the difficult part comes when plugins need to have their own data as part of the big picture, and the shape of data is different form plugin to plugin.

Below is a small F# snippet that is more or less literal translation of C# code that I would write when implementing this pattern from scratch.
Note the weak points: losing type information in CreateData, necessary upcast in PersistData.
I flinch at casts (whether up or down) every time, but I've learned to accept them as a necessary evil in C#. However, my past experience suggests that the functional approach often offers unexpected, yet beatiful and elegant solutions to this kind of problems. It is such solution that I am after.

type IDataFragment = interface end
type PersistedData = string // Some format used to store data in persistent storage
type PluginID = string // Some form of identity for plugins that would survive app restart/rebuild/upgrade

type IPlugin = interface
  abstract member UniqueID: PluginID
  abstract member CreateData: unit -> IDataFragment

  // NOTE: Persistence is conflated with primary function for simplicity. 
  // Regularly, persistence would be handled by a separate component.
  abstract member PersistData: IDataFragment -> PersistedData option
  abstract member LoadData: PersistedData -> IDataFragment
end

type DataFragment = { Provider: IPlugin; Fragment: IDataFragment }
type WholeData = DataFragment list

// persist: WholeData -> PersistedData
let persist wholeData = 
  let persistFragmt { Provider = provider; Fragment = fmt } = 
    Option.map (sprintf "%s: %s" provider.UniqueID) (provider.PersistData fmt)

  let fragments = wholeData |> Seq.map persistFragmt |> Seq.filter Option.isSome |> Seq.map Option.get
  String.concat "\n" fragments // Not a real serialization format, simplified for example

// load: PersistedData -> WholeData
let load persistedData = // Discover plugins and parse the above format, omitted

// Reference implementation of a plugin
module OnePlugin =
  type private MyData( d: string ) = 
    interface IDataFragment
    member x.ActualData = d

  let create() = 
    {new IPlugin with
      member x.UniqueID = "one plugin"
      member x.CreateData() = MyData( "whatever" ) :> _
      member x.LoadData d = MyData( d ) :> _

      member x.PersistData d = 
        match d with
        | :? MyData as typedD -> Some typedD.ActualData
        | _ -> None
    }

Some updates and clarifications

• I do not need to be educated in functional programming "in general" (or at least that's what I like to think :-). I do realize how interfaces are related to functions, I do know what higher-order functions are, and how function composition works. I even understand monads warm fluffy things (as well as some other mumbo-jumbo from category theory).
• I realize that I don't need to use interfaces in F#, because functions are generally better. But both interfaces in my example are actually justified: IPlugin serves to bind together UniqueID and CreateData; if not interface, I would use a record of similar shape. And IDataFragment serves to limit the types of data fragments, otherwise I would have to use obj for them, which would give me even less type safety. (and I can't even imagine how I would go about it in Haskell, short of using Dynamic)

Answer 1

I can only sympathize with your statements. While functional programming in the small has been talked to death, there is precious little advice on how to do functional programming in the large. I think for F# in particular most solutions will gravitate towards more object-oriented (or at least, interface-oriented) style as your system grows. I don't think it's necessarily bad - but if there is a convincing FP solution, I would like to see it as well.

One pattern that I have seen used in a similar scenario was to have a pair of interfaces, a typed and an untyped one, and a reflection based mechanism to go between them. So in your scenario you'd have something like this:

type IPlugin =
    abstract member UniqueID: PluginID
    abstract member DataType: System.Type
    abstract member CreateData: unit -> IDataFragment

type IPlugin<'data> = 
    inherit IPlugin

    abstract member CreateData: unit -> 'data
    abstract member PersistData: 'data -> PersistedData option
    abstract member LoadData: PersistedData -> 'data

and an implementation would look like this:

let create() = 
    let createData () = MyData( "whatever" )
    {
        new IPlugin with
            member x.UniqueID = "one plugin"
            member x.DataType = typeof<MyData>                
            member x.CreateData() = upcast createData()
        interface IPlugin<MyData> with
            member x.LoadData d = MyData( d )
            member x.PersistData (d:MyData) = Some d.ActualData
            member x.CreateData() = createData()            
    }

Note that CreateData is part of both interfaces - it's just there to illustrate that there's a balance to strike between how much is duplicated between the typed and untyped interface and how often you need to jump through the hoops to convert between them. Ideally CreateData shouldn't be there in IPlugin, but if it saves you time, I wouldn't look back twice.

For going from IPlugin to IPlugin<'a> you'd need a reflection-based helper function, but at least you explicitly know the type argument since it's part of IPlugin interface. And while it's not pretty, at least the type conversion code is contained in a single part of the code, rather than being sprinkled across all the plugins.

Answer 2

You don't have to define interfaces in order to make an architecture pluggable in F#. Functions are already composable.

You can write your system Outside-In, starting with the desired, overall behaviour of your system. For example, here's a function I recently wrote that transitions a Polling Consumer from a state where no message was received, into a new state:

let idle shouldSleep sleep (nm : NoMessageData) : PollingConsumer =
    if shouldSleep nm
    then sleep () |> Untimed.withResult nm.Result |> ReadyState
    else StoppedState ()

This is a higher order function. While I was writing it, I discovered that it depended on the auxiliary functions shouldSleep and sleep, so I added these to the argument list. The compiler then automatically infers that e.g. shouldSleep must have the type NoMessageData -> bool. That function is a Dependency. The same goes for the sleep function.

As a second step, it turns out that a reasonable implementation of a shouldSleep function ends up looking like this:

let shouldSleep idleTime stopBefore (nm : NoMessageData) =
    nm.Stopped + idleTime < stopBefore

Never mind if you don't know what it all does. It's the composition of functions that matter here. In this case, we've learned that this particular shouldSleep function has the type TimeSpan -> DateTimeOffset -> NoMessageData -> bool, which isn't quite the same as NoMessageData -> bool.

It's pretty close, though, and you can use partial function application to go the rest of the distance:

let now' = DateTimeOffset.Now
let stopBefore' = now' + TimeSpan.FromSeconds 20.
let idleTime' = TimeSpan.FromSeconds 5.
let shouldSleep' = shouldSleep idleTime' stopBefore'

The shouldSleep' function is a partial application of the shouldSleep function, and has the desired type NoMessageData -> bool. You can compose this function into the idle function, together with an implementation of its sleep dependency.

Since the lower-order function has the correct type (the correct function signature), it just clicks into place; no casting is necessary in order to achieve this.

The idle, shouldSleep and shouldSleep' functions can be defined in different modules, in different libraries, and you can pull them all together using a process similar to Pure DI.

If you want to see a more comprehensive example of composing an entire application from individual functions, I provide an example in my Functional Architecture with F# Pluralsight course.