Idiomatic approach to a Go plugin-based system

Question

I have a Go project I would like to open source but there are certain elements which are not suitable for OSS, e.g. company specific logic etc.

I have conceived of the following approach:

• interfaces are defined in the core repository.
• Plugins can then be standalone repositories whose types implement the interfaces defined in core. This allows the plugins to be housed in completely separate modules and therefore have their own CI jobs etc.
• Plugins are compiled into the final binary via symlinks.

This would result in a directory structure something like the following:

|- $GOPATH
  |- src
    |- github.com
      |- jabclab
        |- core-system
          |- plugins <-----|
      |- xxx               | 
        |- plugin-a ------>| ln -s
      |- yyy               |  
        |- plugin-b ------>|

With an example workflow of:

$ go get [email protected]:jabclab/core-system.git
$ go get [email protected]:xxx/plugin-a.git
$ go get [email protected]:yyy/plugin-b.git
$ cd $GOPATH/src/github.com
$ ln -s ./xxx/plugin-a/*.go ./jabclab/core-system/plugins
$ ln -s ./yyy/plugin-b/*.go ./jabclab/core-system/plugins
$ cd jabclab/core-system
$ go build

The one issue I'm not sure about is how to make the types defined in plugins available at runtime in core. I'd rather not use reflect but can't think of a better way at the moment. If I was doing the code in one repo I would use something like:

package plugins

type Plugin interface {
  Exec(chan<- string) error
}

var Registry map[string]Plugin

// plugin_a.go
func init() { Registry["plugin_a"] = PluginA{} }

// plugin_b.go
func init() { Registry["plugin_b"] = PluginB{} }

In addition to the above question would this overall approach be considered idiomatic?

Answer 1

This is one of my favorite issues in Go. I have an open source project that has to deal with this as well (https://github.com/cpg1111/maestrod), it has pluggable DB and Runtime (Docker, k8s, Mesos, etc) clients. Prior to the plugin package that is in the master branch of Go (so it should be coming to a stable release soon) I just compiled all of the plugins into the binary and allowed configuration decide which to use.

As of the plugin package, https://tip.golang.org/pkg/plugin/, you can use dynamic linking for plugins, so similar to C's dlopen() in its loading, and the behavior of go's plugin package is pretty well outlined in the documentation.

Additionally I recommend taking a look at how Hashicorp addresses this by doing RPC over a local unix socket. https://github.com/hashicorp/go-plugin

The added benefit of running a plugin as a separate process like Hashicorp's model does, is that you get great stability in the event that the plugin fails but the main process is able to handle that failure.

I should also mention Docker does its plugins in Go similarly, except Docker uses HTTP instead of RPC. Additionally, a Docker engineer has posted about embedding a Javascript interpreter for dynamic logic in the past http://crosbymichael.com/category/go.html.

The issue I wanted to point out with the sql package's pattern that was mentioned in the comments is that that's not a plugin architecture really, you're still limited to whatever is in your imports, so you could have multiple main.go's but that's not a plugin, the point of a plugin is such that the same program can run either one piece of code or another. What you have with things like the sql package is flexibility where a separate package determines what DB driver to use. Nonetheless, you end up modifying code to change what driver you are using.

I want to add, with all of these plugin patterns, aside from the compiling into the same binary and using configuration to choose, each can have its own build, test and deployment (i.e. their own CI/CD) but not necessarily.