How do Go plugin dependencies work?

Question

Go 1.8 supports Go plugins.

I have created two plugins as follows.

As I understand, the plugin exposes only the functions and variables in the main package. i.e. plugin.Lookup() will fail for non-main variable/function.

But I wanted to test if a plugin can internally invoke a method from another plugin, similar to how a C++ library can invoke another library.

So I tested as follows:

plugin1 github.com/vimal/testplugin

$ cat myplugin.go
package main

import "C"
import "fmt"
import help "github.com/vimal/testplugin1/plug"

func init() {
        fmt.Printf("main.init invoked\n")
}
// TestPlugin 
func TestPlugin() string {
        return help.Help()
}

plugin2 github.com/vimal/testplugin1

$ cat myplugin.go
package main

import "C"

func HelperFunc() string {
        return "help"
}
$ cat plug/helper.go
package help

func Help() string {
        return "help234"
}

Idea here is that plugin1 invokes an internal, non-main function of plugin2.

main program

Main program loads a number of plugins given as arguments, and invokes TestPlugin() from the last plugin.

test 1:

Build both plugins, and load both plugins, and invoke TestPlugin(), the output contains "help234", i.e. the inner function is invoked. This can be understood, since both plugins are loaded, one plugin can invoke another plugin's inner code.

test 2:

Load only plugin1, and invoke TestPlugin(), the output contains "help234", i.e. the inner function is invoked. The same output is observed as in test1. Perhaps this time the method is found from GOPATH.

test 3:

Rename the folder "github.com/vimal/testplugin1" to "github.com/vimal/junk1", delete the plugin2, and load only plugin1, and invoke TestPlugin(). the output still contains "help234", i.e. the inner function is invoked.

I am not able to understand how test3 produces the same output. Does plugin1 contain plugin2 code also? How can I understand the Go plugin dependency on other Go plugins?

Go version : go version go1.8rc3 linux/amd64

Answer 1

You're not doing exactly what you think you are.

Your plugin1 imports and uses a package, namely github.com/vimal/testplugin1/plug. This is not "equal" to plugin2!

What happens here is that when you build plugin1, all its dependencies are built into the plugin file, including the .../testplugin1/plug package. And when you load plugin1, all its dependencies are also loaded from the plugin file, including the plug package. After this, it's not surprising that it works no matter the loaded status of plugin2. These 2 plugins are independent from each another.

The -buildmode=plugin instructs the compiler that you want to build a plugin and not a standalone app, but it doesn't mean dependencies must not be included. They have to be, because the plugin cannot have any guarantee what Go app will load it, and what packages that Go app will have. Because a runnable app also only contains packages even from the standard library that the app itself references explicitly.

The only way to guarantee that the plugin will have everything it needs and so that it will work if it also contains all its dependencies, including those from the standard library. (This is the reason why building simple plugins generate relatively big files, similarly to building simple Go executables resulting in big files.)

Few things that don't need to be added to plugins include the Go runtime for example, because a running Go app that will load the plugin will already have a Go runtime running. (Note that you can only load a plugin from an app compiled with the same version of Go.) But beyond that, the plugin has to contain everything it needs.

Go is a statically linked language. Once a Go app or plugin is compiled, they do not rely nor check the value of GOPATH, it is only used by the Go tool during building them.

Deeper insight

It's possible that your main app and a plugin refers to the same package ("same" by import path). In such cases only one "instance" of the package will be used.

This can be tested if this commonly referred package has "state", e.g a global variable. Let's assume a common, shared package called mymath:

package mymath

var S string

func SetS(s string) {
    S = s
}

And a plugin called pg that uses it:

package main

import (
    "C"
    "mymath"
    "fmt"
)

func Start() {
    fmt.Println("pg:mymath.S", mymath.S)
    mymath.SetS("pghi")
    fmt.Println("pg:mymath.S", mymath.S)
}

And the main app that uses mymath and loads pg (which uses it):

package main

import (
    "plugin"
    "mymath"
    "fmt"
)

func main() {
    fmt.Println("mymath.S", mymath.S)
    mymath.SetS("hi")
    fmt.Println("mymath.S", mymath.S)

    p, err := plugin.Open("../pg/pg.so")
    if err != nil {
        panic(err)
    }

    start, err := p.Lookup("Start")
    if err != nil {
        panic(err)
    }

    start.(func())()

    fmt.Println("mymath.S", mymath.S)
}

Building the plugin:

cd pg
go build -buildmode=plugin

Running the main app, the output is:

mymath.S 
mymath.S hi
pg:mymath.S hi
pg:mymath.S pghi
mymath.S pghi

Analysis: first the main app plays with mymath.S, sets it to "hi" eventually. Then comes the plugin, which prints it (we see the value set by the main app "hi"), then changes it to "pghi". Then comes again the main app and prints mymath.S, and again, sees the last value set by the plugin: "pghi".

So there is only one "instance" of mymath. Now if you go ahead and change mymath, e.g. rename myMath.SetS() to mymath.SetS2(), and you update the call in the main app (to mymath.SetS2("hi")), and without rebuilding the plugin, just running the main app, you get the following output:

mymath.S 
mymath.S hi
panic: plugin.Open: plugin was built with a different version of package mymath

goroutine 1 [running]:
main.main()
    <GOPATH>/src/play/play.go:16 +0x4b5
exit status 2

As you can see, when building the main app and the plugin, the package version (which is most likely a hash) is recorded, which must match if their import paths match in the main app and in the plugin.

(Note that you will also get the above error if you don't change the exported identifiers (and signatures) of the used mymath package, only the implementation; e.g. func SetS(s string) { S = s + "+" }.)