Package version management in Go 1.5

Question

I'm getting my hands dirty with Go, and while I understand and appreciate the principle of simplicity that Go was built upon, I'd like to grasp the rationale behind forgoing a built-in package versioning method in their dependency-fetching tool go get and the import statement.

If I understand correctly, go get and import fetch the package from HEAD and they are unable to refer to a branch or a tag. While there are tools like gopkg.in that circumvent this limitation, the official toolchain:

1. Forces developers to create separate repos for major (breaking) versions of their products.
2. It doesn't allow consumers to downgrade between minor or micro versions in case bugs are found in newer ones.

Truth be told, things are not so easy because package versioning would require a strategy to deal with conflicting transitive dependencies, e.g. X depends on A and B, each of which depend on different versions of C.

Coming from a Java background, it does appear that this limitation poses some risks and problems, amongst others:

1. Product/package evolution and breakage of public APIs of 3rd party deps is unavoidable, therefore versioning must be a first-class citizen in the toolchain IMHO.

2. The Git-repo-per-version policy is highly inefficient:

   • The overall Git history of the package is lost or scattered across repos (merges between versions, backports, etc.)
   • Conflicts with transitive dependencies may still occur, and will go about undetected because the language nor the toolchain impose any semantics to allow detection in the first place.

3. Enterprise adoption may be hindered and development teams may shy away from the language, given that:

   • Always dragging in HEAD means that they can't control or freeze their 3rd party deps, leading to a potentially unpredictable end product.
   • May lack the manpower to keep their product constantly updated and tested with upstream's HEAD (not every company in the world is Google :)).

While I do understand that the latter risk can be – and must be – mitigated with Continuous Integration, it does not solve the underlying root of the problem.

What information am I missing? How do you deal with package upstream changes when deploying Go in an enterprise with limited manpower?

Answer 1

It is being addressed by vendoring which is part of Go 1.5 as an experimental feature, it can be enabled if the go command is run with GO15VENDOREXPERIMENT=1 in its environment, and will be a "full" feature in Go 1.6. Also see Vendor Directories.

The original discussion that led to the Go 1.5 Vedor Experiment can be found here.

The essence of vendoring is that you create a folder named vendor, and you put the exact version of the packages that your code relies on. Code inside the vendor folder is only importable by the code in the directory tree rooted at the parent of vendor, and you can import packages from vendor with an import path as if vendor would be the workspace/src folder (that is, with an import path that omits the prefix up to and including the vendor element).

Example:

/home/user/goworkspace/
    src/
        mymath/
            mymath.go
            vendor/
                github.com/somebob/math
                    math.go

In this example github.com/somebob/math is an external package used by mymath package (from mymath.go). It can be used from mymath.go if it is imported like:

import "github.com/somebob/math"

(And not as import mymath/vendor/github.com/somebob/math which would be bad.)

Answer 2

While Go doesn't come with the standard package manager there are enough options to make builds reproducible (even in an enterprise with limited manpower).

1. Vendoring, which is described in another answer by @icza. This is almost complete equivalent of checking in versioned jar files in Java. This was very common approach with ant build tool before maven became popular. Actually vendoring is much better because you cannot lose source code.

2. This is slight variation of the first option. Instead of checking in vendored source code you can populate vendor folder during build by checking out predefined versions of the dependencies. There tools (e.g. glide) that automate this process.

3. Finally you can maintain predefined versions of all 3-rd party libraries in the internal repository and add it to GOPATH. This approach is described in detail in https://blog.gopheracademy.com/advent-2015/go-in-a-monorepo/

Note that incompatible transitive dependencies are not specific to Go. They exist in Java (and most other languages) too, though Java has a mechanism to partially solve this problem by making programs more complex - class loaders. Note that Go will report all incompatibilities at compile time while in Java some of the incompatibilities are triggered only at run time (because of late linking).

Java toolchain doesn't have concept of versions. It is provided by the external tool - maven. I believe by the time Go becomes more mature and popular a similar standard dependency management tool will emerge.