How to download all dependencies with vgo and a given go.mod?

Question

I'm working on a go project, using vgo, with a Dockerfile, and I'd like docker to cache the dependencies of the project such that it has two properties:

1. If the go.mod file changes, the dependencies will be re-downloaded
2. If I change a file in the package, but do not change go.mod, the dependencies will not be re-downloaded.

Right now, I do:

...
RUN go get -u golang.org/x/vgo
COPY . /go/src/whatever
RUN vgo install
...

But if you change a go file, the dockerfile will have to be rebuilt from the COPY layer onwards.

What I'd like, in short, is to do:

...
RUN go get -u golang.org/x/vgo
COPY go.mod /go/src/whatever
RUN vgo install_dependencies
COPY . /go/src/whatever
RUN vgo install
...

That way, if I change go.mod, all the dependencies will be downloaded and rebuilt, but otherwise, we can proceed right to building the binary.

I can see a couple of ways to get behavior like this, but all of them have drawbacks:

1. I could copy $GOPATH/src/mod into the docker container, but that's going to have lots of files I don't need
2. I could vgo mod -vendor before I build the docker container, and copy the vendor directory, but that relies on the developer to remember to run vgo mod -vendor every time that go.mod changes, otherwise the app won't build and they'll have to run vgo mod -vendor before retrying the docker build.

Can you think of a way for me to get behavior like my imaginary vgo install_dependencies? Am I missing a vgo trick to do so?

Answer 1

tl;dr: In current go master and in future go releases, go mod download will do this job. For now you need a hack.

On the gophers slack, I was referred to this issue: https://github.com/golang/go/issues/26610 where I learned that this would do more or less what I want (assuming here that you're using go version 1.11beta3):

# Populate the module cache based on the go.{mod,sum} files.
COPY go.mod .
COPY go.sum .
RUN go list -e $(go list -f '{{.Path}}' -m all)

Later in the thread, Russ Cox implements go mod download, which will be available in the next release of go, allowing me to remove the silly go list hack above.

update: go mod download is now widely available and you should use it on go 1.12 and later

Answer 2

I had an exact same problem, wanted to make Docker caching more accurate.

So I've just written my own tool for that: https://github.com/gladkikhartem/gomodget

Now my docker-in-docker pipelines are just 10 seconds and the docker build log looks like this:

Step 4/15 : RUN go get -u golang.org/x/vgo
 ---> Using cache
 ---> 12c672a07a16
Step 5/15 : RUN go get github.com/gladkikhartem/gomodget
 ---> Using cache
 ---> acc70fea0edc
Step 7/15 : COPY go.mod .
 ---> Using cache
 ---> 41bae1ca7428
Step 8/15 : RUN gomodget
 ---> Using cache
 ---> 758100f7dde2
Step 9/15 : COPY . .
 ---> cc833c5bc810
Step 10/15 : RUN vgo build -ldflags '-extldflags "-static"' -o /bin/app