How to authenticate a private Go Module using go 1.11 and Google App Engine Standard

Question

I've been updating my entire go gae standard project to use go 1.11's modules.

Main directory structure

app.yaml
app.go
go.mod
go.sum

app.go

package main

import "bitbucket.org/myPrivateRepo"

func main() {
    myImportantModule.Run()
}

go.mod

module myProject

require bitbucket.org/myPrivateRepo v0.0.1

The Error

If I try to gcloud app deploy:

ERROR: (gcloud.app.deploy) Error Response: [9] Cloud build <GUI> 
status: FAILURE.
Build error details: go: bitbucket.org/[email protected]: 
https://api.bitbucket.org/2.0/repositories/myPrivateRepo?fields=scm: 
403 Forbidden

(Note: obviously the repo I'm using has a real name).

So can I do it this way? I'll admit to not fully understanding the migration documentation, particularly when it talked about "Moving files to your GOPATH". https://cloud.google.com/appengine/docs/standard/go111/go-differences

I mean, I thought one of the benefits of the new module system is that you don't need everything under the go path. When I read https://github.com/golang/go/wiki/Modules for example, it very early on says "Create a directory outside of your GOPATH:"

So, to be clear, right now all of my code is outside the go path, but everything builds locally just fine.

I think it all works becausego automatically downloads and caches things within the go path when I run go mod tidy / go build etc.

Yet it fails when I try to gcloud app deploy. How would the google cloud build system ever have access to my private repositories anyway? I'm obviously missing something important. I also read you are not supposed to combine vendoring with the new module system so that can't be it.

I will be very happy if this works, as using DEP forced me to use goapp deploy very awkwardly.

Thanks!

Answer 1

UPDATE: Google has some better documentation now that go 1.14 is out: https://cloud.google.com/appengine/docs/standard/go/specifying-dependencies

My solution:

Instead of dealing with credentials, I'm using go's module replace functionality to point GAE to use my local code. This is working well.

Directory structure:

myService/
    src/
        service.go  // has a run() function to set up routers etc.
        go.mod      // depends on my private module in bitbucket and other things
        …           // other source files
    build/
        gae/
            src/        // simlink to ../../src
            modules/    // git ignored, I clone or copy my modules in build scripts.
            app.go  // see below…
            go.mod  // has main() which calls service.run() and appEngine.Main()
            app.yaml

Method

I use git module replace so that GAE uses my local code. Before building I parse myService/src/go.mod to find the correct version of my private module, then I clone it into the modules folder. I also made an option to copy wip module source code for debugging locally without committing to my module repositories.

go.mod from gae directory:

module myServiceGAE

require (
    bitbucket.org/me/myService v0.0.0
    google.golang.org/appengine v1.4.0
)

replace bitbucket.org/me/myService => ./src

replace bitbucket.org/me/myModule => ./modules/utils

Pros

The package under myService has no references or knowledge of GAE, so I can easily build it into a docker etc. I think parsing the service go.mod files would be like creating my own dependency manager, defeating the benefits of go modules.

Cons

If I had a private module which depended on another private module, I think things would get too complicated.