How to unit test google cloud storage in golang?

Question

I'm writing an appengine app in Go that uses Google cloud storage.

For example, my "reading" code looks like:

client, err := storage.NewClient(ctx)
if err != nil {
    return nil, err
}
defer func() {
    if err := client.Close(); err != nil {
        panic(err)
    }
}()
r, err := client.Bucket(BucketName).Object(id).NewReader(ctx)
if err != nil {
    return nil, err
}
defer r.Close()
return ioutil.ReadAll(r)

... where ctx is a context from appengine.

When I run this code in a unit test (using aetest), it actually sends requests to my cloud storage; I'd like to run this hermetically instead, similar to how aetest allows fake datastore calls.

(Possibly related question, but it deals with python, and the linked github issue indicates it's solved in a python-specific way).

How can I do this?

Answer 1

One approach, also suggested here is to allow your GCS client to have its downloader swapped out for a stub while unit testing. First, define an interface that matches how you use the Google Cloud Storage library, and then reimplement it with fake data in your unit tests.

Something like this:

type StorageClient interface {
  Bucket(string) Bucket  // ... and so on, matching the Google library
}

type Storage struct {
  client StorageClient
}

// New creates a new Storage client
// This is the function you use in your app
func New() Storage {
  return NewWithClient(&realGoogleClient{}) // provide real implementation here as argument
}

// NewWithClient creates a new Storage client with a custom implementation
// This is the function you use in your unit tests
func NewWithClient(client StorageClient) {
  return Storage{
    client: client,
  }
}

It can be a lot of boilerplate to mock entire 3rd party APIs, so maybe you'll be able to make it easier by generating some of those mocks with golang/mock or mockery.

Answer 2

I have done something like this...

Since storage client is sending HTTPS request so I mocked the HTTPS server using httptest

func Test_StorageClient(t *testing.T) {
    tests := []struct {
        name        string
        mockHandler func() http.Handler
        wantErr     bool
    }{
        {
            name: "test1",
            mockHandler: func() http.Handler {
                return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
                    w.Write([]byte("22\n96\n120\n"))
                    return
                })
            },
            wantErr: false,
        },
        {
            name: "test2 ",
            mockHandler: func() http.Handler {
                return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
                    w.WriteHeader(http.StatusNotFound)
                    return
                })
            },
            wantErr: true,
        },
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            serv := httptest.NewTLSServer(tt.mockHandler())
            httpclient := http.Client{
                Transport: &http.Transport{
                    TLSClientConfig: &tls.Config{
                        InsecureSkipVerify: true,
                    },
                },
            }
            client, _ := storage.NewClient(context.Background(), option.WithEndpoint(serv.URL), option.WithoutAuthentication(), option.WithHTTPClient(&httpclient))
            got, err := readFileFromGCS(client)
            if (err != nil) != tt.wantErr {
                t.Errorf("error = %v, wantErr %v", err, tt.wantErr)
                return
            }
        })
    }
}

Answer 3

Cloud Storage on the Python Development server is emulated using local files with the Blobstore service, which is why the solution of using a Blobstore stub with testbed (also Python-specific) worked. However there is no such local emulation for Cloud Storage on the Go runtime.

As Sachin suggested, the way to unit test Cloud Storage is to use a mock. This is the way it's done internally and on other runtimes, such as node.