Attempting to convert some code to Go CDK when using KMS

Question

I have some code to upload and download files from Google Cloud Storage. Below is an abbreviated example:

import (
    "context"
    "io"
    "cloud.google.com/go/storage"
)

func upload(bucket, keyName, path string, reader io.Reader) error {
    ctx := context.Background()
    client, err := storage.NewClient(ctx)

    if err != nil {
        return err
    }
    defer client.Close()

    obj := client.Bucket(bucket).Object(path)
    writer := obj.NewWriter(ctx)

    defer writer.Close()

    writer.KMSKeyName = keyName

    if _, err = io.Copy(writer, reader); err != nil {
        return err
    }
    if err = writer.Close(); err != nil {
        return err
    }
    return nil
}

The tricky part is that I'm using Google KMS to manage the keys I'm using to encrypt files (Google's so-called "customer-managed encryption key" scheme). My understanding is that this encryption happens on Google's end.

The only solution I found using the Go CDK was to encrypt the files using Google KMS and then upload the encrypted blob. Is there no way to specify the encryption key in the same manner I was doing before with the Go CDK?

Thanks

Answer 1

If you need to use provider-specific settings in the Go CDK, you can use the various As functions to get handles to the underlying provider API. In this case, you would want to use the blob.WriterOptions.BeforeWrite option. The benefit of doing it this way is that the BeforeWrite will no-op if you decide to switch bucket providers later (e.g. for unit testing).

import (
    "context"
    "io"

    "cloud.google.com/go/storage"
    "gocloud.dev/blob"
    _ "gocloud.dev/blob/gcsblob" // link in "gs://" URLs
)

func upload(ctx context.Context, bucket, keyName, path string, reader io.Reader) error {
    bucket, err := blob.OpenBucket(ctx, "gs://" + bucket)
    if err != nil {
        return err
    }
    defer bucket.Close()

    writeCtx, cancelWrite := context.WithCancel(ctx)
    defer cancelWrite()
    writer, err := bucket.NewWriter(writeCtx, path, &blob.WriterOptions{
        // Use BeforeWrite to set provider-specific properties.
        BeforeWrite: func(asFunc func(interface{}) bool) error {
            var gcsWriter *storage.Writer
            // asFunc returns true if the writer can be converted to the type
            // pointed to.
            if asFunc(&gcsWriter) {
                gcsWriter.KMSKeyName = keyName
            }
            return nil
        },
    })
    if err != nil {
        return err
    }
    if _, err := io.Copy(writer, reader); err != nil {
        cancelWrite()  // Abort the write to the bucket.
        writer.Close()
        return err
    }
    if err := writer.Close(); err != nil {
        return err
    }
    return nil
}

(While not directly related to your question, I added in code to abort the write on error to avoid partial objects being uploaded to your storage provider. We're adding docs that will demonstrate how to do common tasks with Go CDK APIs in the future, see #1576.)