Uploading file to Google Storage without saving it to memory

Question

I want to upload files from the frontend directly through the backend into a Google Storage bucket, without saving it entirely in memory on the server first. I've added an endpoint similar to the example from the Google docs and it works. However, I'm not sure if this will save the entire file to memory first, since this could lead to issues when uploading larger files.

If it saves the file to memory first, how could I change the code so that it streams the upload directly to Google Storage. The answers to similar questions didn't clarify my question.

Thank you

func Upload(c *gin.Context) {
    file, _, _ := c.Request.FormFile("image")
    ctx := context.Background()

    client, err := storage.NewClient(ctx)
    if err != nil {
        fmt.Printf("Failed to create client with error: %v", err)
        return
    }

    bucket := client.Bucket("test-bucket")

    w := bucket.Object("testfile").NewWriter(ctx)

    w.ContentType = "image/jpeg"

    io.Copy(w, file)
    w.Close()
}

Answer 1

As noted in a comment on question and the answer by Peter, use the multipart reader directly to read the request body.

func Upload(c *gin.Context) {
    mr, err :=  c.Request.MultipartReader()
    if err != nil {
        // handle error
        return
    }
    var foundImage bool
    for {
        p, err := mr.NextPart()
        if err == io.EOF {
            break
        }
        if err != nil {
            // handle error
            return
        }
        if p.FormName() == "image" {
            foundImage = true

            ctx := context.Background()
            client, err := storage.NewClient(ctx)
            if err != nil {
                // handle error
                return
            }
            bucket := client.Bucket("test-bucket")
            w := bucket.Object("testfile").NewWriter(ctx)
            w.ContentType = "image/jpeg"
            if _, err := io.Copy(w, p); err != nil {
              // handle error
              return
            }
            if err := w.Close(); err != nil {
              // handle error
              return
            }
        }
    }
    if !imageFound {
       // handle error
    }
}

Replace the // handle error comments with code that responds to the client with an appropriate error status. It may be useful to log some of the errors as well.

Answer 2

FormFile returns the first file for the provided form key. FormFile calls ParseMultipartForm and ParseForm if necessary.

https://golang.org/pkg/net/http/#Request.FormFile

ParseMultipartForm parses a request body as multipart/form-data. The whole request body is parsed and up to a total of maxMemory bytes of its file parts are stored in memory, with the remainder stored on disk in temporary files.

https://golang.org/pkg/net/http/#Request.ParseMultipartForm

At the time of writing this, FormFile passes 32 MB as the maxMemory argument.

So you with this code you will need up to 32 MB of memory per request, plus googleapi.DefaultUploadChunkSize, which is currently 8 MB, as well as some amount of disk space for everything that doesn't fit in memory.

So uploading will not start until the whole file has been read, but not all of it is kept in memory. If that's not what you want, use Request.MultipartReader instead of ParseMultipartForm:

MultipartReader returns a MIME multipart reader if this is a multipart/form-data or a multipart/mixed POST request, else returns nil and an error. Use this function instead of ParseMultipartForm to process the request body as a stream.