There are a number of tutorials about posting files using http.Request
in go, but almost invariably they start like this:
file, err := os.Open(path)
if err != nil {
return nil, err
}
fileContents, err := ioutil.ReadAll(file)
Which is to say, you read the entire file into memory, and then convert it into a Buffer
and pass that into a request, something like this:
func send(client *http.Client, file *os.File, endpoint string) {
body := &bytes.Buffer{}
io.Copy(body, file)
req, _ := http.NewRequest("POST", endpoint, body)
resp, _ := client.Do(req)
}
If you wanted to post a massive file and avoid reading it into memory, but instead steam the file up in chunks... how would you do that?
Go to localhost:8080/upload, and you will see a form to upload a file. After selecting a file and clicking upload, the file should be created in your local filesystem.
A stream file is a randomly accessible sequence of bytes, with no further structure imposed by the system. The integrated file system provides support for storing and operating on information in the form of stream files. Documents that are stored in your system's folders are stream files.
MaxBytesReader() method is used to limit the size of incoming request bodies. For single file uploads, limiting the size of the request body provides a good approximation of limiting the file size. The ParseMultipartForm() method subsequently parses the request body as multipart/form-data up to the max memory argument.
If you need to set Content-Length
, it can be done manually. The following snippet is an example of uploading file and extra parameters as a stream (the code based on Buffer-less Multipart POST in Golang)
//NOTE: for simplicity, error check is omitted
func uploadLargeFile(uri, filePath string, chunkSize int, params map[string]string) {
//open file and retrieve info
file, _ := os.Open(filePath)
fi, _ := file.Stat()
defer file.Close()
//buffer for storing multipart data
byteBuf := &bytes.Buffer{}
//part: parameters
mpWriter := multipart.NewWriter(byteBuf)
for key, value := range params {
_ = mpWriter.WriteField(key, value)
}
//part: file
mpWriter.CreateFormFile("file", fi.Name())
contentType := mpWriter.FormDataContentType()
nmulti := byteBuf.Len()
multi := make([]byte, nmulti)
_, _ = byteBuf.Read(multi)
//part: latest boundary
//when multipart closed, latest boundary is added
mpWriter.Close()
nboundary := byteBuf.Len()
lastBoundary := make([]byte, nboundary)
_, _ = byteBuf.Read(lastBoundary)
//calculate content length
totalSize := int64(nmulti) + fi.Size() + int64(nboundary)
log.Printf("Content length = %v byte(s)\n", totalSize)
//use pipe to pass request
rd, wr := io.Pipe()
defer rd.Close()
go func() {
defer wr.Close()
//write multipart
_, _ = wr.Write(multi)
//write file
buf := make([]byte, chunkSize)
for {
n, err := file.Read(buf)
if err != nil {
break
}
_, _ = wr.Write(buf[:n])
}
//write boundary
_, _ = wr.Write(lastBoundary)
}()
//construct request with rd
req, _ := http.NewRequest("POST", uri, rd)
req.Header.Set("Content-Type", contentType)
req.ContentLength = totalSize
//process request
client := &http.Client{}
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
} else {
log.Println(resp.StatusCode)
log.Println(resp.Header)
body := &bytes.Buffer{}
_, _ = body.ReadFrom(resp.Body)
resp.Body.Close()
log.Println(body)
}
}
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With