io.Copy cause out of memory in golang

Question

I use io.Copy() to copy a file, about 700Mb, but it cause out of memory

bodyBuf := &bytes.Buffer{}
bodyWriter := multipart.NewWriter(bodyBuf)

//key step
fileWriter, err := bodyWriter.CreateFormFile(paramName, fileName)
if err != nil {
    return nil, err
}

file, err := os.Open(fileName) //the file size is about 700Mb
if err != nil {
    return nil, err
}
defer file.Close()

//iocopy
copyLen, err := io.Copy(fileWriter, file) // this cause out of memory
if err != nil {
    fmt.Println("io.copy(): ", err)

    return nil, err
}

The error message as follow:

runtime: memory allocated by OS (0x752cf000) not in usable range [0x18700000,0x98700000)
runtime: out of memory: cannot allocate 1080229888-byte block (1081212928 in use)
fatal error: out of memory

I allocate enough memory for buf, it cause out of memory in bodyWriter.CreateFormFile()

buf := make([]byte, 766509056)
bodyBuf := bytes.NewBuffer(buf)
bodyWriter := multipart.NewWriter(bodyBuf)

fileWriter, err := bodyWriter.CreateFormFile(paramName, fileName) // out of memory
if err != nil {
    return nil, err
}

Answer 1

That's because you are 'copying', to bodyBuf, which is an in-memory buffer, forcing Go to try an allocate a block of memory as big as the entire file.

Based on your use of multipart it looks like you are trying to stream the file over http? In that case, don't pass a bytes.Buffer to multipart.NewWriter, directly pass your http connection instead.