How do I go from io.ReadCloser to io.ReadSeeker?

Question

I'm trying to download a file from S3 and upload that file to another bucket in S3. Copy API won't work here because I've been told not to use it.

Getting an object from S3 has a response.Body that's an io.ReadCloser and to upload that file, the payload takes a Body that's an io.ReadSeeker.

The only way I can figure this out is by saving the response.Body to a file then passing that file as a io.ReadSeeker. This would require writing the entire file to disk first then reading the entire file from disk which sounds pretty wrong.

What I would like to do is:

resp, _ := conn.GetObject(&s3.GetObjectInput{Key: "bla"}) conn.PutObject(&s3.PutObjectInput{Body: resp.Body}) // resp.Body is an io.ReadCloser and the field type expects an io.ReadSeeker 

Question is, how do I go from an io.ReadCloser to an io.ReadSeeker in the most efficient way possible?

Answer 1

io.ReadSeeker is the interface that groups the basic Read() and Seek() methods. The definition of the Seek() method:

Seek(offset int64, whence int) (int64, error) 

An implementation of the Seek() method requires to be able to seek anywhere in the source, which requires all the source to be available or reproducible. A file is a perfect example, the file is saved permanently to your disk and any part of it can be read at any time.

response.Body is implemented to read from the underlying TCP connection. Reading from the underlying TCP connection gives you the data that the client at the other side sends you. The data is not cached, and the client won't send you the data again upon request. That's why response.Body does not implement io.Seeker (and thus io.ReadSeeker either).

So in order to obtain an io.ReadSeeker from an io.Reader or io.ReadCloser, you need something that caches all the data, so that upon request it can seek to anywhere in that.

This caching mechanism may be writing it to a file as you mentioned, or you can read everything into memory, into a []byte using ioutil.ReadAll(), and then you can use bytes.NewReader() to obtain an io.ReadSeeker from a []byte. Of course this has its limitations: all the content must fit into memory, and also you might not want to reserve that amount of memory for this file copy operation.

All in all, an implementation of io.Seeker or io.ReadSeeker requires all the source data to be available, so your best bet is writing it to a file, or for small files reading all into a []byte and streaming the content of that byte slice.