Asynchronous Reading/Writing to same stream?

Question

Occasionally we need to copy huge files from one bucket to another in AWS S3. Whenever possible we use the CopyRequest to handle this operation all on AWS (since no round trip required back to the client). But sometimes we do not have the option to do this because we need to copy between 2 completely separate accounts which requires a GET and then a PUT.

Problems:

1. The response stream returned from the GET is not seekable so it cannot be passed to the PUT request and have it stream seamlessly from one to the other
2. Copying the response stream to an intermediary stream (MemoryStream) using CopyTo() and then passing that to the PUT operation works well but doesn't scale (large files will throw OutOfMemory exceptions)

So basically I need an intermediary stream that I can read/write to at the same time, basically I would read a chunk from the response stream and write it to my intermediary stream, meanwhile the PUT request is reading out the content and its just a seamless pass-thru sort of scenario.

I found this post on stackoverflow and it seemed promising at first but it still throws an OutOfMemory exception with large files.

.NET Asynchronous stream read/write

Anyone ever had to do something similar to this? How would you tackle it? Thanks in advcance

Answer 1

It's not clear why you would want to use MemoryStream. The Stream.CopyTo method in .NET 4 doesn't need to use an intermediate stream - it will just read into a local buffer of a fixed size, then write that buffer to the output stream, then read more data (overwriting the buffer) etc.

If you're not using .NET 4, it's easy to implement something similar, e.g.

public static void CopyTo(this Stream input, Stream output)
{
    byte[] buffer = new byte[64 * 1024]; // 64K buffer
    int bytesRead;
    while ((bytesRead = input.Read(buffer, 0, buffer.Length)) > 0)
    {
        output.Write(buffer, 0, bytesRead);
    }
}

Answer 2

I found this, but it uses a Queue internally, which the author notes is an order of magnitude slower than a MemoryStream.

http://www.codeproject.com/Articles/16011/PipeStream-a-Memory-Efficient-and-Thread-Safe-Stre

I keep hoping I'll find an official MS library solution, but it seems that this wheel hasn't been properly invented yet.