Using System.IO.Pipelines together with Stream

Question

I consider replacing Stream-based IO in our application with System.IO.Pipelines to avoid unnecessary memory allocation (considered first RecyclableMemoryStream but it seems to be discontinued). But in some places I still have to use Stream because of the interface imposed by an external library. So my PipeWriter will need wrap its data in a Stream.

I didn't find much on this topic, but found a suggestion to use decorator pattern (C# Stream Pipe (Stream Spy)) in an answer to a different question. I am not sure it would be a right to hide Pipelines behind a Stream wrapper but can't find anything else that will let me piped data to a stream. Am I missing something?

UPDATE. Here's an example using SSH.NET open source library to upload a file to an FTP server (https://gist.github.com/DavidDeSloovere/96f3a827b54f20d52bcfda4fe7a16a0b):

using (var fileStream = new FileStream(uploadfile, FileMode.Open))
    {
        Console.WriteLine("Uploading {0} ({1:N0} bytes)", uploadfile, fileStream.Length);
        client.BufferSize = 4 * 1024; // bypass Payload error large files
        client.UploadFile(fileStream, Path.GetFileName(uploadfile));
    }

Note that we open a FileStream to read a file and then pass a Stream reference to an SftpClient. Can I use System.IO.Pipelines here to reduce memory allocation? I will still need to provide a Stream for SftpClient.

Answer 1

Disclaimer: I'm no expert, just putting together the pieces...

---

The answer (as of Janurary 2019) seems to be: there is no official support for this.

System.IO.Pipelines was created primarily for networking use cases. In fact, the pipelines code released in 2.1 had no support for any endpoints:

Here we need a bit of caveat and disclaimer: the pipelines released in .NET Core 2.1 do not include any endpoint implementations.

There is a proposed design for an API for generic stream adapter but that is part of the .NET Core 3.0 Milestone.

There even seems to be some reticence to implementing file-based pipelines access (AKA a FileStream pipelines equivalent). This is particularly disappointing since I too was hoping for pipelines powered file I/O.

I think your best bet at the moment is using the UsePipe() methods in https://github.com/AArnott/Nerdbank.Streams

---

Update: Here's another example I just found https://github.com/tulis/system-io-pipelines-demo/tree/master/src/SystemIoPipelinesDemo/SystemIoPipelinesDemo

---

Update: I had a go at making a Pipeline based file reader. You can read all about it here: https://github.com/atruskie/Pipelines.File.Unofficial

Essentially, from a performance perspective, using a pipeline stream adapter like Nerdbank.Streams is a good way to go!