Difference between Web Streams and Node.js Stream APIs

Question

I'm building a file upload application to familiarize myself with the concept of streams. I'm trying to turn a file list or blobs into a stream then upload it to the backend and store it on the file system.

I had no issue with frontend and backend implementations respectively but I'm having a hard time connecting the two. My main problem is that I don't understand the difference between the Web Streams API and the Node.js Streams API. I managed to turn the blobs of selected input files in the browser into a Web ReadableStream but the packages I tried (axios for requests, socket.io and socket.io-stream for WebSocket) only accept the Node.js version Stream as arguments. I also could not pipe a Web ReadableStream into a Node.js Writeable or Duplex Stream. The method names are also different (e.g.: pipeTo or pipeThrough in Web API and pipe in Node.js API).

I know there are implementation differences between Node.js and browsers but naively, I thought the APIs would be similar. Can I somehow trivially convert between Web streams and browserified Node.js streams and I'm missing something? Does it worth using the Web Stream API over stream-browserify?

Answer 1

It's not too difficult to convert a web stream to a Node.js stream manually, but you should really try to find a library that accepts native web streams instead of shoehorning a Node.js shim for the stream built-in into the browser with Browserify.

However, if it proves necessary to use a Node.js stream shim in the browser, you need to install stream-browserify and use like this:

import { Readable, Writable } from 'stream-browserify;'

// window.ReadableStream to Node.js Readable
const webRSToNodeRS = rs => {
  const reader = rs.getReader();
  const out = new Readable();
  reader.read().then(async ({ value, done }) => {
    while (!done) {
      out.push(value);
      ({ done, value }) = await reader.read();
    }
    out.push(null);
  });
  return out;
}

// window.WritableStream to Node.js Writable
const webWSToNodeWS = ws => {
  const writer = ws.getWriter();
  const out = new Writable();
  out._write = (chunk, encoding, callback) => {
    writer.write(chunk);
    callback();
  };
  out._final = callback => {
    writer.close();
    callback();
  };
  return out;
}

These methods should be enough to have full interop between web and Node streams. For example, if you want to pipe a web ReadableStream to a Node.js Writable/Duplex:

const pipeWebRSToWritable = (rs, writable) => {
  // After converting you can use the normal pipe methods
  webRSToNodeRS(rs).pipe(writable);
}

However I'd like to mention that you don't need a library to stream data from the client to the server. The fetch API natively supports web streams and is probably the way you should go.

// POST a ReadableStream (perhaps of a file) to the server
// Way easier and more performant than using a 3rd party library...
const postRSToServer = rs => fetch('/your/server/endpoint', {
  method: 'POST',
  body: rs
});

Last note: make sure you're directly using the Blob.prototype.stream method (call this on a File object, e.g. file.stream(), since File extends Blob). There are some ways to get a ReadableStream from a file in JS that actually end up loading all of the file into memory in the browser, which you don't want.

Answer 2

As per docs, since version 16, Node.js includes a (still experimental) implementation of the WebStreams API.

And the old Stream API has now (since v. 17) gained some toWeb/FromWeb methods to bridge both.

Some info here.

Answer 3

For those (like myself) hitting this question in regard to the streams returned by node-fetch, note that undici is a web-compatible implementation of fetch that may eventually become the official node implementation.

Also, see leonbloy's answer RE the new toWeb/fromWeb methods, and WebStreams API.