What's the fastest way to parse Node.JS buffer stream chunks' binary data into structures?

Question

Googling for "parse nodejs binary stream" I see lots of examples when the emitted data fits neatly into the expected return size, but zero examples of how to handle the case when the next chunk contains the remaining bytes from the first structure and the new header.

What's the "right" way to parse binary streams when I expect something like:

record-length: 4bytes data1: 8bytes data2: 8bytes 4-byte-records[(record-length - 16) * 4];

The data will come in as various sized chunks. But is there a way to call data.readUInt32(0) and have it wait for the chunk to fill? I'd hate to have to write a pipe stage that emits bytes and a receiving state machine, that seems so very wrong.

This has got to be solved, it is such a basic concept.

Help?

Thanks, PT

Answer 1

Hmm... thats's something that can be solved using the async version of stream..read and a transform stream.

Now you can write (and it will probably be fun) your own version, but the framework I wrote, scramjet already has that async read and I gather you'd want to make this easy.

Here's the easiest I can think of, using AsyncGenerator:

const {BufferStream} = require('scramjet'); // or es6 import;
const input = BufferStream.from(getStreamFromSomewhere());

const output = DataStream.from(async function* () {
  while(true) {
    const recordLength = (await input.whenRead(4)).readUInt32(0);  // read next chunk length
    if (!recordLength) return;                                     // stream ends here;
    const data1 = await input.whenRead(8);
    const data2 = await input.whenRead(8);
    const restOfData = [];
    for (let i = 0; i < recordLength; i += 4)
      restOfData.push((await input.read(4)).readUInt32(0))

    yield {data1, data2, restOfData};
  }
})
  .catch(e => output.end()); // this is a handler for an option where any of the reads past
                             // recordLength was to return null - perhaps should be better.

This is super easy in node v10 or with babel, but if you like I can add the non AsyncGenerator version here.