Buffer.toString('binary') adds "new characters"

Question

I'm trying to compress/decompress this file: Sample of contents of sum (which can be downloaded The Canterbury Corpus).

When I read it using fs.readFile(path, callback), I receive a Buffer and I need to convert this buffer to a string in order to compress, and after that, need to save it again as a file.

The problem is that the Buffer.toString('binary') seems to be modifying the content.

If I run

console.log('original: ', sumBuffer)
let newBuffer = Buffer.from(sumBuffer.toString('binary'))
console.log('modified: ', newBuffer)

I get this:

original: <Buffer 7f 45 4c 46 01 02 01 00 00 00 00 00 00 00 00 00 00 02 00 02 00 00 00 01 00 01 0b 98 00 00 00 34 00 00 91 50 00 00 00 00 00 34 00 20 00 05 00 28 00 1a ... >

modified: <Buffer 7f 45 4c 46 01 02 01 00 00 00 00 00 00 00 00 00 00 02 00 02 00 00 00 01 00 01 0b c2 98 00 00 00 34 00 00 c2 91 50 00 00 00 00 00 34 00 20 00 05 00 28 ... >

In the second one, there are some c2 character appearing where before had nothing.

modified: <Buffer 7f 45 4c 46 01 02 01 00 00 00 00 00 00 00 00 00 00 02 00 02 00 00 00 01 00 01 0b >>c2<< 98 00 00 00 34 00 00 >>c2<< 91 50 00 00 00 00 00 34 00 20 00 05 00 28 ... >

Why is this happening? How can I get rid of this?

Thank you in advance.

Answer 1

Buffer.from function with a string as the first argument takes a second optional encoding argument, which defaults to utf8

So, you code is equivalent to

let newBuffer = Buffer.from(sumBuffer.toString('binary'), 'utf8');

Since sumBuffer is clearly binary, what you'll want to do is

let newBuffer = Buffer.from(sumBuffer.toString('binary'), 'binary');

Note, this is the same as

let newBuffer = Buffer.from(sumBuffer);

However, as you've pointed out, the "real code" is a little more complex - I just felt the need to point it out in case some numpty comes along and says "but why don't you just ... " :p