Node.js piping http.ClientRequest into fs.createWriteStream with Binary Files leads to corrupted/bloated files

Question

I'm having a weird issue with a Node.js (v0.4.12) module I am working on. It's a CouchDB module, and while I'm pretty sure CouchDB is not the problem here, I'll include it as it may be useful information.

Anyways, I'm writing a wrapper for uploading and downloading attachments with CouchDB. I have the upload part working very well, it's the download part I am having difficulty with.

My test process is this:

1. Upload image file (http.request, fs.createReadStream)
2. Download file (http.request)
3. Save downloaded file to different location for comparison/corruption checking (fs.createWriteStream, stream.pipe)

This works fine with plaintext files, but binary files like images behaves differently.

As I mentioned before, the upload appears fine. As I can visit the URL in CouchDB and view the image. (the size matches and it renders the image I uploaded) When I use my module to download the file and save it, the output file is larger than the source file, (50-100% larger) and it cannot be opened by any image editors. (hence, it is "corrupted" somehow)

Am I missing something about Streams when it comes to binary files? I've tried changing the encoding on both sides to "binary" and "base64", but the output file is still 50% larger than the source. Before I discovered the encoding option, it was set to "utf8", and the output file was 100% larger, so that leads me to think this is an encoding issue somewhere in there.

I've also tried manually piping the streams (rather than using the pipe method itself) but the result is the same.

Answer 1

Sorry if I got this question wrong but it seems you are trying to use encodings with a binary file. You don't need to: node gives you raw buffers with binary data by default.

Here is an example of me downloading a clown:

var http = require("http"),
    fs   = require("fs");

http.get({ 
  host: "2.bp.blogspot.com", 
  path: "/_IR8qT_UKjOI/TT2P3qleU9I/AAAAAAAAA3I/beckWSLh3nk/s1600/clown.jpg" },
  function(res) {
    var stream = fs.createWriteStream("clown.jpg");
    res.pipe(stream);
  });

I hope this helps.