Node: Downloading a zip through Request, Zip being corrupted

Question

I'm using the excellent Request library for downloading files in Node for a small command line tool I'm working on. Request works perfectly for pulling in a single file, no problems at all, but it's not working for ZIPs.

For example, I'm trying to download the Twitter Bootstrap archive, which is at the URL:

http://twitter.github.com/bootstrap/assets/bootstrap.zip

The relevant part of the code is:

var fileUrl = "http://twitter.github.com/bootstrap/assets/bootstrap.zip";
var output = "bootstrap.zip";
request(fileUrl, function(err, resp, body) {
  if(err) throw err;
  fs.writeFile(output, body, function(err) {
    console.log("file written!");
  }
}

I've tried setting the encoding to "binary" too but no luck. The actual zip is ~74KB, but when downloaded through the above code it's ~134KB and on double clicking in Finder to extract it, I get the error:

Unable to extract "bootstrap" into "nodetest" (Error 21 - Is a directory)

I get the feeling this is an encoding issue but not sure where to go from here.

Answer 1

Yes, the problem is with encoding. When you wait for the whole transfer to finish body is coerced to a string by default. You can tell request to give you a Buffer instead by setting the encoding option to null:

var fileUrl = "http://twitter.github.com/bootstrap/assets/bootstrap.zip";
var output = "bootstrap.zip";
request({url: fileUrl, encoding: null}, function(err, resp, body) {
  if(err) throw err;
  fs.writeFile(output, body, function(err) {
    console.log("file written!");
  });
});

Another more elegant solution is to use pipe() to point the response to a file writable stream:

request('http://twitter.github.com/bootstrap/assets/bootstrap.zip')
  .pipe(fs.createWriteStream('bootstrap.zip'))
  .on('close', function () {
    console.log('File written!');
  });

A one liner always wins :)

pipe() returns the destination stream (the WriteStream in this case), so you can listen to its close event to get notified when the file was written.

Answer 2

I was searching about a function which request a zip and extract it without create any file inside my server, here is my TypeScript function, it use JSZIP module and Request:

let bufs : any = [];
let buf : Uint8Array;
request
    .get(url)
    .on('end', () => {
        buf = Buffer.concat(bufs);

        JSZip.loadAsync(buf).then((zip) => {
            // zip.files contains a list of file
            // chheck JSZip documentation
            // Example of getting a text file : zip.file("bla.txt").async("text").then....
        }).catch((error) => {
            console.log(error);
        });
    })
    .on('error', (error) => {
        console.log(error);
    })
    .on('data', (d) => {
        bufs.push(d);
    })