node.js axios download file stream and writeFile

Question

i want download a pdf file with axios and save on disk (server side) with fs.writeFile, i have tried:

axios.get('https://xxx/my.pdf', {responseType: 'blob'}).then(response => {
    fs.writeFile('/temp/my.pdf', response.data, (err) => {
        if (err) throw err;
        console.log('The file has been saved!');
    });
});

the file is saved but the content is broken...

how do I correctly save the file?

Answer 1

You can simply use response.data.pipe and fs.createWriteStream to pipe response to file

axios({
    method: "get",
    url: "https://xxx/my.pdf",
    responseType: "stream"
}).then(function (response) {
    response.data.pipe(fs.createWriteStream("/temp/my.pdf"));
});

Answer 2

Actually, I believe the previously accepted answer has some flaws, as it will not handle the writestream properly, so if you call "then()" after Axios has given you the response, you will end up having a partially downloaded file.

This is a more appropriate solution when downloading slightly larger files:

export async function downloadFile(fileUrl: string, outputLocationPath: string) {
  const writer = createWriteStream(outputLocationPath);

  return Axios({
    method: 'get',
    url: fileUrl,
    responseType: 'stream',
  }).then(response => {

    //ensure that the user can call `then()` only when the file has
    //been downloaded entirely.

    return new Promise((resolve, reject) => {
      response.data.pipe(writer);
      let error = null;
      writer.on('error', err => {
        error = err;
        writer.close();
        reject(err);
      });
      writer.on('close', () => {
        if (!error) {
          resolve(true);
        }
        //no need to call the reject here, as it will have been called in the
        //'error' stream;
      });
    });
  });
}

This way, you can call downloadFile(), call then() on the returned promise, and making sure that the downloaded file will have completed processing.

Or, if you use a more modern version of NodeJS, you can try this instead:

import * as stream from 'stream';
import { promisify } from 'util';

const finished = promisify(stream.finished);

export async function downloadFile(fileUrl: string, outputLocationPath: string): Promise<any> {
  const writer = createWriteStream(outputLocationPath);
  return Axios({
    method: 'get',
    url: fileUrl,
    responseType: 'stream',
  }).then(response => {
    response.data.pipe(writer);
    return finished(writer); //this is a Promise
  });
}

Answer 3

The problem with broken file is because of backpressuring in node streams. You may find this link useful to read: https://nodejs.org/es/docs/guides/backpressuring-in-streams/

I'm not really a fan of using Promise base declarative objects in JS codes as I feel it pollutes the actual core logic & makes the code hard to read. On top of it, you have to provision event handlers & listeners to make sure the code is completed.

A more cleaner approach on the same logic which the accepted answer proposes is given below. It uses the concepts of stream pipelines.

const util = require('util');
const stream = require('stream');
const pipeline = util.promisify(stream.pipeline);

const downloadFile = async () => {
  try {
    const request = await axios.get('https://xxx/my.pdf', {
      responseType: 'stream',
    });
    await pipeline(request.data, fs.createWriteStream('/temp/my.pdf'));
    console.log('download pdf pipeline successful');   
  } catch (error) {
    console.error('download pdf pipeline failed', error);
  }
}

exports.downloadFile = downloadFile

I hope you find this useful.

Answer 4

// This works perfectly well! 
const axios = require('axios'); 

axios.get('http://www.sclance.com/pngs/png-file-download/png_file_download_1057991.png', {responseType: "stream"} )  
.then(response => {  
// Saving file to working directory  
    response.data.pipe(fs.createWriteStream("todays_picture.png"));  
})  
    .catch(error => {  
    console.log(error);  
});