createReadStream not working/extremely slow for large files

Question

Im using DropBox API to upload files. To upload the files to dropbox I am going through the following steps:

1. First upload file from form to a local directory on the server.
2. Read File from local directory using fs.createReadStream
3. Send file to Dropbox via the dropbox API.

The issue:

For some reason fs.createReadStream takes absolute ages when reading and uploading a large file. Now the file I'm trying to upload is only 12MB which is not a big file and it takes approximately 18mins to upload/process a 12MB file.

I don't know where the issue is either it's in createReadStream or dropbox api code.

It works with files of size within kb.

My Code:

let options = {
    method: 'POST',
    uri: 'https://content.dropboxapi.com/2/files/upload',
    headers: {
        'Authorization': 'Bearer TOKEN HERE',
        'Dropbox-API-Arg': "{\"path\": \"/test/" + req.file.originalname + "\",\"mode\": \"overwrite\",\"autorename\": true,\"mute\": false}",
        'Content-Type': 'application/octet-stream'
    }, 

      // I think the issue is here.
      body: fs.createReadStream(`uploads/${req.file.originalname}`)
};

 rp(options)
    .then(() => {
        return _deleteLocalFile(req.file.originalname)
    })
    .then(() => {
        return _generateShareableLink(req.file.originalname)
    })
    .then((shareableLink) => {
        sendJsonResponse(res, 200, shareableLink)
    })
    .catch(function (err) {
        sendJsonResponse(res, 500, err)
    });

Update:

const rp = require('request-promise-native');

Answer 1

I had an experience similar to this issue before and after a large amount of head scratching and digging around, I was able to resolve the issue, in my case anyway.

For me, the issue arose due to the default chunking size for createReadStream() being quite small 64kb and this for some reason having a knock on effect when uploading to Dropbox.

The solution therefore was to increase the chunk size.

// Try using chunks of 256kb
body: fs.createReadStream(`uploads/${req.file.originalname}`, {highWaterMark : 256 * 1024});