Stream File Directly to s3 using NodeJS+Express, aws-sdk

Question

I want to upload some large files directly to s3 via the browser with NodeJS, it is unclear how to prepare this file for upload to s3. There might be a better module (like Knox) to handle this case but I am not sure. Any thoughts?

File Object

  file: { 
     webkitRelativePath: '',
     lastModifiedDate: '2013-06-22T02:43:54.000Z',
     name: '04-Bro Safari & UFO! - Animal.mp3',
     type: 'audio/mp3',
     size: 11082039 
  }

S3 putObject

var params = {Bucket: 'bucket_name/'+req.user._id+'/folder', Key: req.body['file']['name'], Body: ???};
s3.putObject(params, function(err, data) {
    if (err)
      console.log(err);
    else
      console.log("Successfully uploaded data to myBucket/myKey");
});    

Answer 1

Streaming is now supported (see docs), simply pass the stream as the Body:

var fs = require('fs');
var someDataStream = fs.createReadStream('bigfile');
var s3 = new AWS.S3({ params: { Bucket: 'myBucket', Key: 'myKey' } });
s3.putObject({ Body: someDataStream, ... }, function(err, data) {
  // handle response
})

Answer 2

The s3.putObject() method does not stream, and from what I see, the s3 module doesn't support streaming. However, with Knox, you can use Client.putStream(). Using the file object from your question, you can do something like this:

var fs = require('fs');
var knox = require('knox');

var stream = fs.createReadStream('./file');
var client = knox.createClient({
  key: '<api-key-here>',
  secret: '<secret-here>',
  bucket: 'learnboost'
});

var headers = {
  'Content-Length': file.size,
  'Content-Type': file.type
};

client.putStream(stream, '/path.ext', headers, function(err, res) {
  // error or successful upload
});

Answer 3

One option is to use multer-s3 instead: https://www.npmjs.com/package/multer-s3.

This post has some details also: Uploading images to S3 using NodeJS and Multer. How to upload whole file onFileUploadComplete