Pushing binary data to Amazon S3 using Node.js

Question

I'm trying to take an image and upload it to an Amazon S3 bucket using Node.js. In the end, I want to be able to push the image up to S3, and then be able to access that S3 URL and see the image in a browser. I'm using a Curl query to do an HTTP POST request with the image as the body.

curl -kvX POST --data-binary "@test.jpg" 'http://localhost:3031/upload/image'

Then on the Node.js side, I do this:

exports.pushImage = function(req, res) {
    var image = new Buffer(req.body);
    var s3bucket = new AWS.S3();
    s3bucket.createBucket(function() {
        var params = {Bucket: 'My/bucket', Key: 'test.jpg', Body: image};
        // Put the object into the bucket.
        s3bucket.putObject(params, function(err) {
            if (err) {
                res.writeHead(403, {'Content-Type':'text/plain'});
                res.write("Error uploading data");
                res.end()
            } else {
                res.writeHead(200, {'Content-Type':'text/plain'});
                res.write("Success");
                res.end()
            }
        });
    });
};

My file is 0 bytes, as shown on Amazon S3. How do I make it so that I can use Node.js to push the binary file up to S3? What am I doing wrong with binary data and buffers?

UPDATE:

I found out what I needed to do. The curl query is the first thing that should be changed. This is the working one:

curl -kvX POST -F foobar=@my_image_name.jpg 'http://localhost:3031/upload/image'

Then, I added a line to convert to a Stream. This is the working code:

exports.pushImage = function(req, res) {
    var image = new Buffer(req.body);
    var s3bucket = new AWS.S3();
    s3bucket.createBucket(function() {
        var bodyStream = fs.createReadStream(req.files.foobar.path);
        var params = {Bucket: 'My/bucket', Key: 'test.jpg', Body: bodyStream};
        // Put the object into the bucket.
        s3bucket.putObject(params, function(err) {
            if (err) {
                res.writeHead(403, {'Content-Type':'text/plain'});
                res.write("Error uploading data");
                res.end()
            } else {
                res.writeHead(200, {'Content-Type':'text/plain'});
                res.write("Success");
                res.end()
            }
        });
    });
};

So, in order to upload a file to an API endpoint (using Node.js and Express) and have the API push that file to Amazon S3, first you need to perform a POST request with the "files" field populated. The file ends up on the API side, where it resides probably in some tmp directory. Amazon's S3 putObject method requires a Stream, so you need to create a read stream by giving the 'fs' module the path where the uploaded file exists.

I don't know if this is the proper way to upload data, but it works. Does anyone know if there is a way to POST binary data inside the request body and have the API send that to S3? I don't quite know what the difference is between a multi-part upload vs a standard POST to body.

Answer 1

I believe you need to pass the content-length in the header as documented on the S3 docs: http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html

After spending quite a bit of time working on pushing assets to S3, I ended up using the AwsSum library with excellent results in production:

https://github.com/awssum/awssum-amazon-s3/

(See the documentation on setting your AWS credentials)

Example:

var fs = require('fs');
var bucket_name = 'your-bucket name'; // AwsSum also has the API for this if you need to create the buckets

var img_path = 'path_to_file';
var filename = 'your_new_filename';

// using stat to get the size to set contentLength
fs.stat(img_path, function(err, file_info) {

    var bodyStream = fs.createReadStream( img_path );

    var params = {
        BucketName    : bucket_name,
        ObjectName    : filename,
        ContentLength : file_info.size,
        Body          : bodyStream
    };

    s3.putObject(params, function(err, data) {
        if(err) //handle
        var aws_url = 'https://s3.amazonaws.com/' + DEFAULT_BUCKET + '/' + filename;
    });

});

UPDATE

So, if you are using something like Express or Connect which are built on Formidable, then you don't have access to the file stream as Formidable writes files to disk. So depending on how you upload it on the client side the image will either be in req.body or req.files. In my case, I use Express and on the client side, I post other data as well so the image has it's own parameter and is accessed as req.files.img_data. However you access it, that param is what you pass in as img_path in the above example.

If you need to / want to Stream the file that is trickier, though certainly possible and if you aren't manipulating the image you may want to look at taking a CORS approach and uploading directly to S3 as discussed here: Stream that user uploads directly to Amazon s3