Listing all the directories and all the files and uploading them to my bucket (S3 Amazon) with Node.JS

Question

Code below:

I'm using the findit walker, documentation here -> https://github.com/substack/node-findit

With this package i'm listing all the directories and files of my application, and i'm trying to send to my bucket on Amazon S3 (with my own code).

I'm not sure if the code is right, and i don't know what i need to put in the Body, inside the params object.

This part it's listening all the Directories of my app:

finder.on('directory', function (dir, stat, stop) {
    var base = path.basename(dir);
    if (base === '.git' || base === 'node_modules' || base === 'bower_components') {
        stop();
    }
    else {
        console.log(dir + '/');
    }
});

And this one it's listening all the files of my app:

finder.on('file', function (file, stat) {
  console.log(file);
});

I updated it to send data to my bucket, like this:

finder.on('file', function (file, stat) {
    console.log(file);
    var params = {
        Bucket: BUCKET_NAME,
        Key: file,
        //Body:
    };
    //console.log(params.body);


    s3.putObject(params, function(err) {
        if(err) {
            console.log(err);
        }
        else {
            console.log("Success!");
        }
    });
});

I really don't know what i need to put inside the Body, and i don't know if the code is right. Anyone could help me?

Thanks.

to help, all code, all the code:

var fs = require('fs');
var finder = require('findit')(process.argv[2] || '.');
var path = require('path');
var aws = require('aws-sdk');

var s3 = new aws.S3();
aws.config.loadFromPath('./AwsConfig.json');
var BUCKET_NAME = 'test-dev-2';



finder.on('directory', function (dir, stat, stop) {
    var base = path.basename(dir);
    if (base === '.git' || base === 'node_modules' || base === 'bower_components') {
        stop();
    }
    else {
        console.log(dir + '/');
    }
});

finder.on('file', function (file, stat) {
    console.log(file);
    var params = {
        Bucket: BUCKET_NAME,
        Key: file,
        //Body:
    };
    //console.log(params.body);


    s3.putObject(params, function(err) {
        if(err) {
            console.log(err);
        }
        else {
            console.log("Success");
        }
    });
});

finder.on('error', function (err) {
    console.log(err);
});

finder.on('end', function () {
    console.log('Done!');
});

Answer 1

Based on the documentation, the Body parameter of s3.putObject can take a Buffer, Typed Array, Blob, String, or ReadableStream. The best one of those to use in most cases would be a ReadableString. You can create a ReadableString from any file using the createReadStream() function in the fs module.

So, that part your code would look something like:

finder.on('file', function (file, stat) {
    console.log(file);
    var params = {
        Bucket: BUCKET_NAME,
        Key: file,
        Body: fs.createReadStream(file) // NOTE: You might need to adjust "file" so that it's either an absolute path, or relative to your code's directory.
    };

    s3.putObject(params, function(err) {
        if(err) {
            console.log(err);
        }
        else {
            console.log("Success!");
        }
    });
});

I also want to point out that you might run in to a problem with this code if you pass it a directory with a lot of files. putObject is an asynchronous function, which means it'll be called and then the code will move on to something else while it's doing its thing (ok, that's a gross simplification, but you can think of it that way). What that means in terms of this code is that you'll essentially be uploading all the files it finds at the same time; that's not good.

What I'd suggest is to use something like the async module to queue your file uploads so that only a few of them happen at a time.

Essentially you'd move the code you have in your file event handler to the queue's worker method, like so:

var async = require('async');

var uploadQueue = async.queue(function(file, callback) {
    var params = {
        Bucket: BUCKET_NAME,
        Key: file,
        Body: fs.createReadStream(file) // NOTE: You might need to adjust "file" so that it's either an absolute path, or relative to your code's directory.
    };

    s3.putObject(params, function(err) {
        if(err) {
            console.log(err);
        }
        else {
            console.log("Success!");
        }

        callback(err); // <-- Don't forget the callback call here so that the queue knows this item is done
    });
}, 2); // <-- This "2" is the maximum number of files to upload at once

Note the 2 at the end there, that specifies your concurrency which, in this case, is how many files to upload at once.

Then, your file event handler simply becomes:

finder.on('file', function (file, stat) {
    uploadQueue.push(file);
});

That will queue up all the files it finds and upload them 2 at a time until it goes through all of them.