Whats the best way pull files to S3 using FPT/SCP?

Question

Whats the best, light-weight, way to FPT/SCP files to S3 every few min? So wget/curl could work. Also a custom solution could work, maybe Java with DB to maintain the state. Any ideas would be helpful.

Answer 1

Just use s3cmd or boto or the S3 command-line tools or... well... quite a few other options.

Basically you're just posting to the S3 API. You can do it in nearly any language using AWS's myriad SDKs. s3cmd, however, is a nice, lightweight little CLI tool that's probably more than you need.

Your comment suggests you're looking for something slightly different. Here are a few recommendations based on the source and destination of the files:

• If you're moving files from EC2 or your localhost to S3, use s3cmd. If you need to run it regularly, consider a cron job. The following will run every five minutes.

  */5 * * * * s3cmd sync --delete-removed my_local_directory/ s3://my-bucket/path/
  

• If you're moving files from S3 to EC2 or your localhost, use s3cmd but reverse the syntax:

  */5 * * * * s3cmd sync s3://my-bucket/path/ my_local_directory/
  

• If you're moving files from your localhost to EC2, consider scp or rsync.

Answer 2

You can mount S3 bucket as a local partition and work with S3 files as they were located on your server's filesystem. There are a number of good Open Source tools available.

But from my side I would recommend you to take a look at my project: RioFS, a userspace filesystem to mount Amazon S3 buckets. Project's goals and the main advantages comparing to other similar tools are: simplicity, the speed of operations and a bugs-free code.

Currently the project is in the “beta” state, but it's been running on several high-loaded fileservers for quite some time (RioFS provides there an access to S3 located files to ftp / sftp servers).

We are building a community around our project and are seeking for more people to join our project to discuss future plans and help with the testing. From our side we offer a quick bugs fix and will listen to your requests to add new features.

A quick how-to:

You could mount a bucket using the following command (assuming you have installed RioFS and have exported AWSACCESSKEYID and AWSSECRETACCESSKEY environment variables):

riofs http://s3.amazonaws.com your_bucket_name /path/to/localfolder

(please refer to project's description and run riofs --help to get help with command line arguments)

Please note that the project is still in the development, there could be still a number of bugs left. If you find that something doesn't work as expected: please fill an issue report on the project's GitHub page.

Hope it helps and we are looking forward to seeing you joined our community !