Automatically detecting file changes and synchronizing via S3

Question

I have a local directory of media files on a Linux system, which I synchronise with an Amazon S3 account using an s3sync script. Currently, I'm manually running the s3sync script when I know the media files have been modified.

How would I automatically run the script when files are modified?

I was thinking of creating a cron job to run the script every few minutes, but that seems like an excessive amount of processing, because even if there are no changes, the script still has to scan the entire directory structure, which is quite large.

I also considered incron/inotify, which allows running commands when a specific file or directory changes, but these tools don't seem to automatically support monitoring changes to the entirety of a nested directory. Correct me if I'm wrong, but it seems that incron/inotify can only monitor files they've been explicitly told to monitor. e.g. If I wanted to monitor changes to all files at any level inside a directory, I'd have to write separate scripts to monitor directory and file additions/deletions to update the list of files and directories monitored by incron.

Are there more efficient solutions?

Answer 1

For this kind of tasks, I'm using fssm gem.

create file watcher.rb

require 'fssm'

FSSM.monitor('/dir_to_watch/', '**/*') do
  update {|base, relative| `your_script` }
  delete {|base, relative| `your_script` }
  create {|base, relative| `your_script` }
end

then:

ruby watcher.rb

Of course you can demonize it, if you want.

Answer 2

Here is a sample scenario you might use instead and utilize simple rsync script.

http://andrewwilkinson.wordpress.com/2011/01/14/rsync-backups-to-amazon-s3/

Basically means using fuse and s3fs ( http://code.google.com/p/s3fs/ ) to mount s3 share as a directory on your local filesystem and use rsync to sync the 2. Simple cron job would do the trick.

Answer 3

Now there is an efficient solution. This was just announced (long overdue):
http://aws.amazon.com/blogs/aws/s3-event-notification/

It is very simple to implement - time to throw out all the ugly cron jobs and list-loops.