Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Get notified when user uploads to an S3 bucket? [duplicate]

Possible Duplicate:
Notification of new S3 objects

We've got an app that stores user data on S3. The part of our app that handles the uploads is decoupled from the part that processes the data. In some cases, the user will be able to upload data directly to S3 without going through our app at all (this may happen if they have their own S3 account and supply us with credentials).

Is it possible to get notified whenever the contents of an S3 bucket change? It would be cool if somehow a message could get sent that says "this file was added/updated/deleted: foo".

Short of that, is there some timestamp somewhere I could poll that would tell the last time the bucket was updated?

If I can't do either of these things, then the only alternative is the crawl the entire bucket and look for changes. This will be slow and expensive.

like image 977
ccleve Avatar asked Dec 29 '11 23:12

ccleve


People also ask

How can you be notified when there's an object uploaded to your S3 bucket?

In the Buckets list, choose the name of the bucket that you want to enable events for. Choose Properties. Navigate to the Event Notifications section and choose Create event notification.

What happens if you upload the same file to S3?

It will overwrite the existing file. In case you want to have the previous file available, you need to enable versioning in the bucket.

Does S3 automatically replicate?

Amazon S3 CRR automatically replicates data between buckets across different AWS Regions. With CRR, you can set up replication at a bucket level, a shared prefix level, or an object level using S3 object tags.


1 Answers

Update 2014-11:

As Alan Illing points out in the comments, AWS now supports notifications from S3 to SNS, which can be forwarded automatically to SQS: http://aws.amazon.com/blogs/aws/s3-event-notification/

S3 can also send notifications to AWS Lambda to run your own code directly.

Original response that predicted S3->SNS notifications:

If Amazon supported this, they would use SNS to send out notifications that an object has been added to a bucket. However, at the moment, the only bucket event supported by S3 and SNS is to notify you when Amazon S3 detects that it has lost all replicas of a Reduced Redundancy Storage (RRS) object and can no longer service requests for that object.

Here's the documentation on the SNS events supported by S3:

http://docs.amazonwebservices.com/AmazonS3/latest/dev/NotificationHowTo.html

Based on the way that the documentation is written, it looks like Amazon has ideas for other notification events to add (like perhaps your idea for finding out when new keys have been added).

Given that it isn't supported directly by Amazon, the S3 client that uploads the object to S3 will need to trigger the notification, or you will need to do some sort of polling.

Custom event notification for uploads to S3 could be done using SNS if you like to get near-real-time updates for processing, or it can be done through SQS if you like to let the notifications pile up and process them out of a queue at your own pace.

If you are polling, you could reduce the number of keys you need to request by having the client upload with a prefix of, say, "unprocessed/..." followed by the unique key. Your polling software can then query just S3 keys starting with that prefix. When it is ready to process, it could change the key to "processing/..." and then later to "processed/..." or whatever. Objects in S3 are currently renamed by copy+delete operations performed by S3.

like image 82
Eric Hammond Avatar answered Oct 15 '22 05:10

Eric Hammond