If I stream a file to s3, will the event trigger once the file is complete?

Question

As title says, if I attached an event to an S3 bucket for put events and I stream a file to that bucket, will the event trigger once the upload starts? That way a receiver can start downloading stream for that file.

Or will the event be triggered after the file has been completed uploading?

Answer 1

There are two problems with what you are contemplating:

• The event does not fire until the upload is finished.

• Writing objects into S3 is always an atomic operation. The write either completes successfully, or it doesn't happen at all... and until it completes successfully, the object does not actually exist in the bucket.

If you are writing a new object into a bucket, authorized requests for the object will return a 404 error at least until the upload completes successfully.

If you are overwriting an existing object, authorized requests for the object will always return the old copy of the object, unchanged and undamaged, at least until the overwrite completes successfully.

Note the use of "at least until," above.

In all regions, except US-Standard (us-east-1) uploads of new objects are generally available immediately after the upload. In US-Standard, there may sometimes be a brief delay. Formerly, the us-east-1 region of S3 (Northern Virginia, which was previously designated the "US Standard" region) did not offer immediate (read-after-write) consistency for new objects, but now it does.

However, there is a catch: the object must not have been requested before it was uploaded. If it is, the consistency model breaks.¹

Amazon S3 provides read-after-write consistency for PUTS of new objects in your S3 bucket in all regions with one caveat. The caveat is that if you make a HEAD or GET request to the key name (to find if the object exists) before creating the object, Amazon S3 provides eventual consistency for read-after-write.

http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#Regions

In all regions, overwrites of existing objects may also encounter a brief delay (and delete objects may remain accessible for a brief time after deletion).

This potential delay is because of S3's eventual consistency model on some operations, as described above. S3 does not guarantee all operations will be immediately visible, though for practical purposes, they almost always are. What S3 does guarantee is that if your upload completes successfully, with a success response from S3, then your object is committed to the S3 backing store.

The above applies to PUT uploads, PUT/Copy, and multipart.

For these reasons, S3 cannot stream your file out to a consumer while the upload has not yet completed.

To do that requires a different solution (though S3 could of course be used as a permanent repository after the streaming is done).

---

¹the consistency model breaks. Almost certainly, this indicates that a request arriving at S3 consults a replica of the bucket index, and if the index has no knowledge of whether an object exists, it consults a more authoritative version of the index. If it still finds nothing, it locally "remembers" that the object doesn't exist, because that upstream lookup is a comparatively expensive operation -- thus it won't look upstream again on subsequent requests -- but once the creation of the new object propagates into the local index, the object would be available. The same theory explains the eventual consistency of overwrites and deletes.