How can I organize a million+ files now that I'm moving to Amazon S3?

Question

Well I'm getting booted from my shared host and I'm switching over to a combination of a VPS from Linode and Amazon S3 to host a few million jpegs.

My big worry is keeping some kind of sanity with all these images. Is there any hope of that? My understanding is you're only allowed 100 "buckets" and "buckets" are the only type of structure within S3.

Is putting a few million files in a bucket something you'd advise against?

Answer 1

You may notice in Bucket Restrictions and Limitations, it is stated:

There is no limit to the number of objects that can be stored in a bucket

My experience is that a very large number of objects in a single bucket will not affect the performance of getting a single object by its key (that is, get appears to be of constant complexity).

Having a very large number of object also does not affect the speed of listing a given number of objects:

List performance is not substantially affected by the total number of keys in your bucket

However, I must warn you, that most S3 management tools I've used (like S3Fox) will choke and die a horrible slow death when attempting to access a bucket with a very large number of objects. One tool that seems to cope well with very large numbers of objects is S3 Browser (they have a free version and a Pro version, I am not affiliated with them in any way).

Using "folders" or prefixes, does not change any of these points (get and listing a given number of objects are still constant, most tools still fall over themselves and hang).