amazon s3 partitioning of files best practices

Question

Hopefully a simple question - apologies if it's already been answered but nothing came up in search.

On S3 is it better to organizes images into smaller subdirectories, or just keep them all in one directory? In a typical filesystem one would namespace the images in directories to improve performance. A flat structure with thousands of images in one directory doesn't normally perform well. Is this the case on Amazon S3?

I can put all user images into a users folder, all post images into a posts folder, etc. OR I can put user images into folders like users/{userId} to avoid having thousands of images in one users folder.

Answer 1

Update 2018-07

It is no longer required to account for performance when devising a partitioning scheme for your use case, see my InfoQ summary Amazon S3 Increases Request Rate Performance and Drops Randomized Prefix Requirement for details:

Amazon Web Services (AWS) recently announced significantly increased S3 request rate performance and the ability to parallelize requests to scale to the desired throughput. Notably this performance increase also "removes any previous guidance to randomize object prefixes" and enables the use of "logical or sequential naming patterns in S3 object naming without any performance implications".

Update 2013-09

The information in the referenced link, while still largely accurate, has been supplanted by a newer document, S3 Request Rate and Performance Considerations.

---

Initial answer

This is a problem with Amazon S3 as well, albeit only for significant storage requirements, see Amazon S3 Performance Tips & Tricks for a detailed answer including strategies for partitioning your object space.