Is it better to have multiple s3 buckets or one bucket with sub folders?

Question

Is it better to have multiple s3 buckets per category of uploads or one bucket with sub folders OR a linked s3 bucket? I know for sure there will be more user-images than there will be profille-pics and that there is a 5TB limit per bucket and 100 buckets per account. I'm doing this using aws boto library and https://github.com/amol-/depot

Which is the structure my folders in which of the following manner?

/app_bucket     /profile-pic-folder     /user-images-folder  OR  profile-pic-bucket user-images-bucket   OR   /app_bucket_1 /app_bucket_2 

The last one implies that its really a 10TB bucket where a new bucket is created when the files within bucket_1 exceeds 5TB. But all uploads will be read as if in one bucket. Or is there a better way of doing what I'm trying to do? Many thanks!

I'm not sure if this is correct... 100 buckets per account?

https://www.reddit.com/r/aws/comments/28vbjs/requesting_increase_in_number_of_s3_buckets/

Answer 1

Yes, there is actually a 100 bucket limit per account. I asked the reason for that to an architect in an AWS event. He said this is to avoid people hosting unlimited static websites on S3 as they think this may be abused. But you can apply for an increase.

By default, you can create up to 100 buckets in each of your AWS accounts. If you need additional buckets, you can increase your bucket limit by submitting a service limit increase.

Source: http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html

Also, please note that there are actually no folders in S3, just a flat file structure:

Amazon S3 has a flat structure with no hierarchy like you would see in a typical file system. However, for the sake of organizational simplicity, the Amazon S3 console supports the folder concept as a means of grouping objects. Amazon S3 does this by using key name prefixes for objects.

Source: http://docs.aws.amazon.com/AmazonS3/latest/UG/FolderOperations.html

Finally, the 5TB limit only applies to a single object. There is no limit on the number of objects or total size of the bucket.

Q: How much data can I store?

The total volume of data and number of objects you can store are unlimited.

Source: https://aws.amazon.com/s3/faqs/

Also the documentation states there is no performance difference between using a single bucket or multiple buckets so I guess both option 1 and 2 would be suitable for you.

Hope this helps.

Answer 2

Simpler Permission with Multiple Buckets

If the images are used in different use cases, using multiple buckets will simplify the permissions model, since you can give clients/users bucket level permissions instead of directory level permissions.

2-way doors and migrations

On a similar note, using 2 buckets is more flexible down the road.

1 to 2: If you switch from 1 bucket to 2, you now have to move all clients to the new set-up. You will need to update permissions for all clients, which can require IAM policy changes for both you and the client. Then you can move your clients over by releasing a new client library during the transition period.

2 to 1: If you switch from 2 buckets to 1 bucket, your clients will already have access to the 1 bucket. All you need to do is update the client library and move your clients onto it during the transition period.

*If you don't have a client library than code changes are required in both cases for the clients.