S3 SlowDown: Please reduce your request rate exception

Question

I have a service that sends 10k PUT requests to S3 for every second. The S3 was able to handle those loads for several minutes but started to throw SlowDown exception after that. It slows down my service to an unacceptable rate.

I have read this and implemented the suggested best practice. Following is the format of the prefix: bucket-name/[First four of UUID]-[YYYYmmddhhiiss]/[random UUID]/[random UUID].json. The method didn't work though.

Any idea how to overcome this error? Thank you! P.S: I have requested the PUT limit increase to AWS Support center. They suggested the above steps which didn't work.

Answer 1

S3 is distributed, and you need to make sure that you don't create hotspots. You can avoid this by ensuring that your object keys are truly random.

So move the [randome UUID] to the first part of your object key. And if you're not generating truly random UUIDs, (i.e. it sounds like perhaps the first 4 characters are similar for each object) try reversing the UUID.

More tips can be found here