Amazon S3: maximum PUT requests per second

Question

I'm putting a large number of small items to S3 using the REST API. The average payload is ~10 bytes.

The items are all going into one bucket, and have randomized names (i.e. there is no lexicographical order)

From EC2, I've managed a rate of 4-500 per second. I'm using a thread pool of 96 threads, with 64 TCP connections.

I occasionally get an HTTP 500, but have not yet received 503 - meant to indicate that the client slows the rate of requests.

Does anyone know what I can realistically attain? I know the pipe between EC2 and S3 can manage a throughput of 20 MB/s, so I'm hoping to do a bit better.

Answer 1

There should be no surprise that you are seeing poor performance using REST for transfer for such tiny payloads.

The way to do better is to restructure the nature of your protocol or storage so that the transaction overhead doesn't isn't the dominant factor.

Indeed, the size of the pipe is sort of immaterial to your question as you're filling it completely with HTTP overhead; for example if you could double the throughput of the connection, you'd be able to send twice as much useless overhead with effectively no change in usable data.