Throttling S3 commands with aws cli

Question

I'm running a backup script using AWS CLI to perform an S3 sync command every night on my MediaTemple server. This has run without fail for months, but I updated my Plesk installation and now every night, when the backup script runs, MediaTemple disables my server due to excessive usage. The limits I seem to be crossing are as follows:

RESOURCE INFO:
Packets per second limit: 35000
Packets per second detected: 42229.11667000000306870788
Bytes per second limit: 50000000
Bytes per second detected: 61801446.10000000149011611938

They also include a networking snapshot at the time they take the server offline which includes many open connections to Amazon IP addresses (9 at time of the snapshot).

Is there anything I can do to throttle the connections to AWS? Preferably I'm looking for an option within the AWS API (though I haven't seen anything useful in the documentation), but barring that, is there something I can do on my end to manage the connections at the network level?

Answer 1

As well as changing the max default connections and chunk size already mentions you can also set the max_bandwidth. This is very effective when uploading large single files.

aws configure set default.s3.max_bandwidth 50MB/s

Answer 2

The AWS CLI S3 transfer commands (which includes sync) have the following relevant configuration options:

• max_concurrent_requests -
  • Default: 10
  • The maximum number of concurrent requests.
• multipart_threshold -
  • Default: 8MB
  • The size threshold the CLI uses for multipart transfers of individual files.
• multipart_chunksize -
  • Default: 8MB
  • When using multipart transfers, this is the chunk size that the CLI uses for multipart transfers of individual files.

This isn't so granular as throttling packets per second, but it seems like setting a lower concurrent request value and lowering both multipart threshold and chunksize will help. If the values you pasted are close to average, I would start with these values and tweak until you're reliably not exceeding the limits anymore:

$ aws configure set default.s3.max_concurrent_requests 8
$ aws configure set default.s3.multipart_threshold 6MB
$ aws configure set default.s3.multipart_chunksize 6MB