AWS CLI S3 CP performance is painfully slow

Question

I've got an issue whereby uploads to and downloads from AWS S3 via the aws cli are very slow. By very slow I mean it consistently takes around 2.3s for a 211k file which indicates an average download speed of less than 500Kb/s which is extremely slow for such a small file. My webapp is heavily reliant on internal APIs and I've narrowed down that the bulk of the API's round-trip performance is predominantly related to uploading and downloading files from S3.

Some details:

• Using the latest version of aws cli (aws-cli/1.14.44 Python/3.6.6, Linux/4.15.0-34-generic botocore/1.8.48) on an AWS hosted EC2 instance
• Instance is running the latest version of Ubuntu (18.04)
• Instance is in region ap-southeast-2a (Sydney)
• Instance is granted role based access to S3 via a least privilege policy (i.e. minimum rights to the buckets that it needs access to)
• Type is t2.micro which should have Internet Bandwidth of ~60Mb or so
• S3 buckets are in ap-southeast-2
• Same result with encrypted (default) and unencrypted files
• Same result with files regardless of whether they have a random collection of alpha numeric characters in the object name
• The issue persists consistently, even after multiple cp attempts and after a reboot the cp attempt consistently takes 2.3s
• This leads me to wonder whether S3 or the EC2 instance (which is using a standard Internet Gateway) is throttled back
• I've tested downloading the same file from the same instance to a webserver using wget and it takes 0.0008s (i.e. 8ms)

So to summarise:

• Downloading the file from S3 via the AWS CLI takes 2.3s (i.e. 2300ms)
• Downloading the same file from a webserver (> Internet > Cloudflare > AWS > LB > Apache) via wget takes 0.0008s (i.e. 8ms)

I need to improve AWS CLI S3 download performance because the API is going to be quite heavily used in the future.

Answer 1

While my download speeds weren't as slow as yours, I managed to max out my ISPs download bandwidth with aws s3 cp by adding the following configuration to my ~/.aws/config:

[profile default]
s3 =
  max_concurrent_requests = 200
  max_queue_size = 5000
  multipart_threshold = 4MB
  multipart_chunksize = 4MB

If you don't want to edit the config file, you can probably use CLI parameters instead. Have a look at the documentation: https://docs.aws.amazon.com/cli/latest/topic/s3-config.html

Answer 2

I found that if I try to download an object using aws s3 cp, the download would hang close to finishing when the object size is greater than 500MB.

However, using get-object directly causes no hang or slowdown whatsoever. Therefore instead of using

aws s3 cp s3://my-bucket/path/to/my/object .

getting the object with

aws s3api get-object --bucket my-bucket --key path/to/my/object out-file

I experience no slowdown.