S3 Connection timeout when using boto3

Question

I am using boto3 to operate with S3. If my application is unable to reach S3 due to a network issue, the connection will hang until eventually it times out. I would like to set a lower connection timeout. I came across this PR for botocore that allows setting a timeout:

$ sudo iptables -A OUTPUT -p tcp --dport 443 -j DROP  from botocore.client import Config import boto3  config = Config(connect_timeout=5, read_timeout=5)  s3 = boto3.client('s3', config=config)  s3.head_bucket(Bucket='my-s3-bucket')  

This throws a ConnectTimeout, but it still takes too long to error out:

ConnectTimeout: HTTPSConnectionPool(host='my-s3-bucket.s3.amazonaws.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<botocore.awsrequest.AWSHTTPSConnection object at 0x2ad5dd0>, 'Connection to my-s3-bucket.s3.amazonaws.com timed out. (connect timeout=5)')) 

Tweaking both the connect and read timeouts doesn't impact how quickly the connection responds.

Answer 1

You are probably getting bitten by boto3's default behaviour of retrying connections multiple times and exponentially backing off in between. I had good results with the following:

from botocore.client import Config import boto3  config = Config(connect_timeout=5, retries={'max_attempts': 0}) s3 = boto3.client('s3', config=config)