Override S3 endpoint using Boto3 configuration file

Question

OVERVIEW:

I'm trying to override certain variables in boto3 using the configuration file (~/aws/confg). In my use case I want to use fakes3 service and send S3 requests to the localhost.

EXAMPLE:

In boto (not boto3), I can create a config in ~/.boto similar to this one:

[s3] host = localhost calling_format = boto.s3.connection.OrdinaryCallingFormat  [Boto] is_secure = False 

And the client can successfully pick up desired changes and instead of sending traffic to the real S3 service, it will send it to the localhost.

>>> import boto >>> boto.connect_s3() S3Connection:localhost >>>  

WHAT I TRIED:

I'm trying to achieve a similar result using boto3 library. By looking at the source code I found that I can use ~/aws/config location. I've also found an example config in unittests folder of botocore.

I tried to modify the config to achieve the desired behaviour. But unfortunately, it doesn't work.

Here is the config:

[default] aws_access_key_id = XXXXXXXXX aws_secret_access_key = YYYYYYYYYYYYYY region = us-east-1 is_secure = False s3 =     host = localhost 

QUESTION:

1. How to overwrite clients variables using config file?
2. Where can I find a complete list of allowed variables for the configuration?

Answer 1

You cannot set host in config file, however you can override it from your code with boto3.

import boto3  session = boto3.session.Session()  s3_client = session.client(     service_name='s3',     aws_access_key_id='aaa',     aws_secret_access_key='bbb',     endpoint_url='http://localhost', ) 

Then you can interact as usual.

print(s3_client.list_buckets()) 

Answer 2

boto3 only reads the signature version for s3 from that config file. You may want to open a feature request, but for now here is how you can address a custom endpoint:

import boto3 from botocore.utils import fix_s3_host resource = boto3.resource(service_name='s3', endpoint_url='http://localhost') resource.meta.client.meta.events.unregister('before-sign.s3', fix_s3_host) 

That bit about the meta is important because boto3 automatically changes the endpoint to your_bucket_name.s3.amazonaws.com when it sees fit 1. If you'll be working with both your own host and s3, you may wish to override the functionality rather than removing it altogether.