Using boto for AWS S3 Buckets for Signature V4

Question

I have a problem with using Python-Boto SDK for S3 Buckets for region Frankfurt. According to Amazon link this region will only support V4. This document explains how to add V4 support for Boto SDK. I have added a new section:

if not boto.config.get('s3', 'use-sigv4'):
    boto.config.add_section('s3')
    boto.config.set('s3', 'use-sigv4', 'True')

and then I have created new connection and got all buckets:

connection = S3Connection(accesskey, secretkey, host=S3Connection.DefaultHost)
buckets = connection.get_all_buckets()

it works fine, but then I tried to get all keys for my bucket:

for bucket in buckets:
    bucket.get_all_keys()

and I got the following:

S3ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-central-1'</Message><Region>eu-central-1</Region>

Why did it occur? After that I connected to the region and got all needed data:

region_con = boto.s3.connect_to_region('eu-central-1', aws_access_key_id=accesskey, aws_secret_access_key=secretkey)
bucket = region_con.get_bucket(bucket.name)
bucket.get_all_keys()

How can I fix it properly?

Answer 1

hsrv's answer above works for boto 2. For boto3, the following is broadly equivalent:

s3 = boto3.client('s3', region_name='eu-central-1')

Alternatively, you can set the region field in your .aws/config:

[default]
output = json
region = eu-central-1

This sets the default region; you can still pick a specific region in Python as above.

The significance of the region varies from service to service (for example, assuming you're not sat in a VPC, you can access an S3 bucket from anywhere). In this case, however, the important thing is that newer regions (such as Frankfurt) only support the newer authentication scheme (AWS4-HMAC-SHA256). Boto runs into problems if you try to connect to anything in such a region from a region that still uses the old scheme (such as Dublin).