AWS HTTP API - same request but different response in Python Requests vs Dart HTTP

Question

I am trying to use AWS DynamoDB in a Flutter app, and given the lack of an official AWS SDK for Dart I am forced to use the low level HTTP REST API.

The method for signing an AWS HTTP request is quite tedious, but using an AWS supplied sample as a guide, I was able to convert the Python to Dart pretty much line-for-line relatively easily. The end result was both sets of code producing the same auth signatures.

My issue came when I actually went to sent the request. The Python works as expected but sending a POST with Dart's HTTP package gives the error

The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

I'll spare you the actual code for generating the auth signature, as the issue can be replicated simply by sending the same request hard-coded. See the Python and Dart code below.

Note: A valid response will return

Signature expired: 20190307T214900Z is now earlier than 20190307T215809Z (20190307T221309Z - 15 min.)

as the request signature uses current date and is only valid for 15 mins.

*****PYTHON CODE*****

import requests

headers = {'Content-Type':'application/json',
           'X-Amz-Date':'20190307T214900Z',
           'X-Amz-Target':'DynamoDB_20120810.GetItem',
           'Authorization':'AWS4-HMAC-SHA256 Credential=AKIAJFZWA7QQAQT474EQ/20190307/ap-southeast-2/dynamodb/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=297c5a03c59db6da45bfe2fda6017f89a0a1b2ab6da2bb6e0d838ca40be84320'}

endpoint = 'https://dynamodb.ap-southeast-2.amazonaws.com/'
request_parameters =  '{"TableName": "player-exports","Key": {"exportId": {"S": "HG1T"}}}'

r = requests.post(endpoint, data=request_parameters, headers=headers)

print('Response status: %d\n' % r.status_code)
print('Response body: %s\n' % r.text)

*****DART CODE*****

import 'package:http/http.dart' as http;

void main(List<String> arguments) async {

  var headers = {'Content-Type':'application/json',
           'X-Amz-Date':'20190307T214900Z',
           'X-Amz-Target':'DynamoDB_20120810.GetItem',
           'Authorization':'AWS4-HMAC-SHA256 Credential=AKIAJFZWA7QQAQT474EQ/20190307/ap-southeast-2/dynamodb/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=297c5a03c59db6da45bfe2fda6017f89a0a1b2ab6da2bb6e0d838ca40be84320'};

  var endpoint = 'https://dynamodb.ap-southeast-2.amazonaws.com/';
  var request_parameters =  '{"TableName": "player-exports","Key": {"exportId": {"S": "HG1T"}}}';

  http.post(endpoint, body: request_parameters, headers: headers).then((response) {
    print("Response status: ${response.statusCode}");
    print("Response body: ${response.body}");
  });
}

The endpoint, headers and body are literally copy and pasted between the two sets of code.

Is there some nuance to how Dart HTTP works that I am missing here? Is there some map/string/json conversion of the headers or request_paramaters happening?

One thing I did note is that in the AWS provided example it states

For DynamoDB, the request can include any headers, but MUST include "host", "x-amz-date", "x-amz-target", "content-type", and "Authorization". Except for the authorization header, the headers must be included in the canonical_headers and signed_headers values, as noted earlier. Order here is not significant. Python note: The 'host' header is added automatically by the Python 'requests' library.

But

a) When I add 'Host':'dynamodb.ap-southeast-2.amazonaws.com' to the headers in the Dart code I get the same result

and

b) If I look at r.request.headers after the Python requests returns, I can see that it has added a few new headers (Content-Length etc) automatically, but "Host" isn't one of them.

Any ideas why the seemingly same HTTP request works for Python Requests but not Dart HTTP?

Answer 1

Ok this is resolved now. My issue was in part a massive user-error. I was using a new IDE and when I generated the hardcoded example I provided I was actually still executing the previous file. Stupid, stupid, stupid.

But...

I was able to sort out the actual issue that caused me raise the question in the first place. I found that if you set the content type to "application/json" in the headers, the dart HTTP package automatically appends "; charset=utf-8". Because this value is part of the auth signature, when AWS encodes the values from the header to compare to the user-generated signature, they don't match.

The fix is simply to ensure that when you are setting the header content-type, make sure that you manually set it to "application/json; charset=utf-8" and not "application/json".

Found a bit more discussion about this "bug" after the fact here.