cURL and s3 GET

Question

I'm trying to figure out what I need to provide in order to do a GET operation on an s3 object that has been encrypted using server-side encryption via a kms key. When trying to do a cURL on my test doc I get the following error:

Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4.

UPDATE: Adding the results from curl

$ curl -v https://s3-us-west-2.amazonaws.com/rkbtest/check.png
*   Trying 54.231.185.12...
* Connected to s3-us-west-2.amazonaws.com (54.231.185.12) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.s3-us-west-2.amazonaws.com
* Server certificate: DigiCert Baltimore CA-2 G2
* Server certificate: Baltimore CyberTrust Root
> GET /rkbtest/check.png HTTP/1.1
> Host: s3-us-west-2.amazonaws.com
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< x-amz-request-id: 2DECE9C69BDB8F0F
< x-amz-id-2: bs8xGSbAHksE2mSb/+r4AG3B9RlRTODasFyr5S3jMU2sNA7eJTEQr0dJTro5P2QKLRuMQtGw6tk=
< x-amz-region: us-west-2
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Wed, 21 Sep 2016 15:26:13 GMT
< Connection: close
< Server: AmazonS3
<
<?xml version="1.0" encoding="UTF-8"?>
* Closing connection 0
<Error><Code>InvalidArgument</Code><Message>Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4.</Message><ArgumentName>Authorization</ArgumentName><ArgumentValue>null</ArgumentValue><RequestId>2DECE9C69BDB8F0F</RequestId><HostId>bs8xGSbAHksE2mSb/+r4AG3B9RlRTODasFyr5S3jMU2sNA7eJTEQr0dJTro5P2QKLRuMQtGw6tk=</HostId></Error>

Answer 1

One really effetive way is to use that script. Just need export keys before use (or copy it's values to the .sh file).

export AWS_ACCESS_KEY_ID=AKxxx
export AWS_SECRET_ACCESS_KEY=zzzz

To download just call

./s3download.sh get s3://mybucket/myfile.txt myfile.txt

That's it, all you need to pass s3 bucket along with file name

Script needed

Create a s3download.sh file, and chmod +x s3download.sh, to use on command above.

#!/bin/bash
set -eu
s3simple() {
  local command="$1"
  local url="$2"
  local file="${3:--}"

  # todo: nice error message if unsupported command?

  if [ "${url:0:5}" != "s3://" ]; then
    echo "Need an s3 url"
    return 1
  fi
  local path="${url:4}"

  if [ -z "${AWS_ACCESS_KEY_ID-}"  ]; then
    echo "Need AWS_ACCESS_KEY_ID to be set"
    return 1
  fi

  if [ -z "${AWS_SECRET_ACCESS_KEY-}" ]; then
    echo "Need AWS_SECRET_ACCESS_KEY to be set"
    return 1
  fi

  local method md5 args
  case "$command" in
  get)
    method="GET"
    md5=""
    args="-o $file"
    ;;
  put)
    method="PUT"
    if [ ! -f "$file" ]; then
      echo "file not found"
      exit 1
    fi
    md5="$(openssl md5 -binary $file | openssl base64)"
    args="-T $file -H Content-MD5:$md5"
    ;;
  *)
    echo "Unsupported command"
    return 1
  esac

  local date="$(date -u '+%a, %e %b %Y %H:%M:%S +0000')"
  local string_to_sign
  printf -v string_to_sign "%s\n%s\n\n%s\n%s" "$method" "$md5" "$date" "$path"
  local signature=$(echo -n "$string_to_sign" | openssl sha1 -binary -hmac "${AWS_SECRET_ACCESS_KEY}" | openssl base64)
  local authorization="AWS ${AWS_ACCESS_KEY_ID}:${signature}"

  curl $args -s -f -H Date:"${date}" -H Authorization:"${authorization}" https://s3.amazonaws.com"${path}"
}

s3simple "$@"

You can find more information about the s3simple script here.

Answer 2

To download the file with curl, you would need to define the following authentication header:

Authorization: AWS AWSAccessKeyId:Signature

The Amazon S3 REST API uses the standard HTTP Authorization header to pass authentication information.

Developers are issued an AWS access key ID and AWS secret access key when they register. For request authentication, the AWSAccessKeyId element identifies the access key ID that was used to compute the signature and, indirectly, the developer making the request.

The Signature element is the RFC 2104 HMAC-SHA1 of selected elements from the request, and so the Signature part of the Authorization header will vary from request to request.

Example GET request:

GET /photos/puppy.jpg HTTP/1.1
Host: johnsmith.s3.amazonaws.com
Date: Tue, 27 Mar 2007 19:36:42 +0000

Authorization: AWS AKIAIOSFODNN7EXAMPLE:
bWq2s1WEIj+Ydj0vQ697zp+IXMU=

Example PUT request:

PUT /photos/puppy.jpg HTTP/1.1
Content-Type: image/jpeg
Content-Length: 94328
Host: johnsmith.s3.amazonaws.com
Date: Tue, 27 Mar 2007 21:15:45 +0000

Authorization: AWS AKIAIOSFODNN7EXAMPLE:
MyyxeRY7whkBe+bq8fHCL/2kKUg=

Source: Signing and Authenticating REST Requests.

---

Alternatively you should be using aws command, e.g.

aws s3 cp s3://rkbtest/check.png ./

Before that, you need to configure your AWS Signature Version.

Signature Version 4, a protocol for authenticating inbound API requests to AWS services, in all AWS regions.

For example:

aws configure set default.s3.signature_version s3v4

or for the specific profile:

aws configure set profile.<profilename>.s3.signature_version s3v4

^{Source: aws/aws-cli/issues/1006 at GitHub.}

---