Download private file from S3 using bash

Question

I am trying to get the following bash script to work (copied from http://curl.haxx.se/mail/archive-2014-10/0006.html#replies):

#!/bin/sh 
file=path/to/file 
bucket=your-bucket 
resource="/${bucket}/${file}" 
contentType="application/x-compressed-tar" 
dateValue="`date +'%a, %d %b %Y %H:%M:%S %z'`" 
stringToSign="GET 
${contentType} 
${dateValue} 
${resource}" 
s3Key=xxxxxxxxxxxxxxxxxxxx 
s3Secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 
signature=`/bin/echo -n "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary |      base64` 
curl -H "Host: ${bucket}.s3.amazonaws.com" \
-H "Date: ${dateValue}" \
-H "Content-Type: ${contentType}" \ 
-H "Authorization: AWS ${s3Key}:${signature}" \ 
https://${bucket}.s3.amazonaws.com/${file}

I am getting a SignatureDoesNotMatch error no matter what I do.

Any ideas on how to fix this will be greatly appreciated.

Answer 1

Using various answers in this thread, I converted it into a handy s3get bash function:

#!/bin/bash

#usage - s3get writes the specified object to stdout
#  s3get <bucket/key> [region]

#set these in your environment/profile (NOT HERE)
AWS_ACCESS_KEY="" 
AWS_SECRET_KEY=""

#example usage
s3get my-bucket/a/path/to/my/file > /tmp/file

function s3get {
    #helper functions
    function fail { echo "$1" > /dev/stderr; exit 1; }
    #dependency check
    if ! hash openssl 2>/dev/null; then fail "openssl not installed"; fi
    if ! hash curl 2>/dev/null; then fail "curl not installed"; fi
    #params
    path="${1}"
    bucket=$(cut -d '/' -f 1 <<< "$path")
    key=$(cut -d '/' -f 2- <<< "$path")
    region="${2:-us-west-1}"
    #load creds
    access="$AWS_ACCESS_KEY"
    secret="$AWS_SECRET_KEY"
    #validate
    if [[ "$bucket" = "" ]]; then fail "missing bucket (arg 1)"; fi;
    if [[ "$key" = ""    ]]; then fail "missing key (arg 1)"; fi;
    if [[ "$region" = "" ]]; then fail "missing region (arg 2)"; fi;
    if [[ "$access" = "" ]]; then fail "missing AWS_ACCESS_KEY (env var)"; fi;
    if [[ "$secret" = "" ]]; then fail "missing AWS_SECRET_KEY (env var)"; fi;
    #compute signature
    contentType="text/html; charset=UTF-8" 
    date="`date -u +'%a, %d %b %Y %H:%M:%S GMT'`"
    resource="/${bucket}/${key}"
    string="GET\n\n${contentType}\n\nx-amz-date:${date}\n${resource}"
    signature=`echo -en $string | openssl sha1 -hmac "${secret}" -binary | base64` 
    #get!
    curl -H "x-amz-date: ${date}" \
        -H "Content-Type: ${contentType}" \
        -H "Authorization: AWS ${access}:${signature}" \
        "https://s3-${region}.amazonaws.com${resource}"
}

Tested on OSX and Ubuntu. Saved in this Github gist.