Uploading file to s3 via bash using AWS4 auth?

Question

I keep getting this error while trying to autoload a file to S3:

The request signature we calculated does not match the signature you provided. Check your key and signing method.

HMAC-SHA256s(){
 KEY="$1"
 DATA="$2"
 shift 2
 printf "$DATA" | openssl dgst -binary -sha256 -hmac "$KEY" | od -An -vtx1 | sed 's/[ \n]//g' | sed 'N;s/\n//'
}

HMAC-SHA256h(){
 KEY="$1"
 DATA="$2"
 shift 2
 printf "$DATA" | openssl dgst -binary -sha256 -mac HMAC -macopt "hexkey:$KEY" | od -An -vtx1 | sed 's/[ \n]//g' | sed 'N;s/\n//'
}


FILE_TO_UPLOAD=/var/www/cool/main.txt
BUCKET="temporaltestingstorage"
STARTS_WITH="Schiller/Zauberlehrling"

REQUEST_TIME=$(date +"%Y%m%dT%H%M%SZ")
REQUEST_REGION="eu-central-1"
REQUEST_SERVICE="s3"
REQUEST_DATE=$(printf "${REQUEST_TIME}" | cut -c 1-8)
AWS4SECRET="AWS4"$AWS_SECRET_KEY
ALGORITHM="AWS4-HMAC-SHA256"
EXPIRE="2015-01-01T00:00:00.000Z"
ACL="private"

POST_POLICY='{"expiration":"'$EXPIRE'","conditions": [{"bucket":"'$BUCKET'" },{"acl":"'$ACL'" },["starts-with", "$key", "'$STARTS_WITH'"],["eq", "$Content-Type", "application/octet-stream"],{"x-amz-credential":"'$AWS_ACCESS_KEY'/'$REQUEST_DATE'/'$REQUEST_REGION'/'$REQUEST_SERVICE'/aws4_request"},{"x-amz-algorithm":"'$ALGORITHM'"},{"x-amz-date":"'$REQUEST_TIME'"}]}'

UPLOAD_REQUEST=$(printf "$POST_POLICY" | openssl base64 )
UPLOAD_REQUEST=$(echo -en $UPLOAD_REQUEST |  sed "s/ //g")

SIGNATURE=$(HMAC-SHA256h $(HMAC-SHA256h $(HMAC-SHA256h $(HMAC-SHA256h $(HMAC-SHA256s $AWS4SECRET $REQUEST_DATE ) $REQUEST_REGION) $REQUEST_SERVICE) "aws4_request") $UPLOAD_REQUEST)

curl \
    --limit-rate 300k \
    --connect-timeout 120 \
  -F "key=$STARTS_WITH" \
  -F "acl=$ACL" \
  -F "Content-Type=application/octet-stream" \
  -F "x-amz-algorithm=$ALGORITHM" \
  -F "x-amz-credential=$AWS_ACCESS_KEY/$REQUEST_DATE/$REQUEST_REGION/$REQUEST_SERVICE/aws4_request" \
  -F "x-amz-date=$REQUEST_TIME" \
  -F "Policy=$UPLOAD_REQUEST" \
  -F "X-Amz-Signature=$SIGNATURE" \
  -F "file=@"$FILE_TO_UPLOAD http://$BUCKET.s3.amazonaws.com/

am I missing something?

Thanks

Answer 1

Several things I suggest your to change in your script before it can be debugged properly:

1. "Double quote" every literal that contains spaces/metacharacters and every expansion: "$var", "$(command "$var")", "${array[@]}", "a & b". See

   • Quotes
   • Arguments
   • Words

2. Regarding the variable POST_POLICY : Single quotes ( i.e. ') cause everything between them to be taken literally by bash. In your script, expressions like "$EXPIRE", "$BUCKET", "$ACL" don't expand in single quotes. If you want to embed a ' inside a '...', write it as the four characters, '\'' : printf '%s\n' 'It'\''s a blast!'

   Check if this problem exist in more places in your script, and modify accordingly.

3. By convention, environment variables (PATH, EDITOR, SHELL, ...) and internal shell variables (BASH_VERSION, RANDOM, ...) are fully capitalized. All other variable names should be lowercase. Since variable names are case-sensitive, this convention avoids accidentally overriding environmental and internal variables.

4. echo has many portability problems, and should never be used with option flags. Use printf instead: printf 'name: %s\n' "$name". See

   • http://wiki.bash-hackers.org/commands/builtin/echo
   • http://cfajohnson.com/shell/cus-faq.html#Q0b
   • http://www.in-ulm.de/~mascheck/various/echo+printf

5. Not so important, but worth mentioning: Don't use variables in printf's format string ( e.g. printf '%s' "$DATA" instead of printf "$DATA" ).

Answer 2

I would debug this by placing a set -x at the top of the file so that you can see all the command output.

I've suffered a lot lately from using quotation marks too much or too little in my bash scripts lately. I'm not an expert, but it seems to me you might be double wrapping some variables in quotations. That could create an incorrect digest.

Also, the way you're creating the digest seems a bit error prone since you're doing some modifications on the openssl output. There must be a way to avoid that our get openssl to create the digest as Amazon specifies. Going to peek at that.

Also, in the documentation they provide a way for you to test that you are creating the signature correctly. I would try out your signature logic on that and that would narrow it down to being a problem with the way you're doing the SHA256.

And here's a gist with some like minded individuals. Saw some blog posts that were outdated, and this gist seems to diverge from the documentation, but I appreciated their simple take on SHAing. Perhaps you should share your question there and ask them to update the gist.

I'm sorry this isn't more of an answer, but was too long for a comment. I hope it does get answered better! Always interested in learning more about openssl.