Amazon S3 file download through curl by using IAM user credentials

Question

I have created an IAM user with access to only one bucket. I have tested the credentials and permissions through web and python boto. Its working fine.

Now I have requirement to use these credentials and download the private file from that bucket through curl.

signature="$(echo -n "GET" | openssl sha1 -hmac "f/rHQ8yCvPthxxxxxxxXxxxx" -binary | base64)"
date="$(LC_ALL=C date -u +"%a, %d %b %Y %X %z")"

curl -H "Host: my-bucket.s3.amazonaws.com" -H "Date: $date" -H "Authorization: AWS 'XXXAJX2NY3QXXX35XXX':$signature" -H "Content-Type: 'text/plain'" https://my-bucket.s3.amazonaws.com/path/to_file.txt

but i am getting the following error:

InvalidAccessKeyIdThe AWS Access Key Id you provided does not exist in our records.

Please help, how do I download the file using curl ? Is there anything am I missing or its not possible through curl command?

Thanks!

Answer 1

Following is the example on how you can download with s3 curl script,

#!/bin/sh 
file=path/to/file 
bucket=your-bucket 
resource="/${bucket}/${file}" 
contentType="application/x-compressed-tar" 
dateValue="`date +'%a, %d %b %Y %H:%M:%S %z'`" 
stringToSign="GET 
${contentType} 
${dateValue} 
${resource}" 
s3Key=xxxxxxxxxxxxxxxxxxxx 
s3Secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 
signature=`/bin/echo -en "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -H "Host: ${bucket}.s3.amazonaws.com" \
-H "Date: ${dateValue}" \
-H "Content-Type: ${contentType}" \ 
-H "Authorization: AWS ${s3Key}:${signature}" \ 
https://${bucket}.s3.amazonaws.com/${file}

Hope it helps.