Can't upload a file to S3 with pre-signed URL no matter what I do. AWS command line works. CURL and anything else = 403

Question

My AWS credentials in ~/.aws/credentials are correct and working. Proof?

$ aws s3api put-object --bucket <my bucket name> --key videos/uploads/yoda.jpeg --body /Users/r<my_name>/Desktop/Archive/yoda.jpeg

getting back:

{
    "ETag": "\"66bee0b7caf3d127900e0a70f2da4b5f\""
}

The upload worked from command line. And I can see my file when I see my S3 bucket in AWS's management console.

NOW- I delete the successfully uploaded file from S3 and I'm trying to upload it again, this time via a presigned URL

$ aws s3 presign s3://<my-bucket>/videos/uploads/yoda.jpeg

for which I get:

https://<my-bucket>.s3.amazonaws.com/videos/uploads/yoda.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Credential=<MY-AWS-KEY-ID>%2F20210207%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Date=20210207T222859Z&X-Amz-Signature=3a3624b9e264c119ebdf93c989efb73337f7ab8793e89554c7b000e1fc93c85c

From this moment on, any PUT attempt with CURL, POSTMAN or any other tool, with this URL fails to upload the file always ends up with 403 (yes, it's not expiring, it fails immediately) and The request signature we calculated does not match the signature you provided is the excuse provided by AWS.

The S3 bucket has a policy allowing the user whose credentials are in /.aws/credentials to Put* on that very bucket.

What is going on? Why doesn't pre-signed URL work?

CURL ATTEMPT

$ curl --location --request PUT 'https://<my-bucket-name>.s3.amazonaws.com/videos/uploads/yoda.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=3600&X-Amz-Credential=<MY-AWS-KEY-ID>%2F20210207%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-SignedHeaders=host&X-Amz-Date=20210207T224403Z&X-Amz-Signature=8a8625591e6c4e0871f97bf5e15c2f93b3e373cfc1c2daddb2cf34edb10a5670%0A' \
--header 'Content-Type: image/jpeg' \
--data-binary '@/Users/<MY-NAME>/Desktop/Archive/yoda.jpeg'

to which I get:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
    <Code>SignatureDoesNotMatch</Code>
    <Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
    <AWSAccessKeyId><---MY--ACCESS--KEY--ID--->/AWSAccessKeyId>
    <StringToSign>AWS4-HMAC-SHA256
20210207T224403Z
20210207/us-east-2/s3/aws4_request
da93cc1a0ec196fe0726ec6d5cace8c1b2b4865b20663bf0240454e276dbef6f</StringToSign>
    <SignatureProvided>8a8625591e6c4e0871f97bf5e15c2f93b3e373cfc1c2daddb2cf34edb10a5670
</SignatureProvided>
    <StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 31 30 32 30 37 54 32 32 34 34 30 33 5a 0a 32 30 32 31 30 32 30 37 2f 75 73 2d 65 61 73 74 2d 32 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 64 61 39 33 63 63 31 61 30 65 63 31 39 36 66 65 30 37 32 36 65 63 36 64 35 63 61 63 65 38 63 31 62 32 62 34 38 36 35 62 32 30 36 36 33 62 66 30 32 34 30 34 35 34 65 32 37 36 64 62 65 66 36 66</StringToSignBytes>
    <CanonicalRequest>PUT
/videos/uploads/yoda.jpeg
X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Credential=<---MY--ACCESS--KEY--ID--->%2F20210207%2Fus-east-2%2Fs3%2Faws4_request&amp;X-Amz-Date=20210207T224403Z&amp;X-Amz-Expires=3600&amp;X-Amz-SignedHeaders=host
host:<my-bucket-name>.s3.amazonaws.com

host
UNSIGNED-PAYLOAD</CanonicalRequest>
    <CanonicalRequestBytes>50 55 54 0a 2f 76 69 64 65 6f 73 2f 75 70 6c 6f 61 64 73 2f 79 6f 64 61 2e 6a 70 65 67 0a 58 2d 41 6d 7a 2d 41 6c 67 6f 72 69 74 68 6d 3d 41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 26 58 2d 41 6d 7a 2d 43 72 65 64 65 6e 74 69 61 6c 3d 41 4b 49 41 51 33 44 34 36 52 4e 50 48 51 4e 4b 47 42 46 4b 25 32 46 32 30 32 31 30 32 30 37 25 32 46 75 73 2d 65 61 73 74 2d 32 25 32 46 73 33 25 32 46 61 77 73 34 5f 72 65 71 75 65 73 74 26 58 2d 41 6d 7a 2d 44 61 74 65 3d 32 30 32 31 30 32 30 37 54 32 32 34 34 30 33 5a 26 58 2d 41 6d 7a 2d 45 78 70 69 72 65 73 3d 33 36 30 30 26 58 2d 41 6d 7a 2d 53 69 67 6e 65 64 48 65 61 64 65 72 73 3d 68 6f 73 74 0a 68 6f 73 74 3a 6c 73 74 76 32 2d 70 75 62 6c 69 63 2e 73 33 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 0a 68 6f 73 74 0a 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 44</CanonicalRequestBytes>
    <RequestId>CBJT0Y4SX9A7RB26</RequestId>
    <HostId>h+5b/u8cdi34yuSDBX0Z/mZGQMtRZIMS4rvIwiKzOZSOZhRoQfak8cOdVBq2BgtU1qbqlHrO2TY=</HostId>
</Error>

TRYING TO GENERATE THE PRESIGN URL FROM PYTHON. STILL DOES NOT WORK. THE URL IS FAULTY- AWS REJECTS WITH THE SAME 403

 def get_upload_pre_signed_url(bucket_name, object_name, expiration=3600):
        s3_client = boto3.client('s3')
        try:
            response = s3_client.generate_presigned_url('put_object',
                                                        Params={'Bucket': bucket_name,
                                                                'Key': object_name},
                                                        ExpiresIn=expiration)
        except ClientError as e:
            return None
    
            # The response contains the presigned URL
        return response

URL generated from this:

https://<my-bucket>.s3.amazonaws.com//videos/uploads/yoda.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<my-AWS-KEY-ID>%2F20210207%2Fus-east-2%2Fs3%2Faws4_request&X-Amz-Date=20210207T231306Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=968a3e2cab9b7e907df69e24aae24d79ea40f52a52d407591d7cbd69c86fe67b

Curling it gets the same 403. Doesn't change.

Answer 1

The aws s3 presign command creates URLs that can be used for downloading files. It does not create URLs that can be used for uploading. To quote the docs-

Generate a pre-signed URL for an Amazon S3 object. This allows anyone who receives the pre-signed URL to retrieve the S3 object with an HTTP GET request. All presigned URL’s now use sigv4 so the region needs to be configured explicitly.

To create upload URLs you need to jump out of the command line and into your language of choice and use the full AWS SDK.

Python example:

s3.generate_presigned_post(
        Bucket=BUCKET_NAME,
        Key=FILE_KEY,
        ExpiresIn=(5*60)
    )

Note that it uses the generate_presigned_post function to do this.