AWS BOTO3 S3 python - An error occurred (404) when calling the HeadObject operation: Not Found

Question

I am trying download a directory inside s3 bucket. I am trying to use transfer to download a directory from S3 bucket but I am getting an error as "An error occurred (404) when calling the HeadObject operation: Not Found". Please help.

S3 structure:
  **Bucket
     Folder1
        File1**

Note: Trying to download Folder1

transfer.download_file(self.bucket_name, self.dir_name, self.file_dir + self.dir_name)

Answer 1

I had the same issue recently. You are probably misspelling the path and folder name. In my case, for example, I was messing up with the '/'.

To fix it, make sure the variables you are using as arguments for the function contains the correct names of the directories, folders and files as it is in S3. Also, make sure you put the '/' in the correct places in the correct variables. For instance, in my case I found that:

• bucket name: bucket_name (with no '/' in the end, and no 's3://')
• directory name: folder1/folder2/file_name (with no '/' in the beginning)

I hope it helps you and others to get around this error easily.

Answer 2

Yet another possibly is that you entered an incorrect endpoint_url parameter when creating your S3 resource.

For future users, create your resource like this:

s3 = boto3.resource(
  's3',
  region_name=[your region, e.g. eu-central-1],
  aws_access_key_id=[your access key],
  aws_secret_access_key=[your secret key]
)

In the above, it is possible to pass an endpoint_url, as I erroneously did (I later found out that I had accidentally passed the endpoint URL to a different AWS service).

If you are using AWS CLI in order to authenticate, you can omit the region_name, aws_access_key, and aws_secret_access_key parameters, like so:

s3 = boto3.resource('s3')