How to find a file in Amazon S3 bucket without knowing the containing folder

Question

My Amazon S3 bucket has a folder structure that looks like the below.

• bucket-name\00001\file1.txt
• bucket-name\00001\file2.jpg
• bucket-name\00002\file3.doc
• bucket-name\00001\file4.ppt

If I only know file name file3.doc and bucket name bucket-name how can i search for file3.doc in bucket-name. If I knew, it is in folder 00002, I could simply go to the folder and start typing the file name but I have no way to know in which folder the file I am searching for is under.

Answer 1

You can easily do this with the AWS CLI.

aws s3 ls s3://BUCKET-NAME/ --recursive | grep FILE-NAME.TXT

Answer 2

Using only the AWS CLI, you can run a list-objects against the bucket with the --query parameter. This will not be a fast operation, as it runs locally after fetching the file list, rather than inside s3's api.

$ aws s3api list-objects --bucket bucket-name --query "Contents[?contains(Key, 'file3')]"

[
    {
        "LastModified": "2017-05-31T20:36:28.000Z",
        "ETag": "\"b861daa5cc3775f38519f5de6566cbe7\"",
        "StorageClass": "STANDARD",
        "Key": "00002/file3.doc",
        "Owner": {
            "DisplayName": "owner",
            "ID": "123"
        },
        "Size": 27032
    }
]

The benefit of using --query over just piping to grep is that you'll get the full response including all available metadata usually included in list-objects, without having to monkey around with before and after arguments for the grep.

See this post on Finding Files in S3 for further information including a similar example which shows the benefit of having metadata, when files of the same name end up in different directories.

Answer 3

You'll probably need to use a command line tool like s3cmd if you don't know where it is at all:

s3cmd --recursive ls s3://mybucket | grep "file3"

but some limited search is possible:

https://stackoverflow.com/a/21836343/562557