Checking if a file is in a S3 bucket using the s3cmd

Question

I having a program that successfully uploads all of the files that I need. I have new files everyday that I need to upload. After I have uploaded the files I no longer need them and thus am not looking to sync them.

I am curious if there is a way to check if given a path and file name if that exists within S3 using the s3cmd.

Answer 1

You can use the ls command in s3cmd to know if a file is present or not in S3.

Bash code

path=$1
count=`s3cmd ls $path | wc -l`

if [[ $count -gt 0 ]]; then
        echo "exist"
else
        echo "do not exist"
fi

Usage: ./s3_exist.sh s3://foo/bar.txt

Edit:

As cocoatomo pointed out in comments, s3cmd ls $path lists all file that begins with $path. A safer approach would be to use s3cmd info $path and check the exit code.

New Bash code

path=$1
s3cmd info $path >/dev/null 2>&1

if [[ $? -eq 0 ]]; then
    echo "exist"
else
    echo "do not exist"
fi

Answer 2

Assuming that bar.txt and bar.txt.bak exist in a bucket s3://foo, "s3cmd ls s3://foo/bar.txt" shows a following output.

$ s3cmd ls s3://foo/bar.txt
2013-11-11 11:11    5   s3://foo/bar.txt
2013-11-11 11:11    5   s3://foo/bar.txt.bak

Since we should remove 2nd line from the command result, we use "awk" command to filter unnecessary lines.

$ filename=s3://foo/bar.txt
$ s3cmd ls ${filename} | awk "\$4 == \"${filename}\" { print \$4 }"
2013-11-11 11:11    5   s3://foo/bar.txt

Finally, we build up all commands.

filename=s3://foo/bar.txt
count=$(s3cmd ls ${filename} | awk "\$4 == \"${filename}\" { print \$4 }" | wc -l)

if [ $count -eq 0 ]; then
  echo "file does not exist"
else
  echo "file exists"
fi