AWS CLI S3 rm command does not produce error if file does not exist

Question

I have a bash script which iterates over an array of file names and removes them from S3.

The following command:

aws s3 rm "s3://myBucket/myFolder/myFile.txt"

will produce this output.

delete: s3://myBucket/myFolder/myFile.txt

I can see that the delete was successful by verifying it has been removed in the AWS console. However if I iterate over the same list again, I get the same output even though the file is gone.

Is there any way -- using just the rm command -- of indicating that AWS CLI tried to delete the file but could not find it?

Answer 1

The s3 cli rm command uses the s3 API Delete Object operation

As you can see in the documentation, this adds a "delete marker" to the object. In a sense, it is "labelled" as deleted. There doesn't seem to be any check before these markers are made that the underlying object actually exists

As the S3 storage is distributed, consistency isn't guaranteed under all circumstances.

What this means is that if you were to carry out some operations on a file and then check it, the answer wouldn't be certain

In the case of S3 the AWS docs say

Amazon S3 offers eventual consistency for overwrite PUTS and DELETES in all regions.

"eventual consistency" means that at some undefined point in the future all the distributed nodes will catch up with the changes and the results returned from a query will be as expected, given the changes you have done

So basically, this is a long-winded way of saying: No, you can't get a confirmation that the file is deleted. Checking to see if it exists afterwards will not work reliably