Different behavior for "aws s3 ls" and "aws s3api list-objects"

Question

I am seeing different behavior for aws s3 ls and aws s3api list-buckets

Here is the 1st one:

$ aws s3 ls s3://demo.for.customers

Bucket: demo.for.customers
Prefix: 

      LastWriteTime     Length Name
      -------------     ------ ----
                           PRE 5CE4D191-FD14-4C85-8146-9FB8C29B7A7B/
                           PRE FFBC4675-F864-40E9-8AB8-BDF7A0437010/

So, I am able to list the objects inside the bucket demo.for.customers

Now, When I run the same thing using s3api, I get access denied:

$ aws s3api list-objects --bucket demo.for.customers
A client error (AccessDenied) occurred: Access Denied

QUESTION: Why do I get access denied for listing objects via s3api.

The reason I am raising this issues is that I get the same problem if I use AWS S3 Ruby SDK.

However, things are fine when I use aws s3 ls.

So the AWS S3 Ruby SDK and the aws s3api are exhibiting the same behavior. So, I am pasting only the aws s3api CLI issue here.

BTW, here is the IAM policy that has been applied to the user who is running all above commands:

{
  "Statement": [
    {
      "Action": [
        "s3:ListAllMyBuckets",
        "s3:GetBucketLocation"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::"
      ]
    },
    {
      "Action": [
        "s3:ListBucket"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::demo.for.customers"
      ],
      "Condition": {
        "StringEquals": {
          "s3:prefix": [
            "",
            "FFBC4675-F864-40E9-8AB8-BDF7A0437010/"
          ],
          "s3:delimiter": [
            "/"
          ]
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::demo.for.customers/FFBC4675-F864-40E9-8AB8-BDF7A0437010/"
      ]
    }
  ]
}

And here is the ruby code which produces exact same error;

#!/usr/bin/ruby

require 'aws-sdk'
require 'awesome_print'
AWS.config( :access_key_id      => 'whatever', 
            :secret_access_key  => 'again whatever',
            :region             => 'us-west-2')

s3 = AWS.s3
buckets = s3.client.list_objects(:bucket_name => "demo.for.customers")
ap buckets

and the output is:

# ruby s3policies.rb 
/var/lib/gems/1.9.1/gems/aws-sdk-1.14.1/lib/aws/core/client.rb:366:in `return_or_raise': Access Denied (AWS::S3::Errors::AccessDenied)

Answer 1

Based upon the Role you have defined, a call to list-objects requires both a Prefix and a Delimiter.

The following command will work for you:

aws s3api list-objects --bucket demo.for.customers --prefix "" --delimiter "/"

If you remove the Delimiter condition within your ListBucket policy, then this will work for you:

aws s3api list-objects --bucket demo.for.customers --prefix ""

And if you also remove the Prefix condition, then this will work for you:

aws s3api list-objects --bucket demo.for.customers

A good way to test the above is to duplicate your Role Policy and progressively remove Conditions until it functions as expected.