Python boto, list contents of specific dir in bucket

Question

I have S3 access only to a specific directory in an S3 bucket.

For example, with the s3cmd command if I try to list the whole bucket:

    $ s3cmd ls s3://bucket-name 

I get an error: Access to bucket 'my-bucket-url' was denied

But if I try access a specific directory in the bucket, I can see the contents:

    $ s3cmd ls s3://bucket-name/dir-in-bucket 

Now I want to connect to the S3 bucket with python boto. Similary with:

    bucket = conn.get_bucket('bucket-name') 

I get an error: boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden

But if I try:

    bucket = conn.get_bucket('bucket-name/dir-in-bucket') 

The script stalls for about 10 seconds, and prints out an error afterwards. Bellow is the full trace. Any idea how to proceed with this?

Note question is about the boto version 2 module, not boto3.

Traceback (most recent call last):   File "test_s3.py", line 7, in <module>     bucket = conn.get_bucket('bucket-name/dir-name')   File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 471, in get_bucket     return self.head_bucket(bucket_name, headers=headers)   File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 490, in head_bucket     response = self.make_request('HEAD', bucket_name, headers=headers)   File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 633, in make_request     retry_handler=retry_handler   File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1046, in make_request     retry_handler=retry_handler)   File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 922, in _mexe     request.body, request.headers)   File "/usr/lib/python2.7/httplib.py", line 958, in request     self._send_request(method, url, body, headers)   File "/usr/lib/python2.7/httplib.py", line 992, in _send_request     self.endheaders(body)   File "/usr/lib/python2.7/httplib.py", line 954, in endheaders     self._send_output(message_body)   File "/usr/lib/python2.7/httplib.py", line 814, in _send_output     self.send(msg)   File "/usr/lib/python2.7/httplib.py", line 776, in send     self.connect()   File "/usr/lib/python2.7/httplib.py", line 1157, in connect     self.timeout, self.source_address)   File "/usr/lib/python2.7/socket.py", line 553, in create_connection     for res in getaddrinfo(host, port, 0, SOCK_STREAM): socket.gaierror: [Errno -2] Name or service not known 

Answer 1

For boto3

import boto3  s3 = boto3.resource('s3') my_bucket = s3.Bucket('my_bucket_name')  for object_summary in my_bucket.objects.filter(Prefix="dir_name/"):     print(object_summary.key) 

Answer 2

By default, when you do a get_bucket call in boto it tries to validate that you actually have access to that bucket by performing a HEAD request on the bucket URL. In this case, you don't want boto to do that since you don't have access to the bucket itself. So, do this:

bucket = conn.get_bucket('my-bucket-url', validate=False) 

and then you should be able to do something like this to list objects:

for key in bucket.list(prefix='dir-in-bucket'):      <do something> 

If you still get a 403 Errror, try adding a slash at the end of the prefix.

for key in bucket.list(prefix='dir-in-bucket/'):      <do something> 

Note: this answer was written about the boto version 2 module, which is obsolete by now. At the moment (2020), boto3 is the standard module for working with AWS. See this question for more info: What is the difference between the AWS boto and boto3