aws-sdk for Ruby access folder within bucket

Question

I have a bucket on the Amazon S3 with the folder in it. I'm trying to access it the following way via aws-sdk gem:

s3 = AWS::S3.new(
    :access_key_id => "KEY",
    :secret_access_key => "SECRET"
)

bucket = s3.buckets["my_bucket/my_folder"]
bucket.do_stuff....

I am getting the following error in return:

The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

Any ideas what I may be doing wrong?

Answer 1

probably the S3 bucket are trying to use is located outside the US-EAST (default location), so this should help you:

s3 = AWS::S3.new(
    :access_key_id => "KEY",
    :secret_access_key => "SECRET"
    :s3_endpoint => 's3-eu-west-1.amazonaws.com'
)

Choose your S3 enpdpoint from the following list:

US Standard *                           s3.amazonaws.com(default)
US West (Oregon) Region                 s3-us-west-2.amazonaws.com
US West (Northern California) Region    s3-us-west-1.amazonaws.com
EU (Ireland) Region                     s3-eu-west-1.amazonaws.com
Asia Pacific (Singapore) Region         s3-ap-southeast-1.amazonaws.com
Asia Pacific (Tokyo) Region             s3-ap-northeast-1.amazonaws.com
South America (Sao Paulo) Region        s3-sa-east-1.amazonaws.com

In terms of object access, the bucket name is my_bucket, but my_folder should be a part of object.

Answer 2

You need to configure your region specific endpoint for the bucket (where it was created). You can do this with:

AWS.config(:s3_endpoint => '...')
s3 = AWS::S3.new

or

s3 = AWS::S3.new(:s3_endpoint => '...')

You can avoid this in the future by using DNS comptible bucket names (also avoid dots in bucket names). If a bucket name is a valid subdomain, then you can address your bucket without configuring the region specific endpoint. Consider the following:

http:://bucket-name.s3.amazonaws.com/path/to/object.txt

Where the bucket is named "bucket-name" and the object key is "path/to/object.txt". This bucket could exist in any region, and yet you can access it using the "default" region. When the bucket name is not dns-compatible, then the url looks like:

http://s3.amazon.com/bucket/name/path/to/object.txt

In the example above, the bucket is "bucket/name", which is not dns compatible. It becomes part of the path, and now s3.amazon.com must be replaced with the region specific endpoint (if the bucket was not created in the classic region).

As someone else mentioned, paths should be part of the object key, not bucket name. This allows you to group objects by a common prefix. The '/' is used as a virtual folder (by convention only).

# print the key of every object with the given prefix
s3.buckets['bucket-name'].objects.with_prefix('path/to/').each do |object|
  puts object.key
end