Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Check if folder exist in s3 bucket

How can I check if some folder exists in my s3 bucket using Ruby on Rails?

I’m using AWS:S3 official gem After initializing the global connection

AWS::S3::Base.establish_connection!(:access_key_id => 'my_key_id', :secret_access_key => ‘my_secret’) 

I have bucket named: myfirstbucket

With folder inside named: my_folder

With file inside my_folder named: my_pic.jpg

When I try to check if my_pic.jpg exist it work just fine

s3object.exists? “/my_folder/my_pic.jpg” , “myfirstbucket”
=>  True

How can I check only if the folder exists?

s3object = AWS::S3::S3Object
s3object.exists? “/my_folder/” , “myfirstbucket”
=>  False
like image 755
ofer goli Avatar asked Jul 09 '16 19:07

ofer goli


People also ask

Do folders exist in S3?

In Amazon S3, folders are used to group objects and organize files. Unlike a traditional file system, Amazon S3 doesn't use hierarchy to organize its objects and files. Amazon S3 console supports the folder concept only as a means of grouping (and displaying) objects.

How do you check if S3 bucket exists or not?

To check whether a bucket already exists before attempting to create one with the same name, call the doesBucketExist method. It will return true if the bucket exists, and false otherwise.

What is _$ folder in S3?

The "_$folder$" files are placeholders. Apache Hadoop creates these files when you use the -mkdir command to create a folder in an S3 bucket. Hadoop doesn't create the folder until you PUT the first object. If you delete the "_$folder$" files before you PUT at least one object, Hadoop can't create the folder.

Can I ping S3?

You cannot "ping a bucket" because bucket names are merely logical containers served by Amazon S3. There is no direct relationship between IP addresses and buckets. Resolving the DNS name of an S3 bucket would only return the address of an Amazon S3 server.


2 Answers

Use Bucket#objects:

bucket.objects({prefix: 'my/folder/'}).limit(1).any?

Returns a Collection of ObjectSummary resources. No API requests are made until you call an enumerable method on the collection. Client#list_objects will be called multiple times until every ObjectSummary has been yielded.

—http://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Bucket.html#objects-instance_method

like image 83
SoAwesomeMan Avatar answered Oct 19 '22 00:10

SoAwesomeMan


You have to check

  1. object key with trailing slash "/" is exists
  2. If not exists, Is there any item in the directory?

For example: You have file "my_file" under folder "my_folder". The AWS S3 record my_folder/my_file will exist offcourse, but my_folder/ may not exists.

So you have to check twice

if bucket.object("my_folder/").exists?
  true
else
  bucket.objects({prefix: "my_folder/"}).limit(1).any?
end

See the document: http://docs.aws.amazon.com/AmazonS3/latest/UG/FolderOperations.html

like image 20
Cateyes Lin Avatar answered Oct 19 '22 02:10

Cateyes Lin