Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

how to check if a particular directory exists in S3 bucket using python and boto3

How to check if a particular file is present inside a particular directory in my S3? I use Boto3 and tried this code (which doesn't work):

import boto3

s3 = boto3.resource('s3')
bucket = s3.Bucket('my-bucket')
key = 'dootdoot.jpg'
objs = list(bucket.objects.filter(Prefix=key))
if len(objs) > 0 and objs[0].key == key:
    print("Exists!")
else:
    print("Doesn't exist")
like image 791
AshuGG Avatar asked Sep 16 '19 13:09

AshuGG


2 Answers

While checking for S3 folder, there are two scenarios:

Scenario 1

import boto3

def folder_exists_and_not_empty(bucket:str, path:str) -> bool:
    '''
    Folder should exists. 
    Folder should not be empty.
    '''
    s3 = boto3.client('s3')
    if not path.endswith('/'):
        path = path+'/' 
    resp = s3.list_objects(Bucket=bucket, Prefix=path, Delimiter='/',MaxKeys=1)
    return 'Contents' in resp
  • The above code uses MaxKeys=1. This it more efficient. Even if the folder contains lot of files, it quickly responds back with just one of the contents.
  • Observe it checks Contents in response

Scenario 2

import boto3

def folder_exists(bucket:str, path:str) -> bool:
    '''
    Folder should exists. 
    Folder could be empty.
    '''
    s3 = boto3.client('s3')
    path = path.rstrip('/') 
    resp = s3.list_objects(Bucket=bucket, Prefix=path, Delimiter='/',MaxKeys=1)
    return 'CommonPrefixes' in resp
  • Observe it strips off the last / from path. This prefix will check just that folder and doesn't check within that folder.
  • Observe it checks CommonPrefixes in response and not Contents
like image 175
Sairam Krish Avatar answered Oct 07 '22 11:10

Sairam Krish


import boto3
import botocore

client = boto3.client('s3')
def checkPath(file_path):
  result = client.list_objects(Bucket="Bucket", Prefix=file_path )
  exists=False
  if 'Contents' in result:
      exists=True
  return exists

if the provided file_path will exist then it will return True. example: 's3://bucket/dir1/dir2/dir3/file.txt' file_path: 'dir1/dir2' or 'dir1/' Note:- file path should start with the first directory just after the bucket name.

like image 31
Vikrant chaudhary Avatar answered Oct 07 '22 11:10

Vikrant chaudhary