Boto3 to download all files from a S3 Bucket

Question

I'm using boto3 to get files from s3 bucket. I need a similar functionality like aws s3 sync

My current code is

#!/usr/bin/python import boto3 s3=boto3.client('s3') list=s3.list_objects(Bucket='my_bucket_name')['Contents'] for key in list:     s3.download_file('my_bucket_name', key['Key'], key['Key']) 

This is working fine, as long as the bucket has only files. If a folder is present inside the bucket, its throwing an error

Traceback (most recent call last):   File "./test", line 6, in <module>     s3.download_file('my_bucket_name', key['Key'], key['Key'])   File "/usr/local/lib/python2.7/dist-packages/boto3/s3/inject.py", line 58, in download_file     extra_args=ExtraArgs, callback=Callback)   File "/usr/local/lib/python2.7/dist-packages/boto3/s3/transfer.py", line 651, in download_file     extra_args, callback)   File "/usr/local/lib/python2.7/dist-packages/boto3/s3/transfer.py", line 666, in _download_file     self._get_object(bucket, key, filename, extra_args, callback)   File "/usr/local/lib/python2.7/dist-packages/boto3/s3/transfer.py", line 690, in _get_object     extra_args, callback)   File "/usr/local/lib/python2.7/dist-packages/boto3/s3/transfer.py", line 707, in _do_get_object     with self._osutil.open(filename, 'wb') as f:   File "/usr/local/lib/python2.7/dist-packages/boto3/s3/transfer.py", line 323, in open     return open(filename, mode) IOError: [Errno 2] No such file or directory: 'my_folder/.8Df54234' 

Is this a proper way to download a complete s3 bucket using boto3. How to download folders.

Answer 1

I have the same needs and created the following function that download recursively the files.

The directories are created locally only if they contain files.

import boto3 import os  def download_dir(client, resource, dist, local='/tmp', bucket='your_bucket'):     paginator = client.get_paginator('list_objects')     for result in paginator.paginate(Bucket=bucket, Delimiter='/', Prefix=dist):         if result.get('CommonPrefixes') is not None:             for subdir in result.get('CommonPrefixes'):                 download_dir(client, resource, subdir.get('Prefix'), local, bucket)         for file in result.get('Contents', []):             dest_pathname = os.path.join(local, file.get('Key'))             if not os.path.exists(os.path.dirname(dest_pathname)):                 os.makedirs(os.path.dirname(dest_pathname))             if not file.get('Key').endswith('/'):                 resource.meta.client.download_file(bucket, file.get('Key'), dest_pathname) 

The function is called that way:

def _start():     client = boto3.client('s3')     resource = boto3.resource('s3')     download_dir(client, resource, 'clientconf/', '/tmp', bucket='my-bucket')