Recursively copying Content from one path to another of s3 buckets using boto in python

Question

I am not able to find any solution for recusively copying contents from one to another in s3 buckets using boto in python.

suppose a bucket B1 contains has key structure like: B1/x/* I want to copy all the objects recursively from key like B/x/* to B/y/*

Answer 1

There is not "directory" in S3. Those "/" separator is just part of object name, that's why boto doesn't have such features. Either write a script to deal with it or use third party tools.

AWS customerapps show s3browser that provide such arbitrary directory copying functionality. The typical free version only spawn two threads to move file, the paid version allow you to specify more threads and run faster.

Or you just write script and use s3.client.copy_object to copy the file to another name, then delete them afterwards. e.g.

import boto3
s3 = boto3.client("s3")
# list_objects_v2() give more info

more_objects=True
found_token = True
while more_objects :
  if found_token :
    response= s3.list_objects_v2(
      Bucket="mybucket", 
      Prefix="B1/x/",
      Delimiter="/")
  else:   
    response= s3.list_objects_v2(
      Bucket="mybucket",
      ContinuationToken=found_token,
      Prefix="B1/x/",
      Delimiter="/")
  # use copy_object or copy_from
  for source in object_list["Contents"]:
    raw_name = source["Key"].split("/")[-1] 
    new_name = "new_structure/{}".format(raw_name)
    s3.copy_object(
      ....
    )       
    # Now check there is more objects to list
    if "NextContinuationToken" in response:
      found_token = response["NextContinuationToken"]
      more_objects = True
    else:
      more_objects = False

** IMPORTANT NOTES ** : list_object only return maximum 1000 keys per listing, MaxKey will not change the limit. So you must use list_objects_v2 and check whether NextContinuationToken is returned, to make sure the is more object, repeat it until exhausted.