How to check if boto3 S3.Client.upload_fileobj succeeded?

Question

I want to save the result of a long running job on S3. The job is implemented in Python, so I'm using boto3. The user guide says to use S3.Client.upload_fileobj for this purpose which works fine, except I can't figure out how to check if the upload has succeeded. According to the documentation, the method doesn't return anything and doesn't raise an error. The Callback param seems to be intended for progress tracking instead of error checking. It is also unclear if the method call is synchronous or asynchronous.

If the upload failed for any reason, I would like to save the contents to the disk and log an error. So my question is: How can I check if a boto3 S3.Client.upload_fileobj call succeeded and do some error handling if it failed?

Answer 1

I use a combination of head_object and wait_until_exists.

import boto3

from botocore.exceptions import ClientError, WaiterError

session = boto3.Session()
s3_client = session.client('s3')
s3_resource = session.resource('s3')


def upload_src(src, filename, bucketName):
    success = False

    try:
        bucket = s3_resource.Bucket(bucketName)
    except ClientError as e:
        bucket = None

    try:
        # In case filename already exists, get current etag to check if the 
        # contents change after upload
        head = s3_client.head_object(Bucket=bucketName, Key=filename)
    except ClientError:
        etag = ''
    else:
        etag = head['ETag'].strip('"')

    try:
        s3_obj = bucket.Object(filename)
    except ClientError, AttributeError:
        s3_obj = None

    try:
        s3_obj.upload_fileobj(src)
    except ClientError, AttributeError:
        pass
    else:
        try:
            s3_obj.wait_until_exists(IfNoneMatch=etag)
        except WaiterError as e:
            pass
        else:
            head = s3_client.head_object(Bucket=bucketName, Key=filename)
            success = head['ContentLength']

    return success

Answer 2

There is a wait_until_exists() helper function that seems to be for this purpose in the boto3.resource object.

This is how we are using it:

s3_client.upload_fileobj(file, BUCKET_NAME, file_path)
s3_resource.Object(BUCKET_NAME, file_path).wait_until_exists()