raise ValueError('Fileobj must implement read')

Question

I am getting the file size and uploading it to S3:

def transfer_file_from_ftp_to_s3(bucket_name, ftp_file_path, s3_file_path, ftp_username, ftp_password, chunk_size):
ftp_connection = open_ftp_connection(FTP_HOST, int(FTP_PORT), ftp_username, ftp_password) 
ftp_file = ftp_connection.file(ftp_file_path, 'r')
s3_connection = boto3.client('s3')
ftp_file_size = ftp_file._get_size()

if ftp_file_size <= int(chunk_size):
    #upload file in one go
    print('Transferring complete File from FTP to S3...')
    ftp_file_data = ftp_file.read()
    s3_connection.upload_fileobj(ftp_file_data, bucket_name, s3_file_path)
    print('Successfully Transferred file from FTP to S3!')
    ftp_file.close()

I got this error message below:

Transferring complete File from FTP to S3...
Traceback (most recent call last):

  File "/Users/admin/anaconda2/lib/python2.7/site-packages/boto3/s3/inject.py", line 520, in upload_fileobj
    raise ValueError('Fileobj must implement read')

ValueError: Fileobj must implement read

Can you give me some pointers? Thank you.

Answer 1

A more general answer on how to upload a byte representation of your data would make use of the IO package.

s3 = boto3.resource('s3')
bucket = s3.Bucket(name="your-bucket-name-here")
data = requests.get('https://www.google.de/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png').content
bucket.upload_fileobj(io.BytesIO(data), 'googleLogoTestUpload.png')

I'm using the request package which must be downloaded using pip first. This answer is based on jwodder's answer on another question

Answer 2

You're uploading the data you read from the file object, but the method name suggests that you should pass the file object itself:

s3_connection.upload_fileobj(ftp_file, bucket_name, s3_file_path)