Write in memory object to S3 via boto3

Question

I am attempting to write files directly to S3 without creating a local file which is then uploaded.

I am using cStringIO to generate a file in memory, but I am having trouble figuring out the proper way to upload it in boto3.

def writetos3(sourcedata, filename, folderpath):
     s3 = boto3.resource('s3')
     data = open(sourcedata, 'rb')
     s3.Bucket('bucketname').put_object(Key= folderpath + "/" + filename, Body=data)

Above is the standard boto3 method that I was using previously with the local file, it does not work without a local file, I get the following error: coercing to Unicode: need string or buffer, cStringIO.StringO found .

Because the in memory file (I believe) is already considered open, I tried changing it to the code below, but it still does not work, no error is given the script simply hangs on the last line of the method.

def writetos3(sourcedata, filename, folderpath):
    s3 = boto3.resource('s3')
    s3.Bucket('bucketname').put_object(Key= folderpath + "/" + filename, Body=sourcedata)

Just for more info, the value I am attempting to write looks like this

(cStringIO.StringO object at 0x045DC540)

Does anyone have an idea of what I am doing wrong here?

Answer 1

It looks like you want this:

    data = open(sourcedata, 'rb').decode()

It defaults to utf8. Also, I encourage you to run your code under python3, and to use appropriate language tags for your question.