Upload image available at public URL to S3 using boto

Question

I'm working in a Python web environment and I can simply upload a file from the filesystem to S3 using boto's key.set_contents_from_filename(path/to/file). However, I'd like to upload an image that is already on the web (say https://pbs.twimg.com/media/A9h_htACIAAaCf6.jpg:large).

Should I somehow download the image to the filesystem, and then upload it to S3 using boto as usual, then delete the image?

What would be ideal is if there is a way to get boto's key.set_contents_from_file or some other command that would accept a URL and nicely stream the image to S3 without having to explicitly download a file copy to my server.

def upload(url):     try:         conn = boto.connect_s3(settings.AWS_ACCESS_KEY_ID, settings.AWS_SECRET_ACCESS_KEY)         bucket_name = settings.AWS_STORAGE_BUCKET_NAME         bucket = conn.get_bucket(bucket_name)         k = Key(bucket)         k.key = "test"         k.set_contents_from_file(url)         k.make_public()                 return "Success?"     except Exception, e:             return e 

Using set_contents_from_file, as above, I get a "string object has no attribute 'tell'" error. Using set_contents_from_filename with the url, I get a No such file or directory error . The boto storage documentation leaves off at uploading local files and does not mention uploading files stored remotely.

Answer 1

Here is how I did it with requests, the key being to set stream=True when initially making the request, and uploading to s3 using the upload.fileobj() method:

import requests import boto3  url = "https://upload.wikimedia.org/wikipedia/en/a/a9/Example.jpg" r = requests.get(url, stream=True)  session = boto3.Session() s3 = session.resource('s3')  bucket_name = 'your-bucket-name' key = 'your-key-name' # key is the name of file on your bucket  bucket = s3.Bucket(bucket_name) bucket.upload_fileobj(r.raw, key)