Download a file directly to S3

Question

Given a resource link, for example:

http://www.google.com/images/srpr/logo3w.png

Is there a way to download that png directly to S3 (preferably using Boto)? If so, how would this be done?

Answer 1

You can use urllib2 to fetch the file and use the response object to write it into S3 using boto.

from boto.s3.connection import S3Connection
from boto.s3.key import Key
import urllib2

request = urllib2.Request('http://www.google.com/images/srpr/logo3w.png')
response = urllib2.urlopen(request)

conn = S3Connection("MYAWSID", "MYAWSSECRET")
bucket = conn.create_bucket('MyBucket')
k = Key(bucket)
k.name = "logo3w"
k.set_contents_from_string(response.read(), {'Content-Type': response.info().gettype()})

If you don't have the packages you can install them from PIP using

pip install urllib2_file
pip install boto

Answer 2

Any 'download to S3' implicitly means 'download and then upload to S3' - whether you do that upload manually or a script or library like boto does it.

If using a script or library (boto) it would download the image to a file-system attached to the system it was running on - your local workstation or a server - and then use the AWS keys and libraries to upload it to S3.