How can I access s3 files in Python using urls?

Question

I want to write a Python script that will read and write files from s3 using their url's, eg:'s3:/mybucket/file'. It would need to run locally and in the cloud without any code changes. Is there a way to do this?

Edit: There are some good suggestions here but what I really want is something that allows me to do this:

 myfile = open("s3://mybucket/file", "r") 

and then use that file object like any other file object. That would be really cool. I might just write something like this for myself if it doesn't exist. I could build that abstraction layer on simples3 or boto.

Answer 1

For opening, it should be as simple as:

import urllib opener = urllib.URLopener() myurl = "https://s3.amazonaws.com/skyl/fake.xyz" myfile = opener.open(myurl) 

This will work with s3 if the file is public.

To write a file using boto, it goes a little something like this:

from boto.s3.connection import S3Connection conn = S3Connection(AWS_KEY, AWS_SECRET) bucket = conn.get_bucket(BUCKET) destination = bucket.new_key() destination.name = filename destination.set_contents_from_file(myfile) destination.make_public() 

lemme know if this works for you :)