Converting str data to file object in Python

Question

I am posting videos to Google Cloud Buckets and a signed PUT url does the trick. However, if the file size is greater than 10MB it will not work, so I found an open source that will allow me to do this however, it uses a file like object.

def read_in_chunks(file_object, chunk_size=65536):
while True:
    data = file_object.read(chunk_size)
    if not data:
        break
    yield data

def main(file, url):
content_name = str(file)
content_path = os.path.abspath(file)
content_size = os.stat(content_path).st_size

print content_name, content_path, content_size

f = open(content_path)

index = 0
offset = 0
headers = {}

for chunk in read_in_chunks(f):
    offset = index + len(chunk)
    headers['Content-Type'] = 'application/octet-stream'
    headers['Content-length'] = content_size
    headers['Content-Range'] = 'bytes %s-%s/%s' % (index, offset, content_size)
    index = offset
    try:
        r = requests.put(url, data=chunk, headers=headers)
        print "r: %s, Content-Range: %s" % (r, headers['Content-Range'])
    except Exception, e:
        print e

The way that I was uploading videos was passing in json formatted data.

class GetData(webapp2.RequestHandler):
def post(self):
    data = self.request.get('file')

Then all I did was a request.put(url, data=data). This worked seamlessly.

How do I convert this data, that Python recognizes as str to a file like object?

Answer 1

A so called 'file-like' object is in most cases just an object that implements the Python buffer interface; that is, has methods like read, write, seek, and so on.

The standard library module for buffer interface tools is called io. You're looking for either io.StringIO or io.BytesIO, depending on the type of data you have — if it's a unicode encoded string, you're supposed to use io.StringIO, but you're probably working with a raw bytestream (such as in an image file) as opposed to just text, so io.BytesIO is what you're looking for. When working with files, this is the same distinction as doing open(path, 'r') for unicode files and open(path, 'rb') for raw processing of the bytes.

Both classes take the data for the file-like object as the first parameter, so you just do:

f = io.BytesIO(b'test data')

After this, f will be an object that works just like a file, except for the fact that it holds its data in memory as opposed to on disk.