Writing a BytesIO object to a file, 'efficiently'

Question

So a quick way to write a BytesIO object to a file would be to just use:

with open('myfile.ext', 'wb') as f:     f.write(myBytesIOObj.getvalue()) myBytesIOObj.close() 

However, if I wanted to iterate over the myBytesIOObj as opposed to writing it in one chunk, how would I go about it? I'm on Python 2.7.1. Also, if the BytesIO is huge, would it be a more efficient way of writing by iteration?

Thanks

Answer 1

shutil has a utility that will write the file efficiently. It copies in chunks, defaulting to 16K. Any multiple of 4K chunks should be a good cross platform number. I chose 131072 rather arbitrarily because really the file is written to the OS cache in RAM before going to disk and the chunk size isn't that big of a deal.

import shutil  myBytesIOObj.seek(0) with open('myfile.ext', 'wb') as f:     shutil.copyfileobj(myBytesIOObj, f, length=131072) 

BTW, there was no need to close the file object at the end. with defines a scope, and the file object is defined inside that scope. The file handle is therefore closed automatically on exit from the with block.