Copying files with python

Question

I'm relatively new to python, and am part of the way through "Learning python the Hard Way," but have a question.

So, from what I've read, if you want to make a copy of a text file, you can just open and read it contents to a variable, then write this variable to a different file. I've tested it out with images, and it actually seems to work. Are there downsides to this method of copying that I'll run into later, and are there any file types it specifically won't work for?

Thank you very much!

Answer 1

You should use shutil.copyfile() or shutil.copyfileobj() instead, which does this efficiently and correctly using a buffer.

Not that it is particularly hard, shutil.copyfileobj() is implemented as:

def copyfileobj(fsrc, fdst, length=16*1024):
    """copy data from file-like object fsrc to file-like object fdst"""
    while 1:
        buf = fsrc.read(length)
        if not buf:
            break
        fdst.write(buf)

This makes sure that your memory isn't filled up by a big file, by reading the file in chunks instead. Also, .read() is not guaranteed to return all of the data of the file, you could end up not copying all of the data if you don't loop until .read() returns an empty string.

Answer 2

One caveat is that .read() isn't necessarily guaranteed to read the entire file at once, so you must make sure to repeat the read/write cycle until all of the data has been copied. Another is that there may not be enough memory to read all of the data at once, in which case you'll need to perform multiple partial reads and writes in order to complete the copy.