readinto() replacement?

Question

Copying a File using a straight-forward approach in Python is typically like this:

def copyfileobj(fsrc, fdst, length=16*1024):
    """copy data from file-like object fsrc to file-like object fdst"""
    while 1:
        buf = fsrc.read(length)
        if not buf:
            break
        fdst.write(buf)

(This code snippet is from shutil.py, by the way).

Unfortunately, this has drawbacks in my special use-case (involving threading and very large buffers) [Italics part added later]. First, it means that with each call of read() a new memory chunk is allocated and when buf is overwritten in the next iteration this memory is freed, only to allocate new memory again for the same purpose. This can slow down the whole process and put unnecessary load on the host.

To avoid this I'm using the file.readinto() method which, unfortunately, is documented as deprecated and "don't use":

def copyfileobj(fsrc, fdst, length=16*1024):
    """copy data from file-like object fsrc to file-like object fdst"""
    buffer = array.array('c')
    buffer.fromstring('-' * length)
    while True:
        count = fsrc.readinto(buffer)
        if count == 0:
            break
        if count != len(buffer):
            fdst.write(buffer.toString()[:count])
        else:
            buf.tofile(fdst)

My solution works, but there are two drawbacks as well: First, readinto() is not to be used. It might go away (says the documentation). Second, with readinto() I cannot decide how many bytes I want to read into the buffer and with buffer.tofile() I cannot decide how many I want to write, hence the cumbersome special case for the last block (which also is unnecessarily expensive).

I've looked at array.array.fromfile(), but it cannot be used to read "all there is" (reads, then throws EOFError and doesn't hand out the number of processed items). Also it is no solution for the ending special-case problem.

Is there a proper way to do what I want to do? Maybe I'm just overlooking a simple buffer class or similar which does what I want.

Answer 1

This code snippet is from shutil.py

Which is a standard library module. Why not just use it?

First, it means that with each call of read() a new memory chunk is allocated and when buf is overwritten in the next iteration this memory is freed, only to allocate new memory again for the same purpose. This can slow down the whole process and put unnecessary load on the host.

This is tiny compared to the effort required to actually grab a page of data from disk.

Answer 2

Normal Python code would not be in need off such tweaks as this - however if you really need all that performance tweaking to read files from inside Python code (as in, you are on the rewriting some server coe you wrote and already works for performance or memory usage) I'd rather call the OS directly using ctypes - thus having a copy performed as low level as I want too.

It may even be possible that simple calling the "cp" executable as an external process is less of a hurdle in your case (and it would take full advantages of all OS and filesystem level optimizations for you).