In Python I have a file stream, and I want to copy some part of it into a StringIO
. I want this to be fastest as possible, with minimum copy.
But if I do:
data = file.read(SIZE)
stream = StringIO(data)
I think 2 copies was done, no? One copy into data from file, another copy inside StringIO
into internal buffer. Can I avoid one of the copies? I don't need temporary data
, so I think one copy should be enough
In short: you can't avoid 2 copies using StringIO.
Some assumptions:
file.read(SOME_BYTE_COUNT)
if your file is binary.Long answer: Since python strings are immutable and the StringIO buffer is not, a copy will have to be made sooner or later; otherwise you'd be altering an immutable object! For what you want to be possible, the StringIO object would need to have a dedicated method that read directly from a file object given as an argument. There is no such method.
Outside of StringIO, there are solutions that avoid the extra copy. Off the top of my head, this will read a file directly into a modifiable byte array, no extra copy:
import numpy as np
a = np.fromfile("filename.ext", dtype="uint8")
It may be cumbersome to work with, depending on the usage you intend, since it's an array of values from 0 to 255, not an array of characters. But it's functionally equivalent to a StringIO object, and using np.fromstring
, np.tostring
, np.tofile
and slicing notation should get you where you want. You might also need np.insert
, np.delete
and np.append
.
I'm sure there are other modules that will do similar things.
TIMEIT:
How much does all this really matter? Well, let's see. I've made a 100MB file, largefile.bin
. Then I read in the file using both methods and change the first byte.
$ python -m timeit -s "import numpy as np" "a = np.fromfile('largefile.bin', 'uint8'); a[0] = 1" 10 loops, best of 3: 132 msec per loop $ python -m timeit -s "from cStringIO import StringIO" "a = StringIO(); a.write(open('largefile.bin').read()); a.seek(0); a.write('1')" 10 loops, best of 3: 203 msec per loop
So in my case, using StringIO is 50% slower than using numpy.
Lastly, for comparison, editing the file directly:
$ python -m timeit "a = open('largefile.bin', 'r+b'); a.seek(0); a.write('1')" 10000 loops, best of 3: 29.5 usec per loop
So, it's nearly 4500 times faster. Of course, it's extremely dependent on what you're going to do with the file. Altering the first byte is hardly representative. But using this method, you do have a head start on the other two, and since most OS's have good buffering of disks, the speed may be very good too.
(If you're not allowed to edit the file and so want to avoid the cost of making a working copy, there are a couple of possible ways to increase the speed. If you can choose the filesystem, Btrfs has a copy-on-write file copy operation -- making the act of taking a copy of a file virtually instant. The same effect can be achieved using an LVM snapshot of any filesystem.)
No, there is not an extra copy made. The buffer used to store the data is the same. Both data
and the internal attribute accessible using StringIO.getvalue()
are different names for the same data.
Python 2.7 (r27:82500, Jul 30 2010, 07:39:35)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import StringIO
>>> data = open("/dev/zero").read(1024)
>>> hex(id(data))
'0xea516f0'
>>> stream = StringIO.StringIO(data)
>>> hex(id(stream.getvalue()))
'0xea516f0'
A quick skim through the source shows that cStringIO
doesn't make a copy on construction either, but it does make a copy on calling cStringIO.getvalue()
, so I can't repeat the above demonstration.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With