IOError: [Errno 22] Invalid argument when reading/writing large bytestring

Question

I'm getting

IOError: [Errno 22] Invalid argument

when I try to write a large bytestring to disk with f.write(), where f was opened with mode wb.

I've seen lots of people online getting this error when using a Windows network drive, but I'm on OSX (10.7 when I originally asked the question but 10.8 now, with a standard HFS+ local filesystem). I'm using Python 3.2.2 (happens on both a python.org binary and a homebrew install). I don't see this problem with the system Python 2.7.2.

I also tried mode w+b based on this Windows bug workaround, but of course that didn't help.

The data is coming from a large numpy array (almost 4GB of floats). It works fine if I manually loop over the string and write it out in chunks. But because I can't write it all in one pass, np.save and np.savez fail -- since they just use f.write(ary.tostring()). I get a similar error when I try to save it into an existing HDF5 file with h5py.

Note that I get the same problem when reading a file opened with file(filename, 'rb'): f.read() gives this IOError, while f.read(chunk_size) for reasonable chunk_size works.

Any thoughts?

Answer 1

This appears to be a general OSX bug with fread / fwrite and so isn't really fixable by a Python user. See numpy #3858, this torch7 commit, this SO question/answer, ....

Supposedly it's been fixed in Mavericks, but I'm still seeing the issue.

Python 2 may have worked around this or its io module may have always buffered large reads/writes; I haven't investigated thoroughly.