Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Should I use `readinto` method of python file or not?

I've recently meet the readinto method of file object (in Python 2.7), it is similar to fread in C. It seems to be convenient and powerful in some case. I plan to use it to read several files into one pre-allocated numpy array without data copied.

e.g.

a = np.empty(N)
b = memoryview(a)
fp1.readinto(b[0:100])
fp2.readinto(b[100:200])

and

fp1.readinto(b[0:100])
fp1.seek(400, 1)
fp1.readinto(b[100:200])

I've used Cython and fread to do this before I met readinto. So I'm very happy to know a pure python solution.

However its document string says,

file.readinto?
Type:        method_descriptor
String form: <method 'readinto' of 'file' objects>
Namespace:   Python builtin
Docstring:   readinto() -> Undocumented.  Don't use this; it may go away.

Don't use this? What happend?

So I'm confused, should I use readinto or not? It may cause any unwanted problem?

Is there any alternative implementation for the code above without readinto but also avoid data copy? (To avoid copy means np.concatenate or np.stack is not a good choice.)

Any sugguestion is welcome! Thank you.

-------upate-------

It seems that I can use io.FileIO in standard library instead of the build-in function open. It looks OK so I've post it as an answer.

Any comment or other solution is still welcome!

-------upate-------

If you meet the same problem, you may want to have a look at the comments below by
Andrea Corbellini and Padraic Cunningham.

like image 997
Syrtis Major Avatar asked Jan 13 '16 14:01

Syrtis Major


People also ask

What is Readinto in Python?

readinto(b, /) Read bytes into a pre-allocated, writable bytes-like object b, and return the number of bytes read. For example, b might be a bytearray . If the object is in non-blocking mode and no bytes are available, None is returned.

What is a file like object in Python?

File objects are also called file-like objects or streams. There are actually three categories of file objects: raw binary files, buffered binary files and text files. Their interfaces are defined in the io module. The canonical way to create a file object is by using the open() function.


1 Answers

You may use io.FileIO in python standard library instead of the build-in function open or file, if you are not sure with file.readinto.

Here's the docstring:

#io.FileIO.readinto?
Type:        method_descriptor
String form: <method 'readinto' of '_io.FileIO' objects>
Docstring:   readinto() -> Same as RawIOBase.readinto().

The document of io.RawIOBase.readinto can be found here.

class io.RawIOBase

...

readinto(b)

Read up to len(b) bytes into bytearray b and return the number of bytes read. If the object is in non-blocking mode and no bytes are available, None is returned.

It's available in both Python 2 and 3.

like image 88
Syrtis Major Avatar answered Sep 18 '22 15:09

Syrtis Major