I have a file that contains about 200,000 long values that I want to read as fast as possible into a long[]. It's for an Android app; function calls are slow (so anything involving reading a long at a time with a "for" loop will be super slow) and I need loading to be fast. What can I use? Everything I look at seems to read only bytes fast.
I've used ByteBuffer
and FileChannel
from the NIO package before and this seems a really fast way to load arrays of values from files. However, I cannot work out how to use this to read data into a long[]. I've tried wrapping a long[] as a LongBuffer, but I cannot see any way I can feed data from a file into a LongBuffer.
Edit: Whatever method I use, I need to be able to use Arrays.binarySearch
on the long[]
array at the end.
BufferedReader is used to read the file line by line. Basically, BufferedReader() is used for the processing of large files. BufferedReader is very efficient for reading. Note: Specify the size of the BufferReader or keep that size as a Default size of BufferReader.
There are several ways to read a plain text file in Java e.g. you can use FileReader, BufferedReader, or Scanner to read a text file. Every utility provides something special e.g. BufferedReader provides buffering of data for fast reading, and Scanner provides parsing ability. Methods: Using BufferedReader class.
Approach to Read Large Files Similar to DOM parser and SAX parser for XML files, we can read a file with two approaches: Reading the complete file in memory before processing it. Reading the file content line by line and processing each line independently.
There is no way to cast a byte[]
into a long[]
. However, you could try to use a FileChannel
to read the content into a ByteBuffer
and then get hold of a LongBuffer
through ByteBuffer.asLongBuffer
from which you could get a long[]
through LongBuffer.array()
.
You could also try FileChannel.map
to get a MappedByteBuffer
of the file. This may be faster than going through a FileChannel.read
.
If this doesn't work you could try to use a FileChannel
to read the content into a ByteBuffer
and then access the long
s inside it, using ByteBuffer.getLong(index)
.
An alternative solution. (No method-calls in the loop :-)
byte[] byteArray = new byte[longCount * 8];
FileInputStream fis = new FileInputStream("lotsoflongs");
fis.read(byteArray);
fis.close();
for (int i = 0; i < longCount; i += 8)
longArray[i >> 3] = ((long) byteArray[0+i] << 56) +
((long)(byteArray[1+i] & 255) << 48) +
((long)(byteArray[2+i] & 255) << 40) +
((long)(byteArray[3+i] & 255) << 32) +
((long)(byteArray[4+i] & 255) << 24) +
((byteArray[5+i] & 255) << 16) +
((byteArray[6+i] & 255) << 8) +
((byteArray[7+i] & 255) << 0);
I've benchmarked a few solutions now, and this one seems to be the fastest way of doing it. Also, note that the actual bytes read in fis.read(byteArray)
may be less than the actual size of byteArray. Thus, if this should be done properly, you need to put it in a loop that iterates until all bytes have been read.
Try using a DataInputStream. Since you can easily find out the length of the file, you also know how many elements it contains (filesize / 8 byte).
DataInputStream dataStream = new DataInputStream(inputStream);
long count = filesize/8;
long[] longArray = new long[count];
for(int i=0;i<count;i++)
longArray[i] = dataStream.getLong();
Guess that should be enough to give you an idea.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With