Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Fastest way to read long[] from file?

I have a file that contains about 200,000 long values that I want to read as fast as possible into a long[]. It's for an Android app; function calls are slow (so anything involving reading a long at a time with a "for" loop will be super slow) and I need loading to be fast. What can I use? Everything I look at seems to read only bytes fast.

I've used ByteBuffer and FileChannel from the NIO package before and this seems a really fast way to load arrays of values from files. However, I cannot work out how to use this to read data into a long[]. I've tried wrapping a long[] as a LongBuffer, but I cannot see any way I can feed data from a file into a LongBuffer.

Edit: Whatever method I use, I need to be able to use Arrays.binarySearch on the long[] array at the end.

like image 730
memcom Avatar asked Sep 29 '10 16:09

memcom


People also ask

How can I read large files efficiently?

BufferedReader is used to read the file line by line. Basically, BufferedReader() is used for the processing of large files. BufferedReader is very efficient for reading. Note: Specify the size of the BufferReader or keep that size as a Default size of BufferReader.

What is the fastest way to read file in Java?

There are several ways to read a plain text file in Java e.g. you can use FileReader, BufferedReader, or Scanner to read a text file. Every utility provides something special e.g. BufferedReader provides buffering of data for fast reading, and Scanner provides parsing ability. Methods: Using BufferedReader class.

How to read bulk data from file in Java?

Approach to Read Large Files Similar to DOM parser and SAX parser for XML files, we can read a file with two approaches: Reading the complete file in memory before processing it. Reading the file content line by line and processing each line independently.


2 Answers

There is no way to cast a byte[] into a long[]. However, you could try to use a FileChannel to read the content into a ByteBuffer and then get hold of a LongBuffer through ByteBuffer.asLongBuffer from which you could get a long[] through LongBuffer.array().

You could also try FileChannel.map to get a MappedByteBuffer of the file. This may be faster than going through a FileChannel.read.

If this doesn't work you could try to use a FileChannel to read the content into a ByteBuffer and then access the longs inside it, using ByteBuffer.getLong(index).


An alternative solution. (No method-calls in the loop :-)

byte[] byteArray = new byte[longCount * 8];
FileInputStream fis = new FileInputStream("lotsoflongs");
fis.read(byteArray);
fis.close();
for (int i = 0; i < longCount; i += 8)
    longArray[i >> 3] = ((long) byteArray[0+i]        << 56) +
                        ((long)(byteArray[1+i] & 255) << 48) +
                        ((long)(byteArray[2+i] & 255) << 40) +
                        ((long)(byteArray[3+i] & 255) << 32) +
                        ((long)(byteArray[4+i] & 255) << 24) +
                              ((byteArray[5+i] & 255) << 16) +
                              ((byteArray[6+i] & 255) <<  8) +
                              ((byteArray[7+i] & 255) <<  0);

I've benchmarked a few solutions now, and this one seems to be the fastest way of doing it. Also, note that the actual bytes read in fis.read(byteArray) may be less than the actual size of byteArray. Thus, if this should be done properly, you need to put it in a loop that iterates until all bytes have been read.

like image 105
aioobe Avatar answered Oct 14 '22 06:10

aioobe


Try using a DataInputStream. Since you can easily find out the length of the file, you also know how many elements it contains (filesize / 8 byte).

 DataInputStream dataStream = new DataInputStream(inputStream);

 long count = filesize/8;
 long[] longArray = new long[count];

 for(int i=0;i<count;i++) 
     longArray[i] = dataStream.getLong();

Guess that should be enough to give you an idea.

like image 37
Tseng Avatar answered Oct 14 '22 05:10

Tseng