Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Fastest way of reading relatively huge byte-files in Java

what's the probably fastest way of reading relatively huge files with Java's I/O-methods? My current solution uses the BufferedInputStream saving to an byte-array with 1024 bytes allocated to it. Each buffer is than saved in an ArrayList for later use. The whole process is called via a separate thread (callable-interface).

Not very fast though.

    ArrayList<byte[]> outputArr = new ArrayList<byte[]>();      
    try {
        BufferedInputStream reader = new BufferedInputStream(new FileInputStream (dir+filename));

        byte[] buffer = new byte[LIMIT]; // == 1024 
            int i = 0;
            while (reader.available() != 0) {
                reader.read(buffer);
                i++;
                if (i <= LIMIT){
                    outputArr.add(buffer);
                    i = 0;
                    buffer = null;
                    buffer = new byte[LIMIT];
                }
                else continue;              
            }

         System.out.println("FileReader-Elements: "+outputArr.size()+" w. "+buffer.length+" byte each.");   
like image 929
chollinger Avatar asked Feb 01 '12 10:02

chollinger


People also ask

How large is a byte in Java?

byte: The byte data type is an 8-bit signed two's complement integer. It has a minimum value of -128 and a maximum value of 127 (inclusive). The byte data type can be useful for saving memory in large arrays, where the memory savings actually matters.

How much can a byte store in Java?

There are eight primitive data types in Java: byte : This data type can store whole numbers ranging from -128 to 127. The size of Byte in the storage is 1 byte. short : This data type has the storage size of 2 bytes and can store numbers from -32,768 to 32,767.


1 Answers

I would use a memory mapped file which is fast enough to do in the same thread.

final FileChannel channel = new FileInputStream(fileName).getChannel();
MappedByteBuffer buffer = channel.map(FileChannel.MapMode.READ_ONLY, 0, channel.size());

// when finished
channel.close();

This assumes the file is smaller than 2 GB and will take 10 milli-seconds or less.

like image 84
Peter Lawrey Avatar answered Sep 29 '22 13:09

Peter Lawrey