Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Quickest way to read text-file line by line in Java

For log processing my application needs to read text files line by line. First I used the function readLine() of BufferedReader but I read on the internet that BufferedReader is slow when reading files.
Afterwards I tried to use FileInputStream together with a FileChannel and MappedByteBuffer but in this case there's no function similar to readLine() so I search my text for a line-break and process it:

    try {
        FileInputStream f = new FileInputStream(file);
        FileChannel ch = f.getChannel( );
        MappedByteBuffer mb = ch.map(FileChannel.MapMode.READ_ONLY, 0L, ch.size());
        byte[] bytes = new byte[1024];
        int i = 0;
        while (mb.hasRemaining()) {
            byte get = mb.get();
            if(get == '\n') {
                if(ra.run(new String(bytes)))
                    cnt++;
                for(int j = 0; j<=i; j++)
                    bytes[j] = 0;
                i = 0;
            }
            else
                bytes[i++] = get;
        }
    } catch(Exception ex) {
        ex.printStackTrace();
    }

I know this is probably not a good way to implement it but when I just read the text-file in bytes it is 3 times faster then using BufferedReader but calling new String(bytes) creates a new String and makes the program even slower then when using a BufferedReader.
So I wanted to ask what is the fastest way to read a text-file line by line? Some say BufferedReader is the only solution to this problem.

P.S.: ra is an instance of RunAutomaton from the dk.brics.Automaton library.

like image 646
Yoni Avatar asked Apr 27 '11 06:04

Yoni


People also ask

What is the easiest way to read text files line by line in Java 8?

Java 8 has added a new method called lines() in the Files class which can be used to read a file line by line in Java. The beauty of this method is that it reads all lines from a file as Stream of String, which is populated lazily as the stream is consumed.

Which method is used to read file line by line?

We can use java. io. BufferedReader readLine() method to read file line by line to String. This method returns null when end of file is reached.

How do you read the second line of a text file in Java?

The hasNextLine() method returns true if there is another line in the input of this scanner, but the scanner itself does not advance past any input or read any data at this point. To read the line and move on, we should use the nextLine() method.

How read data from line from file in Java?

Using the Java BufferedRedaer class is the most common and simple way to read a file line by line in Java. It belongs to java.io package. Java BufferedReader class provides readLine() method to read a file line by line.


1 Answers

Using plain BufferedReader I got 100+ MB/s. It is highly likely that the speed you can read the data from disk is your bottle neck, so how you do the reading won't make much difference.

BufferedReader is not the only solution, but it is fast enough for 99% of use cases, so why make things more complicated than they need to be?

like image 83
Peter Lawrey Avatar answered Sep 23 '22 15:09

Peter Lawrey