java.util.Scanner malfunctioning while reading large files

Question

I wrote a program where I used a Scanner to read lines from log files and parse each line to find something important. It is important that I read every line of the log file. I wrote the following piece of code to scan each line

Scanner s = new Scanner(new File("Large.log"));
while(s.hasNextLine())
{
    String line = s.nextLine();
    //do the processing of the log line
}

The above code behaves in a weird manner. It stops reading lines after a random number of lines [around after 1 million lines]. I modified the above code to check the last line read and also checked the log file using Notepad++. There were a lot of lines remaining in the file after that particular line. I added another System.out.println(s.hasNextLine()) after the end of the while loop and it prints false.

However if I try to do the above using a BufferedReader the program works fine. Is there any limitation with the util IO classes in Java?

Answer 1

This sounds like a large file support issue with your particular JVM implementation. It is a common problem for a lot of standard file I/O to not work with files > 4 GB on 32-bit OSs. There are typically alternative versions of the file APIs to explicitly support large files, but the person implementing the JVM would have to remember to use those. Out of curiosity what OS are you using and is it 64-bit?