Why does my file not get read correctly when running through ANT

Question

I have a suite of unit tests I run in Eclipse which all work fine. They depend on data loaded from a very large > 20MB file. However, when I run the unit tests from ANT the tests fail because some of the data is not loaded. What happens is my file reading mechanism does not read the entire file, it just stops , without giving any error after reading about 10,000 of 900,000 lines

Here is my file reading code

    private static void initializeListWithFileContents(
        TreeMap<String, List<String>> treeMap, String fileName)
{
    File file = new File(fileName);
    Scanner scanner = null;
    int count = 0;
    try
    {

        scanner = new Scanner(file);
        while (scanner.hasNextLine())
        {
            String line = scanner.nextLine().toLowerCase().trim();      
            String[] tokens = line.split(" ");

            if (tokens.length == 3)
            {
                String key = tokens[0] + tokens[1];
                if (treeMap.containsKey(key))
                {
                    List<String> list = treeMap.get(key);
                    list.add(tokens[2]);
                }
                else
                {
                    List<String> list = new ArrayList<String>();
                    list.add(tokens[2]);
                    treeMap.put(key, list);
                }
                count++;
            }
        }           
        scanner.close();
    }

    catch (IOException ioe)
    {
        ioe.printStackTrace();
    }
    System.out.println(count + " rows added");
}

This is part of a Web app. The web app also works fine, the entire file gets loaded to memory. If the data my Unit tests depends on are contained in the first 10,000 lines then unit tests pass ok with ANT. The only thing I can think of is it must be a memory issue but why then do I not get an exception thrown. I run my ANT target from within Eclipse. It is configured with the same JVM args as my Eclipse JUnit runner is , ie -Xms512m -Xmx1234m. I know it picks these up correctly because if ANT launches with the default JVM parameters then it will fail with Heap error. Any other ideas what I could check?

Answer 1

The Scanner type swallows I/O errors. You must check for errors explicitly using the ioException() method.

If the problem is an encoding error you need to pass the encoding of the file explicitly when you instantiate the scanner.

If the file is a corrupt text file, you may need to provide your own reader that does more fault-tolerant decoding. This should be avoided if possible as it is less correct.