Java read file got a leading BOM [  ]

Question

I am reading a file containing keywords line by line and found a strange problem. I hope lines that following each other if their contents are the same, they should be handled only once. Like

sony
sony

only the first one is getting processed. but the problems is, java doesn't treat them as equals.

INFO: [, s, o, n, y]
INFO: [s, o, n, y]

My code looks like the following, where's the problem?

    FileReader fileReader = new FileReader("some_file.txt");
    BufferedReader bufferedReader = new BufferedReader(fileReader);
    String prevLine = "";
    String strLine
    while ((strLine = bufferedReader.readLine()) != null) {
        logger.info(Arrays.toString(strLine.toCharArray()));
        if(strLine.contentEquals(prevLine)){
            logger.info("Skipping the duplicate lines " + strLine);
            continue;
        }
        prevLine = strLine;
    }

Update:

It seems like there's leading a space in the first line, but actually not, and the trim approach doesn't work for me. They're not the same:

INFO: [, s, o, n, y]
INFO: [ , s, o, n, y]

I don't know what's the first Char added by java.

Solved: the problem was solved with BalusC's solution, thanks for pointing out it's BOM problem which helped me to find out the solution quickly.

Answer 1

Try trimming whitespace at the beginning and end of lines read. Just replace your while with:

while ((strLine = bufferedReader.readLine()) != null) {
        strLine = strLine.trim();
        logger.info(Arrays.toString(strLine.toCharArray()));
    if(strLine.contentEquals(prevLine)){
        logger.info("Skipping the duplicate lines " + strLine);
        continue;
    }
    prevLine = strLine;
}