Why is my String returning "\ufffd\ufffdN a m e"

Question

This is my method

public void readFile3()throws IOException
{
    try
    {
        FileReader fr = new FileReader(Path3);
        BufferedReader br = new BufferedReader(fr);
        String s = br.readLine();
        int a =1;
        while( a != 2)
        {
            s = br.readLine();
            a ++; 

        }
        Storage.add(s);

        br.close();

    }
    catch(IOException e)
    {
        System.out.println(e.getMessage());
    }
}

For some reason I am unable to read the file which only contains this " Name Intel(R) Core(TM) i5-2500 CPU @ 3.30GHz "

When i debug the code the String s is being returned as "\ufffd\ufffdN a m e" and i have no clue as to where those extra characters are coming from.. This is preventing me from properly reading the file.

Answer 1

\ufffd is the replacement character in unicode, it is used when you try to read a code that has no representation in unicode. I suppose you are on a Windows platform (or at least the file you read was created on Windows). Windows supports many formats for text files, the most common is Ansi : each character is represented but its ansi code.

But Windows can directly use UTF16, where each character is represented by its unicode code as a 16bits integer so with 2 bytes per character. Those files uses special markers (Byte Order Mark in Windows dialect) to say :

• that the file is encoded with 2 (or even 4) bytes per character
• the encoding is little or big endian

(Reference : Using Byte Order Marks on MSDN)

As you write after the first two replacement characters N a m e and not Name, I suppose you have an UTF16 encoded text file. Notepad can transparently edit those files (without even saying you the actual format) but other tools do have problems with those ... The excellent vim can read files with different encodings and convert between them.

If you want to use directly this kind of file in java, you have to use the UTF-16 charset. From JaveSE 7 javadoc on Charset : UTF-16 Sixteen-bit UCS Transformation Format, byte order identified by an optional byte-order mark

Answer 2

You must specify the encoding when reading the file, in your case probably is UTF-16.

Reader reader = new InputStreamReader(new FileInputStream(fileName), "UTF-16");
BufferedReader br = new BufferedReader(reader);

Check the documentation for more details: InputStreamReader class.