How to remove accent characters from an InputStream

Question

I am trying to parse a Rss2.0 feed on Android using a Pull parser.

XmlPullParser parser = Xml.newPullParser();
parser.setInput(url.open(), null);

The prolog of the feed XML says the encoding is "utf-8". When I open the remote stream and pass this to my Pull Parser, I get invalid token, document not well formed exceptions.

When I save the XML file and open it in the browser(FireFox) the browser reports presence of Unicode 0x12 character(grave accent?) in the file and fails to render the XML.

What is the best way to handle such cases assuming that I do not have any control over the XML being returned?

Thanks.

Answer 1

Where did you find that 0x12 is the grave accent? UTF-8 has the character range 0x00-0x7F encoded the same as ASCII, and ASCII code point 0x12 is a control character, DC2, or CTRL+R.

It sounds like an encoding problem of some sort. The simplest way to resolve that is to look at the file you've saved in a hex editor. There are some things to check:

1. the byte order mark (BOM) at the beginning might confuse some XML parsers
2. even though the XML declaration says the encoding is in UTF-8, it may not actually have that encoding, and the file will be decoded incorrectly.
3. not all unicode characters are legal in XML, which is why firefox refuses to render it. In particular, the XML spec says that that 0x9, 0xA and 0xD are the only valid characters less than 0x20, so 0x12 will definitely cause compliant parsers to grumble.

If you can upload the file to pastebin or similar, I can help find the cause and suggest a resolution.

EDIT: Ok, you can't upload. That's understandable.

The XML you're getting is corrupted somehow, and the ideal course of action is to contact the party responsible for producing it, to see if the problem can be resolved.

One thing to check before doing that though - are you sure you are getting the data undisturbed? Some forms of communication (SMS) allow only 7-bit characters. This would turn 0x92 (ASCII forward tick/apostrophe - grave accent?) into 0x12. Seems like quite a coincidence, particularly if these appear in the file where you would expect an accent.

Otherwise, you will have to try to make best do with what you have:

1. although not strictly necessary, be defensive and pass "UTF-8" as the second paramter to setInput, on the parser.

2. similarly, force the parser to use another character encoding by passing a different encoding as the second parameter. Encodings to try in addtion to "UTF-8" are "iso-8859-1" and "UTF-16". A full list of supported encodings for java is given on the Sun site - you could try all of these. (I couldn't find a definitive list of supported encodings for Android.)

3. As a last resort, you can strip out invalid characters, e.g. remove all characters below 0x20 that are not whitespace (0x9,0xA and 0xD are all whitepsace.) If removing them is difficult, you can replace them instead.

For example

class ReplacingInputStream extends FilterInputStream
{
   public int read() throws IOException
   {
      int read = super.read();
      if (read!=-1 && read<0x20 && !(read==0x9 || read==0xA || read==0xB))
         read = 0x20;
      return read;          
   }
}

You wrap this around your existing input stream, and it filters out the invalid characters. Note that you could easily do more damage to the XML, or end up with nonsense XML, but equally it may allow you to get out the data you need or to more easily see where the problems lie.