Best practice for handling vertical tabs and other invalid xml characters

Question

I have an application which (like many others) takes in user input, stores it in a database and then later processes it using (amongst other things) XML tools. The application takes in free text input and like many other developers I am very careful with escaping and quoting so it can handle input containing different types of whitespace, quote characters, reserved XML characters etc.

However, occasionally a user will manage to enter a string containing a vertical tab character (hex 0B) or a form feed (hex 0C). this cannot be processed by XML tools at all and causes the app to barf.

In my application it's quite important to preserve the original input during the 'round trip' process, so i'm loath to just strip out any characters I don't like, especially things like form feed which are still occasionally used in plain text files.

is there any accepted best practice or general strategy for handling these characters when XML processing is involved?

Answer 1

Yes, unfortunately some characters are illegal in XML, and have no entity equivalent. As one of those examples, see:

http://www.jdom.org/docs/apidocs.1.1/org/jdom/Element.html#setText(java.lang.String)

which is a String setter... that can throw an exception! Vertical tab is exactly one of those characters for which there is no XML entity, nor a way to "escape" it with XML alone.

I'm working around this myself by using base64 encoding to sanitize strings that might harbor those characters. It's a bit silly, since I have to base64-encode and decode all the time, but I don't think there's a good alternative.