Correctly generating a sitemap XML with special characters

Question

I've a program which generates XML sitemaps for Google Webmaster Tools (among other things).
GWTs is giving me errors for some sitemaps because the URLs contain character sequences like ã¾, ã‹, ã€, etc. **

Sitemap specification says:

Your Sitemap file must be UTF-8 encoded (you can generally do this when you save the file). As with all XML files, any data values (including URLs) must use entity escape codes for the characters listed: &, ', ", <, >.

The special characters are escaped in the XML files (with HTML entities). XML file snippet:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
    <url>
        <loc>http://domain/folder/listing-&#227;&#129;.shtml</loc>
        ...

Are my URLs UTF-8 encoded? If not, how do I do this in Java? The following is the line in my program where I add the URL to the sitemap:

    siteMap.addUrl(StringEscapeUtils.escapeXml(countryName+"/"+twoCharFile.getRelativeFileName().toLowerCase()));

I'm not sure which ones are causing the error, probably the first two examples.

Answer 1

Try using URLEncoder.encode(stringToBeEncoded, "UTF-8") to encode the url.

Answer 2

URLs must be percent-encoded as per the URI spec.

For example, the code point U+00e3 (ã) would become the encoded sequence %C3%A3.

When a URI is emitted in an XML document, it must conform to the markup requirements for XML.

For example, the URI http://foo/bar?a=b&x=%C3%A3 becomes http://foo/bar?a=b&x=%C3%A3. The ampersand is an escape character in XML.

You can find a detailed discussion of URI encoding here.

Answer 3

Don't confuse percentage encoding of non-ASCII characters in URLs with XML entity escapes of characters in URLs. You need to do both when creating XML sitemaps.

In honesty from reading your original post, it seems something funky is going on because the characters you mention remind me of when an unsuccessful conversion has taken place :)

Are you sure those characters truly are part of your URLs when using UTF-8?

Answer 4

All non-ascii characters in URL has to be 'x-url-encoding' encoded.

Here is the wiki link that explains it: http://en.wikipedia.org/wiki/Percent-encoding.

In addition all XML special symbols (&, >, <, etc.) also have to be escaped.

Jai's answer shows the correct method to x-url-encode arbitrary string. Note, however, that it does not do XML escaping.