How do I translate 8bit characters into 7bit characters? (i.e. Ü to U)

Question

I'm looking for pseudocode, or sample code, to convert higher bit ascii characters (like, Ü which is extended ascii 154) into U (which is ascii 85).

My initial guess is that since there are only about 25 ascii characters that are similar to 7bit ascii characters, a translation array would have to be used.

Let me know if you can think of anything else.

Answer 1

For .NET users the article in CodeProject (thanks to GvS's tip) does indeed answer the question more correctly than any other I've seen so far.

However the code in that article (in solution #1) is cumbersome. Here's a compact version:

// Based on http://www.codeproject.com/Articles/13503/Stripping-Accents-from-Latin-Characters-A-Foray-in private static string LatinToAscii(string inString) {     var newStringBuilder = new StringBuilder();     newStringBuilder.Append(inString.Normalize(NormalizationForm.FormKD)                                     .Where(x => x < 128)                                     .ToArray());     return newStringBuilder.ToString(); } 

---

To expand a bit on the answer, this method uses String.Normalize which:

Returns a new string whose textual value is the same as this string, but whose binary representation is in the specified Unicode normalization form.

Specifically in this case we use the NormalizationForm FormKD, described in those same MSDN docs as such:

FormKD - Indicates that a Unicode string is normalized using full compatibility decomposition.

For more information about unicode normalization forms, see Unicode Annex #15.

Answer 2

Most languages have a standard way to replace accented characters with standard ASCII, but it depends on the language, and it often involves replacing a single accented character with two ASCII ones. e.g. in German ü becomes ue. So if you want to handle natural languages properly it's a lot more complicated than you think it is.