How to get all characters within a certain UTF-8 language group?

Question

I don't know the exact technical terminology, but UTF-8 as a standard includes characters from certain language groupings, which can be observed in the Windows Character Map with a font like Arial Unicode MS.

• Latin
• Cyrillic
• Greek
• Hebrew
• Arabic
• Devnagari
• Gujrati
• Kannada
• Lao
• Hiragana
• Currency Symbols
• Box Drawings

How do I obtain a list of the characters under each set? This could be an API or just a plain list/DB somewhere on the net. I found the wiki article that lists everything, but not in an iterable form. Any ideas?

Answer 1

You can access the entire list of unicode chars at the published UnicodeData.txt which is a CSV formatted file listing every character with group information.

Grouping by class

The third column specifies the character class, in a 2 digit shortform, longforms specified here.

• letter-character -- classes Lu, Ll, Lt, Lm, Lo, or Nl
• combining-character -- classes Mn or Mc
• decimal-digit-character -- class Nd
• connecting-character -- class Pc
• formatting-character -- class Cf

Its even possible to iterate through chars of a certain group using C# LINQ:

var charInfo = Enumerable.Range(0, 0x110000)
                         .Where(x => x < 0x00d800 || x > 0x00dfff)
                         .Select(char.ConvertFromUtf32)
                         .GroupBy(s => char.GetUnicodeCategory(s, 0))
                         .ToDictionary(g => g.Key);

foreach (var ch in charInfo[UnicodeCategory.LowercaseLetter])
{
    Console.Write(ch);
}

Grouping by language

However, the language grouping is not explicitly mentioned so you'll have to parse the first word of the name to group each char by language. This is the most reliable method to do so, since every Latin unicode character begins with the prefix "Latin". Examples follow:

• Latin: Latin Capital Letter A
• Latin Extended A: Latin Small Letter C with acute
• Latin Extended B: Latin Capital Letter Tone Six
• Latin Extended Additional: Latin Capital Letter B With Dot Above