Obtaining unicode characters of a language in Java

Question

Is there any way in Java so that I can obtain all the Unicode characters of a particular language (for example Bengali or Arabic)?

Answer 1

The java.lang.Character class has a inner static class called UnicodeBlock. You can, for example, get the Arabic Unicode Block thusly:

Character.UnicodeBlock block = Character.UnicodeBlock.ARABIC;

By iterating over all characters (or more precisely, Unicode code points) it is possible to check each to find its Unicode Block:

public static void main(String[] args) {
    Set<Character> arabicChars = findCharactersInUnicodeBlock(Character.UnicodeBlock.ARABIC);
    Set<Character> bengaliChars = findCharactersInUnicodeBlock(Character.UnicodeBlock.BENGALI);
}

private static Set<Character> findCharactersInUnicodeBlock(final Character.UnicodeBlock block) {
    final Set<Character> chars = new HashSet<Character>();
    for (int codePoint = Character.MIN_CODE_POINT; codePoint <= Character.MAX_CODE_POINT; codePoint++) {
        if (block == Character.UnicodeBlock.of(codePoint)) {
            chars.add((char) codePoint);
        }
    }
    return chars;
}

Answer 2

Until 1.7, Java has no support for scripts in Unicode. Java has very sketchy Unicode property support, though. It is basically stuck at antemillennial incarnations of Unicode. This is a real problem. They claim they’ll catch up to Unicode 6 with JDK7, but I haven’t seen any evidence yet that they will have proper property support.

In Unicode 6.0, there are 1,051 code points that count as Arabic overall, with 1,020 of those in the Basic Multilingual Plane:

% unichars --bmp  '\p{Script=Arabic}' | wc -l
    1020

% unichars -a '\p{Script=Arabic}' | wc -l
    1051

The reason that works is that the unichars program is written in Perl, and Perl has always had excellent Unicode property support. I’m running that against Unicode 6.0; there were somewhat fewer in previous releases of Unicode. In fact, 17 new Arabic characters were added for Unicode 6.0:

 % unichars -a '\p{Script=Arabic}' '\p{Age:6.0}' | wc -l
         17

You just cannot try to use blocks for this. Scripts are different from blocks. Not all code points in a given block are of the same script. Equally important, you often find characters of a given script scattered all over in strange blocks.

For example, there are 18 non-Greek characters in the Greek block:

% unichars '\p{InGreek}' '\P{IsGreek}'ˋ | wc -l
     18

And 13 non-Arabic characters in the Arabic block:

% unichars '\p{InArabic}' '\P{IsArabic}' | wc -l
     13

Plus there are 4 Greek blocks and 4 (or 5) Arabic ones:

% uniprops -l | grep 'Block:.*Greek'
Block:Ancient_Greek_Musical_Notation
Block:Ancient_Greek_Numbers
Block:Greek
Block:Greek_And_Coptic
Block:Greek_Extended

% uniprops -l | grep 'Block:.*Arab'
Block:Arabic
Block:Arabic_Presentation_Forms_A
Block:Arabic_Presentation_Forms_B
Block:Arabic_Supplement 
Block:Old_South_Arabian

\p{Block:Greek} and \p{Greek_and_Coptic} are aliases, but the rest are all distinct.

But even if you look at all those blocks, you’ll miss some. For example:

% unichars '\p{IsGreek}' '[^\p{InAncient_Greek_Musical_Notation}\p{InAncient_Greek_Numbers}\p{InGreek}\p{InGreek_Extended}]' 
 ᴦ  7462 1D26 GREEK LETTER SMALL CAPITAL GAMMA
 ᴧ  7463 1D27 GREEK LETTER SMALL CAPITAL LAMDA
 ᴨ  7464 1D28 GREEK LETTER SMALL CAPITAL PI
 ᴩ  7465 1D29 GREEK LETTER SMALL CAPITAL RHO
 ᴪ  7466 1D2A GREEK LETTER SMALL CAPITAL PSI
 ᵝ  7517 1D5D MODIFIER LETTER SMALL BETA
 ᵞ  7518 1D5E MODIFIER LETTER SMALL GREEK GAMMA
 ᵟ  7519 1D5F MODIFIER LETTER SMALL DELTA
 ᵠ  7520 1D60 MODIFIER LETTER SMALL GREEK PHI
 ᵡ  7521 1D61 MODIFIER LETTER SMALL CHI
 ᵦ  7526 1D66 GREEK SUBSCRIPT SMALL LETTER BETA
 ᵧ  7527 1D67 GREEK SUBSCRIPT SMALL LETTER GAMMA
 ᵨ  7528 1D68 GREEK SUBSCRIPT SMALL LETTER RHO
 ᵩ  7529 1D69 GREEK SUBSCRIPT SMALL LETTER PHI
 ᵪ  7530 1D6A GREEK SUBSCRIPT SMALL LETTER CHI
 ᶿ  7615 1DBF MODIFIER LETTER SMALL THETA
 Ω  8486 2126 OHM SIGN

See the problem?

BTW, you use uniprops for more than just listing all possible properties. It can also give you the properties of any given code point:

% uniprops -a 1dbf 9e6 NEL Greek:Omicron
U+1DBF <ᶿ> \N{ MODIFIER LETTER SMALL THETA }:
    \w \pL \p{L_} \p{Lm}
    All Any Alnum Alpha Alphabetic Assigned Greek Is_Greek InPhoneticExtensionsSupplement Case_Ignorable CI Cased Changes_When_NFKC_Casefolded CWKCF L Lm Gr_Base Grapheme_Base Graph GrBase Grek ID_Continue IDC ID_Start IDS Letter L_ Modifier_Letter Lower Lowercase Print Word XID_Continue XIDC XID_Start XIDS
    Age:4.1 Bidi_Class:L Bidi_Class=Left_To_Right Bidi_Class:Left_To_Right Bc=L Block:Phonetic_Extensions_Supplement Canonical_Combining_Class:0 Canonical_Combining_Class=Not_Reordered Canonical_Combining_Class:Not_Reordered Ccc=NR Canonical_Combining_Class:NR Decomposition_Type:Non_Canon Decomposition_Type=Non_Canonical
       Decomposition_Type:Non_Canonical Dt=NonCanon Decomposition_Type:Sup Decomposition_Type=Super Decomposition_Type:Super Dt=Sup East_Asian_Width=Neutral East_Asian_Width:Neutral Grapheme_Cluster_Break:Other GCB=XX Grapheme_Cluster_Break:XX Grapheme_Cluster_Break=Other Script=Greek Hangul_Syllable_Type:NA
       Hangul_Syllable_Type=Not_Applicable Hangul_Syllable_Type:Not_Applicable Hst=NA Joining_Group:No_Joining_Group Jg=NoJoiningGroup Joining_Type:Non_Joining Jt=U Joining_Type:U Joining_Type=Non_Joining Line_Break:AL Line_Break=Alphabetic Line_Break:Alphabetic Lb=AL Numeric_Type:None Nt=None Numeric_Value:NaN Nv=NaN Present_In:4.1
       In=4.1 Present_In:5.0 In=5.0 Present_In:5.1 In=5.1 Present_In:5.2 In=5.2 Script:Greek Sc=Grek Script:Grek Sentence_Break:LO Sentence_Break=Lower Sentence_Break:Lower SB=LO Word_Break:ALetter WB=LE Word_Break:LE Word_Break=ALetter
U+09E6 <০> \N{ BENGALI DIGIT ZERO }:
    \w \d \pN \p{Nd}
    All Any Alnum Assigned Beng Bengali InBengali Is_Bengali Decimal_Number Digit Nd N Gr_Base Grapheme_Base Graph GrBase ID_Continue IDC Number Print Word XID_Continue XIDC
    Age:1.1 Script=Bengali Block=Bengali Bidi_Class:L Bidi_Class=Left_To_Right Bidi_Class:Left_To_Right Bc=L Block:Bengali Canonical_Combining_Class:0 Canonical_Combining_Class=Not_Reordered Canonical_Combining_Class:Not_Reordered Ccc=NR Canonical_Combining_Class:NR Decomposition_Type:None Dt=None East_Asian_Width=Neutral
       East_Asian_Width:Neutral Grapheme_Cluster_Break:Other GCB=XX Grapheme_Cluster_Break:XX Grapheme_Cluster_Break=Other Hangul_Syllable_Type:NA Hangul_Syllable_Type=Not_Applicable Hangul_Syllable_Type:Not_Applicable Hst=NA Joining_Group:No_Joining_Group Jg=NoJoiningGroup Joining_Type:Non_Joining Jt=U Joining_Type:U
       Joining_Type=Non_Joining Line_Break:NU Line_Break=Numeric Line_Break:Numeric Lb=NU Numeric_Type:De Numeric_Type=Decimal Numeric_Type:Decimal Nt=De Numeric_Value:0 Nv=0 Present_In:1.1 Age=1.1 In=1.1 Present_In:2.0 In=2.0 Present_In:2.1 In=2.1 Present_In:3.0 In=3.0 Present_In:3.1 In=3.1 Present_In:3.2 In=3.2 Present_In:4.0 In=4.0
       Present_In:4.1 In=4.1 Present_In:5.0 In=5.0 Present_In:5.1 In=5.1 Present_In:5.2 In=5.2 Script:Beng Script:Bengali Sc=Beng Sentence_Break:NU Sentence_Break=Numeric Sentence_Break:Numeric SB=NU Word_Break:NU Word_Break=Numeric Word_Break:Numeric WB=NU
U+0085 <U+0085> \N{ NEXT LINE (NEL) }:
    \s \v \R \pC \p{Cc}
    All Any Assigned InLatin1 C Other Cc Cntrl Common Zyyy Control Pat_WS Pattern_White_Space PatWS Space SpacePerl VertSpace White_Space WSpace
    Age:1.1 Bidi_Class:B Bidi_Class=Paragraph_Separator Bidi_Class:Paragraph_Separator Bc=B Block:Latin_1 Block=Latin_1_Supplement Block:Latin_1_Supplement Blk=Latin1 Canonical_Combining_Class:0 Canonical_Combining_Class=Not_Reordered Canonical_Combining_Class:Not_Reordered Ccc=NR Canonical_Combining_Class:NR Script=Common
       Decomposition_Type:None Dt=None East_Asian_Width=Neutral East_Asian_Width:Neutral Grapheme_Cluster_Break:CN Grapheme_Cluster_Break=Control Grapheme_Cluster_Break:Control GCB=CN Hangul_Syllable_Type:NA Hangul_Syllable_Type=Not_Applicable Hangul_Syllable_Type:Not_Applicable Hst=NA Joining_Group:No_Joining_Group Jg=NoJoiningGroup
       Joining_Type:Non_Joining Jt=U Joining_Type:U Joining_Type=Non_Joining Line_Break:Next_Line Lb=NL Line_Break:NL Line_Break=Next_Line Numeric_Type:None Nt=None Numeric_Value:NaN Nv=NaN Present_In:1.1 Age=1.1 In=1.1 Present_In:2.0 In=2.0 Present_In:2.1 In=2.1 Present_In:3.0 In=3.0 Present_In:3.1 In=3.1 Present_In:3.2 In=3.2
       Present_In:4.0 In=4.0 Present_In:4.1 In=4.1 Present_In:5.0 In=5.0 Present_In:5.1 In=5.1 Present_In:5.2 In=5.2 Script:Common Sc=Zyyy Script:Zyyy Sentence_Break:SE Sentence_Break=Sep Sentence_Break:Sep SB=SE Word_Break:Newline WB=NL Word_Break:NL Word_Break=Newline
U+039F <Ο> \N{ GREEK CAPITAL LETTER OMICRON }:
    \w \pL \p{LC} \p{L_} \p{L&} \p{Lu}
    All Any Alnum Alpha Alphabetic Assigned Greek Is_Greek InGreek Cased Cased_Letter LC Changes_When_Casefolded CWCF Changes_When_Casemapped CWCM Changes_When_Lowercased CWL Changes_When_NFKC_Casefolded CWKCF Lu L Gr_Base Grapheme_Base Graph GrBase Grek Greek_And_Coptic ID_Continue IDC ID_Start IDS Letter L_ Uppercase_Letter Print Upper
       Uppercase Word XID_Continue XIDC XID_Start XIDS
    Age:1.1 Bidi_Class:L Bidi_Class=Left_To_Right Bidi_Class:Left_To_Right Bc=L Block:Greek Block=Greek_And_Coptic Block:Greek_And_Coptic Blk=Greek Canonical_Combining_Class:0 Canonical_Combining_Class=Not_Reordered Canonical_Combining_Class:Not_Reordered Ccc=NR Canonical_Combining_Class:NR Decomposition_Type:None Dt=None
       East_Asian_Width:A East_Asian_Width=Ambiguous East_Asian_Width:Ambiguous Ea=A Grapheme_Cluster_Break:Other GCB=XX Grapheme_Cluster_Break:XX Grapheme_Cluster_Break=Other Script=Greek Hangul_Syllable_Type:NA Hangul_Syllable_Type=Not_Applicable Hangul_Syllable_Type:Not_Applicable Hst=NA Joining_Group:No_Joining_Group Jg=NoJoiningGroup
       Joining_Type:Non_Joining Jt=U Joining_Type:U Joining_Type=Non_Joining Line_Break:AL Line_Break=Alphabetic Line_Break:Alphabetic Lb=AL Numeric_Type:None Nt=None Numeric_Value:NaN Nv=NaN Present_In:1.1 Age=1.1 In=1.1 Present_In:2.0 In=2.0 Present_In:2.1 In=2.1 Present_In:3.0 In=3.0 Present_In:3.1 In=3.1 Present_In:3.2 In=3.2
       Present_In:4.0 In=4.0 Present_In:4.1 In=4.1 Present_In:5.0 In=5.0 Present_In:5.1 In=5.1 Present_In:5.2 In=5.2 Script:Greek Sc=Grek Script:Grek Sentence_Break:UP Sentence_Break=Upper Sentence_Break:Upper SB=UP Word_Break:ALetter WB=LE Word_Break:LE Word_Break=ALetter

If you find them useful, you can download the source for the uniprops and unichars programs. There’s a third in the group, uninames. All come with instructions and examples.

Even if some of those properties aren’t directly available in Java yet, it’s ok to use Perl to generate Java code if you want; I do it all the time myself. :)