Are there character collections for all international full stop punctuations?

Question

I am trying to parse utf-8 strings into "bite sized" segments. For example, I would like to break down a text into "sentences".

Is there a comprehensive collection of characters (or regex) that correspond to end of sentences in all languages? I'm looking for something that would capture the Latin period, exclamation and interrogation marks, the Chinese and Japanese full stop, etc.

Something like the above but for the equivalent of a comma would be great too.

Answer 1

You need to look at code points with the \p{Sentence_Break=STerm} or \p{Sentence_Break=ATerm} properties that also have the \p{Terminal_Punctuation} property. Running the unichars script against Unicode v6.1, we learn that these code points meet all those criteria:

$ unichars -gas '[\p{Sentence_Break=STerm}\p{Sentence_Break=ATerm}]' '\p{Terminal_Punctuation}'
U+00021 ‭ !  GC=Po SC=Common       EXCLAMATION MARK
U+0002E ‭ .  GC=Po SC=Common       FULL STOP
U+0003F ‭ ?  GC=Po SC=Common       QUESTION MARK
U+00589 ‭ ։  GC=Po SC=Common       ARMENIAN FULL STOP
U+0061F ‭ ؟  GC=Po SC=Common       ARABIC QUESTION MARK
U+006D4 ‭ ۔  GC=Po SC=Arabic       ARABIC FULL STOP
U+00700 ‭ ܀  GC=Po SC=Syriac       SYRIAC END OF PARAGRAPH
U+00701 ‭ ܁  GC=Po SC=Syriac       SYRIAC SUPRALINEAR FULL STOP
U+00702 ‭ ܂  GC=Po SC=Syriac       SYRIAC SUBLINEAR FULL STOP
U+007F9 ‭ ߹  GC=Po SC=Nko          NKO EXCLAMATION MARK
U+00964 ‭ ।  GC=Po SC=Common       DEVANAGARI DANDA
U+00965 ‭ ॥  GC=Po SC=Common       DEVANAGARI DOUBLE DANDA
U+0104A ‭ ၊  GC=Po SC=Myanmar      MYANMAR SIGN LITTLE SECTION
U+0104B ‭ ။  GC=Po SC=Myanmar      MYANMAR SIGN SECTION
U+01362 ‭ ።  GC=Po SC=Ethiopic     ETHIOPIC FULL STOP
U+01367 ‭ ፧  GC=Po SC=Ethiopic     ETHIOPIC QUESTION MARK
U+01368 ‭ ፨  GC=Po SC=Ethiopic     ETHIOPIC PARAGRAPH SEPARATOR
U+0166E ‭ ᙮  GC=Po SC=Canadian_Aboriginal CANADIAN SYLLABICS FULL STOP
U+01803 ‭ ᠃  GC=Po SC=Common       MONGOLIAN FULL STOP
U+01809 ‭ ᠉  GC=Po SC=Mongolian    MONGOLIAN MANCHU FULL STOP
U+01944 ‭ ᥄  GC=Po SC=Limbu        LIMBU EXCLAMATION MARK
U+01945 ‭ ᥅  GC=Po SC=Limbu        LIMBU QUESTION MARK
U+01AA8 ‭ ᪨  GC=Po SC=Tai_Tham     TAI THAM SIGN KAAN
U+01AA9 ‭ ᪩  GC=Po SC=Tai_Tham     TAI THAM SIGN KAANKUU
U+01AAA ‭ ᪪  GC=Po SC=Tai_Tham     TAI THAM SIGN SATKAAN
U+01AAB ‭ ᪫  GC=Po SC=Tai_Tham     TAI THAM SIGN SATKAANKUU
U+01B5A ‭ ᭚  GC=Po SC=Balinese     BALINESE PANTI
U+01B5B ‭ ᭛  GC=Po SC=Balinese     BALINESE PAMADA
U+01B5E ‭ ᭞  GC=Po SC=Balinese     BALINESE CARIK SIKI
U+01B5F ‭ ᭟  GC=Po SC=Balinese     BALINESE CARIK PAREREN
U+01C3B ‭ ᰻  GC=Po SC=Lepcha       LEPCHA PUNCTUATION TA-ROL
U+01C3C ‭ ᰼  GC=Po SC=Lepcha       LEPCHA PUNCTUATION NYET THYOOM TA-ROL
U+01C7E ‭ ᱾  GC=Po SC=Ol_Chiki     OL CHIKI PUNCTUATION MUCAAD
U+01C7F ‭ ᱿  GC=Po SC=Ol_Chiki     OL CHIKI PUNCTUATION DOUBLE MUCAAD
U+0203C ‭ ‼  GC=Po SC=Common       DOUBLE EXCLAMATION MARK
U+0203D ‭ ‽  GC=Po SC=Common       INTERROBANG
U+02047 ‭ ⁇  GC=Po SC=Common       DOUBLE QUESTION MARK
U+02048 ‭ ⁈  GC=Po SC=Common       QUESTION EXCLAMATION MARK
U+02049 ‭ ⁉  GC=Po SC=Common       EXCLAMATION QUESTION MARK
U+02E2E ‭ ⸮  GC=Po SC=Common       REVERSED QUESTION MARK
U+03002 ‭ 。 GC=Po SC=Common       IDEOGRAPHIC FULL STOP
U+0A4FF ‭ ꓿  GC=Po SC=Lisu         LISU PUNCTUATION FULL STOP
U+0A60E ‭ ꘎  GC=Po SC=Vai          VAI FULL STOP
U+0A60F ‭ ꘏  GC=Po SC=Vai          VAI QUESTION MARK
U+0A6F3 ‭ ꛳  GC=Po SC=Bamum        BAMUM FULL STOP
U+0A6F7 ‭ ꛷  GC=Po SC=Bamum        BAMUM QUESTION MARK
U+0A876 ‭ ꡶  GC=Po SC=Phags_Pa     PHAGS-PA MARK SHAD
U+0A877 ‭ ꡷  GC=Po SC=Phags_Pa     PHAGS-PA MARK DOUBLE SHAD
U+0A8CE ‭ ꣎  GC=Po SC=Saurashtra   SAURASHTRA DANDA
U+0A8CF ‭ ꣏  GC=Po SC=Saurashtra   SAURASHTRA DOUBLE DANDA
U+0A92F ‭ ꤯  GC=Po SC=Kayah_Li     KAYAH LI SIGN SHYA
U+0A9C8 ‭ ꧈  GC=Po SC=Javanese     JAVANESE PADA LINGSA
U+0A9C9 ‭ ꧉  GC=Po SC=Javanese     JAVANESE PADA LUNGSI
U+0AA5D ‭ ꩝  GC=Po SC=Cham         CHAM PUNCTUATION DANDA
U+0AA5E ‭ ꩞  GC=Po SC=Cham         CHAM PUNCTUATION DOUBLE DANDA
U+0AA5F ‭ ꩟  GC=Po SC=Cham         CHAM PUNCTUATION TRIPLE DANDA
U+0AAF0 ‭ ꫰  GC=Po SC=Meetei_Mayek MEETEI MAYEK CHEIKHAN
U+0AAF1 ‭ ꫱  GC=Po SC=Meetei_Mayek MEETEI MAYEK AHANG KHUDAM
U+0ABEB ‭ ꯫  GC=Po SC=Meetei_Mayek MEETEI MAYEK CHEIKHEI
U+0FE52 ‭ ﹒ GC=Po SC=Common       SMALL FULL STOP
U+0FE56 ‭ ﹖ GC=Po SC=Common       SMALL QUESTION MARK
U+0FE57 ‭ ﹗ GC=Po SC=Common       SMALL EXCLAMATION MARK
U+0FF01 ‭ ！ GC=Po SC=Common       FULLWIDTH EXCLAMATION MARK
U+0FF0E ‭ ． GC=Po SC=Common       FULLWIDTH FULL STOP
U+0FF1F ‭ ？ GC=Po SC=Common       FULLWIDTH QUESTION MARK
U+0FF61 ‭ ｡  GC=Po SC=Common       HALFWIDTH IDEOGRAPHIC FULL STOP
U+11047 ‭ 𑁇  GC=Po SC=Brahmi       BRAHMI DANDA
U+11048 ‭ 𑁈  GC=Po SC=Brahmi       BRAHMI DOUBLE DANDA
U+110BE ‭ 𑂾  GC=Po SC=Kaithi       KAITHI SECTION MARK
U+110BF ‭ 𑂿  GC=Po SC=Kaithi       KAITHI DOUBLE SECTION MARK
U+110C0 ‭ 𑃀  GC=Po SC=Kaithi       KAITHI DANDA
U+110C1 ‭ 𑃁  GC=Po SC=Kaithi       KAITHI DOUBLE DANDA
U+11141 ‭ 𑅁  GC=Po SC=Chakma       CHAKMA DANDA
U+11142 ‭ 𑅂  GC=Po SC=Chakma       CHAKMA DOUBLE DANDA
U+11143 ‭ 𑅃  GC=Po SC=Chakma       CHAKMA QUESTION MARK
U+111C5 ‭ 𑇅  GC=Po SC=Sharada      SHARADA DANDA
U+111C6 ‭ 𑇆  GC=Po SC=Sharada      SHARADA DOUBLE DANDA

To go the other way around — that is, too find properties given a code point instead of finding code points given a set of properties — use the companion uniprops script, which pulls out all the properties of a given code point:

$ uniprops -a . \? \!
U+002E ‹.› \N{FULL STOP}
    \pP \p{Po}
    All Any ASCII Assigned Basic_Latin Case_Ignorable CI Common Zyyy Po P Gr_Base Grapheme_Base Graph GrBase Other_Punctuation Punct Pat_Syn
       Pattern_Syntax PatSyn POSIX_Graph POSIX_Print POSIX_Punct Print Punctuation STerm Term Terminal_Punctuation X_POSIX_Graph X_POSIX_Print
       X_POSIX_Punct
    Age=1.1 Block=Basic_Latin Bidi_Class=Common_Separator BC=CS Bidi_Class=CS Block=ASCII BLK=ASCII Canonical_Combining_Class=0
       Canonical_Combining_Class=Not_Reordered CCC=NR Canonical_Combining_Class=NR Script=Common Decomposition_Type=None DT=None East_Asian_Width=Na
       East_Asian_Width=Narrow EA=Na Grapheme_Cluster_Break=Other GCB=XX Grapheme_Cluster_Break=XX Hangul_Syllable_Type=NA
       Hangul_Syllable_Type=Not_Applicable HST=NA Joining_Group=No_Joining_Group JG=NoJoiningGroup Joining_Type=Non_Joining JT=U Joining_Type=U
       Line_Break=Infix_Numeric LB=IS Line_Break=IS Numeric_Type=None NT=None Numeric_Value=NaN NV=NaN Present_In=1.1 IN=1.1 Present_In=2.0 IN=2.0
       Present_In=2.1 IN=2.1 Present_In=3.0 IN=3.0 Present_In=3.1 IN=3.1 Present_In=3.2 IN=3.2 Present_In=4.0 IN=4.0 Present_In=4.1 IN=4.1 Present_In=5.0
       IN=5.0 Present_In=5.1 IN=5.1 Present_In=5.2 IN=5.2 Present_In=6.0 IN=6.0 SC=Zyyy Script=Zyyy Sentence_Break=AT Sentence_Break=ATerm SB=AT
       Word_Break=MB Word_Break=MidNumLet WB=MB _Case_Ignorable _X_Begin
U+003F ‹?› \N{QUESTION MARK}
    \pP \p{Po}
    All Any ASCII Assigned Basic_Latin Common Zyyy Po P Gr_Base Grapheme_Base Graph GrBase Other_Punctuation Punct Pat_Syn Pattern_Syntax PatSyn
       POSIX_Graph POSIX_Print POSIX_Punct Print Punctuation STerm Term Terminal_Punctuation X_POSIX_Graph X_POSIX_Print X_POSIX_Punct
    Age=1.1 Block=Basic_Latin Bidi_Class=ON Bidi_Class=Other_Neutral BC=ON Block=ASCII BLK=ASCII Canonical_Combining_Class=0
       Canonical_Combining_Class=Not_Reordered CCC=NR Canonical_Combining_Class=NR Script=Common Decomposition_Type=None DT=None East_Asian_Width=Na
       East_Asian_Width=Narrow EA=Na Grapheme_Cluster_Break=Other GCB=XX Grapheme_Cluster_Break=XX Hangul_Syllable_Type=NA
       Hangul_Syllable_Type=Not_Applicable HST=NA Joining_Group=No_Joining_Group JG=NoJoiningGroup Joining_Type=Non_Joining JT=U Joining_Type=U
       Line_Break=EX Line_Break=Exclamation LB=EX Numeric_Type=None NT=None Numeric_Value=NaN NV=NaN Present_In=1.1 IN=1.1 Present_In=2.0 IN=2.0
       Present_In=2.1 IN=2.1 Present_In=3.0 IN=3.0 Present_In=3.1 IN=3.1 Present_In=3.2 IN=3.2 Present_In=4.0 IN=4.0 Present_In=4.1 IN=4.1 Present_In=5.0
       IN=5.0 Present_In=5.1 IN=5.1 Present_In=5.2 IN=5.2 Present_In=6.0 IN=6.0 SC=Zyyy Script=Zyyy Sentence_Break=ST Sentence_Break=STerm SB=ST
       Word_Break=Other WB=XX Word_Break=XX _X_Begin
U+0021 ‹!› \N{EXCLAMATION MARK}
    \pP \p{Po}
    All Any ASCII Assigned Basic_Latin Common Zyyy Po P Gr_Base Grapheme_Base Graph GrBase Other_Punctuation Punct Pat_Syn Pattern_Syntax PatSyn
       POSIX_Graph POSIX_Print POSIX_Punct Print Punctuation STerm Term Terminal_Punctuation X_POSIX_Graph X_POSIX_Print X_POSIX_Punct
    Age=1.1 Block=Basic_Latin Bidi_Class=ON Bidi_Class=Other_Neutral BC=ON Block=ASCII BLK=ASCII Canonical_Combining_Class=0
       Canonical_Combining_Class=Not_Reordered CCC=NR Canonical_Combining_Class=NR Script=Common Decomposition_Type=None DT=None East_Asian_Width=Na
       East_Asian_Width=Narrow EA=Na Grapheme_Cluster_Break=Other GCB=XX Grapheme_Cluster_Break=XX Hangul_Syllable_Type=NA
       Hangul_Syllable_Type=Not_Applicable HST=NA Joining_Group=No_Joining_Group JG=NoJoiningGroup Joining_Type=Non_Joining JT=U Joining_Type=U
       Line_Break=EX Line_Break=Exclamation LB=EX Numeric_Type=None NT=None Numeric_Value=NaN NV=NaN Present_In=1.1 IN=1.1 Present_In=2.0 IN=2.0
       Present_In=2.1 IN=2.1 Present_In=3.0 IN=3.0 Present_In=3.1 IN=3.1 Present_In=3.2 IN=3.2 Present_In=4.0 IN=4.0 Present_In=4.1 IN=4.1 Present_In=5.0
       IN=5.0 Present_In=5.1 IN=5.1 Present_In=5.2 IN=5.2 Present_In=6.0 IN=6.0 SC=Zyyy Script=Zyyy Sentence_Break=ST Sentence_Break=STerm SB=ST
       Word_Break=Other WB=XX Word_Break=XX _X_Begin

I suspect you should check more into the whole sentence-break properties.

There’s also a 3^rd script in the suite, uninames, which does things like this:

$ uninames sentence
 ;  037E        GREEK QUESTION MARK
        = erotimatiko
        * sentence-final punctuation
        * 003B is the preferred character
        x (question mark - 003F)
        : 003B semicolon
 ⁚  205A        TWO DOT PUNCTUATION
        * historically used to indicate the end of a sentence or change of speaker
        * extends from baseline to cap height
        x (presentation form for vertical two dot leader - FE30)
        x (greek acrophonic epidaurean two - 1015B)
 𑂾  110BE       KAITHI SECTION MARK
        * marks end of sentence

I find those three programs indispensable for exploring Unicode properties. You can install them all using the CPAN Unicode::Tussle suite, or inspect them indivually here.

Answer 2

I haven’t encountered any compilations of such information, and I would expect it to be a major effort to collect it. For some widely used languages, you could get the information from The Chicago Manual of Style. There is some information about punctuation marks commonly used in different languages at http://unicode.org/repos/cldr-tmp/trunk/diff/by_type/misc.exemplarCharacters-other.html but is covers just a small set of languages and does not distinguish sentence-terminating characters.

Using just characters won’t be enough, since e.g. in English, the full stop “.” occurs in many contexts where it does not terminate a sentence, as in “e.g.” or in “1.5”.