Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How do you customise text segmentation to not break between a digraph?

Works:

#!/usr/bin/env python3
from uniseg.graphemecluster import grapheme_clusters
def albanian_digraph_dh(s, breakables):
    for i, breakable in enumerate(breakables):
        if s.endswith('d', 0, i) and s.startswith('h', i):
            yield 0
        else:
            yield breakable

print(list(grapheme_clusters('dhelpëror', albanian_digraph_dh)))
#['dh', 'e', 'l', 'p', 'ë', 'r', 'o', 'r']

Needs improvement/customisation:

perl -C -Mutf8 -mUnicode::GCString -E'
    say join " ", Unicode::GCString
        ->new("dhelpëror")->as_array
'
#d h e l p ë r o r

perl6 -e'"dhelpëror".comb.say'
#(d h e l p ë r o r)

NB: writing your own segmentation which is almost guaranteed to not implement UAX #29 correctly counts as side-stepping the problem.

like image 673
daxim Avatar asked Aug 23 '19 08:08

daxim


1 Answers

D:\>perl6 -e "'dhelpëror'.comb(/dh|./).say"
(dh e l p ë r o r)

You can do the same in old Perl.

print join ' ', 'dhelpëror' =~ /dh|./g
like image 116
Holli Avatar answered Oct 05 '22 06:10

Holli