Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is There a Way to Match Any Unicode Alphabetic Character?

I have some documents that went through OCR conversion from PDF into HTML. Because of that, they wound up having lots of random unicode punctuation where the converter messed up (i.e. elipses, etc...). They also correctly have a bunch of Non-English, but still Alphabetic characters, like é, and Russian characters, etc...

Is there any way to make a Regex that will match any unicode alphabetic character (from alphabets of any language)? Or one that will only match non-alphabetic characters? Either one would be really helpful and awesome. I'm using Perl, if that changes anything. Thanks!

like image 777
Eli Avatar asked May 14 '11 23:05

Eli


People also ask

What is a Unicode alphabetic character?

The alphabetic characters are those UNICODE characters which are defined as letters by the UNICODE standard, e.g., the ASCII characters. ABCDEFGHIJKLMNOPQRSTUVWXYZ. abcdefghijklmnopqrstuvwxyz.

Does regex work with Unicode?

RegexBuddy's regex engine is fully Unicode-based starting with version 2.0. 0.

Which regex is applicable for alphabets?

[A-Za-z] will match all the alphabets (both lowercase and uppercase).

How do you match letters in regex?

Using character sets For example, the regular expression "[ A-Za-z] " specifies to match any single uppercase or lowercase letter. In the character set, a hyphen indicates a range of characters, for example [A-Z] will match any one capital letter.


2 Answers

Check out Unicode character properties: http://www.regular-expressions.info/unicode.html#prop. I think what you are looking for is probably

\p{L}

which will match any letters or ideographs. You may also want to include letters with marks on them, so you could do

\p{L}\p{M}*

In any case, all the different types of character properties are detailed in the first link.

Edit: You may also want to look at this Stack Overflow answer discussing whether \w matches unicode characters. They suggest that you could also use \p{Word} or \p{Alnum}: Does \w match all alphanumeric characters defined in the Unicode standard?

like image 99
mpdaugherty Avatar answered Oct 25 '22 11:10

mpdaugherty


Depending on which language you're using, the regular expression engine may or may not be Unicode aware. If it is, it may or may not know the \p{} property tokens. If it does, your answer is in Unicode Characters and Properties in Jan Goyvaerts' regex tutorial.

You can use \p{Latin}, if supported, to detect everything that is (or isn't, of course) from a language that uses any of the Unicode Latin blocks.

like image 43
Mike 'Pomax' Kamermans Avatar answered Oct 25 '22 11:10

Mike 'Pomax' Kamermans