Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to use regex for utf8 in ruby

In RoR,how to validate a Chinese or a Japanese word for a posting form with utf8 code.

In GBK code, it uses [\u4e00-\u9fa5]+ to validate Chinese words. In Php, it uses /^[\x{4e00}-\x{9fa5}]+$/u for utf-8 pages.

like image 242
nowa Avatar asked Nov 02 '08 12:11

nowa


4 Answers

Ruby 1.8 has poor support for UTF-8 strings. You need to write the bytes individually in the regular expression, rather then the full code:

>> "acentuação".scan(/\xC3\xA7/)
=> ["ç"]    

To match the range you specified the expression will become a bit complicated:

/([\x4E-\x9E][\x00-\xFF])|(\x9F[\x00-\xA5])/  # (untested)

That will be improved in Ruby 1.9, though.

Edit: As noted in the comments, the unicode characters \u4E00-\u9FA5 only map to the expression above in the UTF16-BE encoding. The UTF8 encoding is likely different. So you need to analyze the mapping carefully and see if you can come up with a byte-matching expression for Ruby 1.8.

like image 162
Rômulo Ceccon Avatar answered Oct 24 '22 00:10

Rômulo Ceccon


This is what i have done:

%r{^[#{"\344\270\200"}-#{"\351\277\277"}]+$}

This is basically a regular expression with the octal values that represent the range between U+4E00 and U+9FFF, the most common Chinese and Japanese characters.

like image 42
Jose Barrera Avatar answered Oct 23 '22 23:10

Jose Barrera


The Oniguruma regexp engine has proper support for Unicode. Ruby 1.9 uses Oniguruma by default. Ruby 1.8 can be recompiled to use it.

With Oniguruma you can use the exact same regex as in PHP, including the /u modifier to force Ruby to treat the string as UTF-8.

like image 30
Jan Goyvaerts Avatar answered Oct 23 '22 23:10

Jan Goyvaerts


activeSupport has a UTF-8 handler

http://api.rubyonrails.org/classes/ActiveSupport/Multibyte/Handlers/UTF8Handler.html


otherwise, look in ruby 1.9, encoding method for Regexp objects

like image 1
Gene T Avatar answered Oct 24 '22 00:10

Gene T