How to custom FTS tokenizer in Android

Question

because sqlite fts default tokenize is 'simple' ,i can change it to 'porter'. but i must support Non-US languages. like Chinese, it can't just split by space or dot.So i want to custom tokenizer. Can somebody give me some advice?

1. How to regist Tokenizer in Sqlite.

   SELECT fts3_tokenizer(<tokenizer-name>, <sqlite3_tokenizer_module ptr>);
   

   cause sqlite is system lib in Android , i don't know if i can regist.

2. Is there a common tokenizer for all language? for my app must i18n. just support Chinese is not useful.

Answer 1

Sure, you need a custom SQLite installation. The SQLite Android bindings, provides an easy way to do just that. This allows an application to use a custom build or version of SQLite, regardless of the Android version to which it is deployed, while continuing to use the standard Android interface. https://sqlite.org/android/doc/trunk/www/index.wiki