Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Solr: strip punctuation before index

I am having a problem with striping punctuation from the solr index When the punctuation sign follow right after a word then this word is not indexed properly.

For example: if we index "hello, John", the asset won't be found by keyword "hello" while there will be no issue if we remove comma after word "hello".

Is there any FilterFactory that suppose to strip punctuation? Any ideas?

Thanks, Bogdan.

like image 869
Bogdan Gusiev Avatar asked Jun 30 '10 13:06

Bogdan Gusiev


2 Answers

You can use the solr.PatternReplaceFilterFactory to strip beginning and trailing punctuation with this:

<filter class="solr.PatternReplaceFilterFactory"
    pattern="^\p{Punct}*(.*?)\p{Punct}*$"
    replacement="$1"/>

And if you wanted to strip all punctuation at the beginning and end, except (for example) the dollar-sign in front of a word, you could use this:

<filter class="solr.PatternReplaceFilterFactory"
    pattern="^[\p{Punct}&&[^$]]*(.*?)\p{Punct}*$"
    replacement="$1"/>
like image 97
claytron Avatar answered Oct 11 '22 15:10

claytron


This is done with the WordDelimiterFilterFactory. Set generateWordParts=1.

There is also the PatternTokenizerFactory that could be used, but I have never tried it.

like image 42
Pascal Dimassimo Avatar answered Oct 11 '22 15:10

Pascal Dimassimo