Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Difference between WhitespaceTokenizerFactory and StandardTokenizerFactory

Tags:

solr

tokenize

I am new to Solr. By reading Solr's wiki, I don't understand the differences between WhitespaceTokenizerFactory and StandardTokenizerFactory. What's their real difference?

like image 637
trillions Avatar asked Jun 25 '12 03:06

trillions


1 Answers

They differ in how they split the analyzed text into tokens.

The StandardTokenizer does this based on the following (taken from lucene javadoc):

  • Splits words at punctuation characters, removing punctuation. However, a dot that's not followed by whitespace is considered part of a token.
  • Splits words at hyphens, unless there's a number in the token, in which case the whole token is interpreted as a product number and is not split.
  • Recognizes email addresses and internet hostnames as one token.

The WhitespaceTokenizer does this based on whitespace characters:

A WhitespaceTokenizer is a tokenizer that divides text at whitespace. Adjacent sequences of non-Whitespace characters form tokens.

You should pick the tokenizer that best fits your application. In any case you have to use the same analyzer/tokenizers for indexing and searching!

like image 95
csupnig Avatar answered Oct 24 '22 00:10

csupnig