Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Difference between StandardTokenizerFactory and KeywordTokenizerFactory in Solr?

I am new to Solr.I want to know when to use StandardTokenizerFactory and KeywordTokenizerFactory?

I read the docs on Apache Wiki, but I am not getting it.

Can anybody explain the difference between StandardTokenizerFactory and KeywordTokenizerFactory?

like image 772
ravidev Avatar asked Oct 04 '11 09:10

ravidev


People also ask

What is Solr tokenization?

A tokenizer in Solr breaks out text streams into tokens and passes them on to the filter for additional text analytics. About - Understand the purpose of a tokenizer. Syntax - See how tokenizers are coded in schema. xml or managed-schema.

What is standard Tokenizer?

The standard tokenizer provides grammar based tokenization (based on the Unicode Text Segmentation algorithm, as specified in Unicode Standard Annex #29) and works well for most languages.


1 Answers

StandardTokenizerFactory :-
It tokenizes on whitespace, as well as strips characters

Documentation :-

Splits words at punctuation characters, removing punctuations. However, a dot that's not followed by whitespace is considered part of a token. Splits words at hyphens, unless there's a number in the token. In that case, the whole token is interpreted as a product number and is not split. Recognizes email addresses and Internet hostnames as one token.

Would use this for fields where you want to search on the field data.

e.g. -

http://example.com/I-am+example?Text=-Hello

would generate 7 tokens (separated by comma) -

http,example.com,I,am,example,Text,Hello

KeywordTokenizerFactory :-

Keyword Tokenizer does not split the input at all.
No processing in performed on the string, and the whole string is treated as a single entity.
This doesn't actually do any tokenization. It returns the original text as one term.

Mainly used for sorting or faceting requirements, where you want to match the exact facet when filtering on multiple words and sorting as sorting does not work on tokenized fields.

e.g.

http://example.com/I-am+example?Text=-Hello

would generate a single token -

http://example.com/I-am+example?Text=-Hello
like image 73
Jayendra Avatar answered Oct 22 '22 08:10

Jayendra