Solr search for hashtag or mentions

Question

We are using solr version 3.5 to search though Tweets, I am using WordDelimiterFactory with the following setting, to be able to search for @username or #hashtags:

<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="0" splitOnNumerics="0" preserveOriginal="1" handleAsChar="@#"/>

I saw the following patch but this doesn’t seem to be working as I expected, am I missing something?

https://issues.apache.org/jira/browse/SOLR-2059

But searching for @username is also returning results for just username or #hashtag is just returning result for hastag. How can I achieve this?

Entire Field Type:

<fieldType name="textnostem" class="solr.TextField" positionIncrementGap="100" autoGeneratePhraseQueries="true">
  <analyzer type="index">
    <charFilter class="solr.HTMLStripCharFilterFactory"/>
    <tokenizer class="solr.WhitespaceTokenizerFactory"/>
    <filter class="solr.StopFilterFactory"
            ignoreCase="true"
            words="stopwords.txt"
            enablePositionIncrements="true"
            />
    <filter class="solr.WordDelimiterFilterFactory" 
            generateWordParts="1" 
            generateNumberParts="1" 
            catenateWords="1" 
            catenateNumbers="1" 
            catenateAll="0" 
            splitOnCaseChange="0" 
            splitOnNumerics="0"
            preserveOriginal="1"
            />
    <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
  <analyzer type="query">
    <tokenizer class="solr.WhitespaceTokenizerFactory"/>
    <filter class="solr.StopFilterFactory"
            ignoreCase="true"
            words="stopwords.txt"
            enablePositionIncrements="true"
            />
    <filter class="solr.WordDelimiterFilterFactory" 
            generateWordParts="1" 
            generateNumberParts="1" 
            catenateWords="1" 
            catenateNumbers="1" 
            catenateAll="0" 
            splitOnCaseChange="0" 
            splitOnNumerics="0"
            preserveOriginal="1"/>
    <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>

<fieldType name="textnostem" class="solr.TextField" positionIncrementGap="100" autoGeneratePhraseQueries="true">
  <analyzer type="index">
    <charFilter class="solr.HTMLStripCharFilterFactory"/>
    <tokenizer class="solr.WhitespaceTokenizerFactory"/>
    <filter class="solr.StopFilterFactory"
            ignoreCase="true"
            words="stopwords.txt"
            enablePositionIncrements="true"
            />
    <filter class="solr.WordDelimiterFilterFactory" 
            generateWordParts="1" 
            generateNumberParts="1" 
            catenateWords="1" 
            catenateNumbers="1" 
            catenateAll="0" 
            splitOnCaseChange="0" 
            splitOnNumerics="0"
            preserveOriginal="1"
            />
    <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
  <analyzer type="query">
    <tokenizer class="solr.WhitespaceTokenizerFactory"/>
    <filter class="solr.StopFilterFactory"
            ignoreCase="true"
            words="stopwords.txt"
            enablePositionIncrements="true"
            />
    <filter class="solr.WordDelimiterFilterFactory" 
            generateWordParts="1" 
            generateNumberParts="1" 
            catenateWords="1" 
            catenateNumbers="1" 
            catenateAll="0" 
            splitOnCaseChange="0" 
            splitOnNumerics="0"
            preserveOriginal="1"/>      
    <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
</fieldType>

<fieldType name="textnostem" class="solr.TextField" positionIncrementGap="100" autoGeneratePhraseQueries="true">
  <analyzer type="index">
    <charFilter class="solr.HTMLStripCharFilterFactory"/>
    <tokenizer class="solr.WhitespaceTokenizerFactory"/>
    <filter class="solr.StopFilterFactory"
            ignoreCase="true"
            words="stopwords.txt"
            enablePositionIncrements="true"
            />
    <filter class="solr.WordDelimiterFilterFactory" 
            generateWordParts="1" 
            generateNumberParts="1" 
            catenateWords="1" 
            catenateNumbers="1" 
            catenateAll="0" 
            splitOnCaseChange="0" 
            splitOnNumerics="0"
            preserveOriginal="1"
            handleAsChar="@#"
            />
    <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
  <analyzer type="query">
    <tokenizer class="solr.WhitespaceTokenizerFactory"/>
    <filter class="solr.StopFilterFactory"
            ignoreCase="true"
            words="stopwords.txt"
            enablePositionIncrements="true"
            />
    <filter class="solr.WordDelimiterFilterFactory" 
            generateWordParts="1" 
            generateNumberParts="1" 
            catenateWords="1" 
            catenateNumbers="1" 
            catenateAll="0" 
            splitOnCaseChange="0" 
            splitOnNumerics="0"
            preserveOriginal="1"
            handleAsChar="@#"
            />      
    <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
</fieldType>

Answer 1

OK, so reading through the SOLR-2059 patch that you mentioned, it looks like they have replaced the handleAsChar attribute on the WordDelimiterFactory with the types attribute. Here is the specification for that attribute from the Analyzers, Tokenizers and Token Filters Solr Wiki page:

types="wdfftypes.txt" allows customized tokenization for this filter. The file should exist in the solr/conf directory, and entries are of the form (without quotes) "% => ALPHA" or "\u002C => DIGIT". Allowable types are: LOWER, UPPER, ALPHA, DIGIT, ALPHANUM, SUBWORD_DELIM.

So then if we take this documentation, plus the example of the file from SOLR-2059, I would recommend the following:

<filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="1" catenateNumbers="1" catenateAll="0" splitOnCaseChange="0" splitOnNumerics="0" preserveOriginal="1" types="twittertypes.txt"/>

Then define the twittertypes.txt file as follows and place it in the same folder as your schema.xml file in your Solr instance (probably the conf folder).

 # A customized type mapping for WordDelimiterFilterFactory
 # the allowable types are: LOWER, UPPER, ALPHA, DIGIT, ALPHANUM, SUBWORD_DELIM
 #    
 # the default for any character without a mapping is always computed from 
 # Unicode character properties

 # Map the $, %, '.', and ',' characters to DIGIT 
 # This might be useful for financial data.
 @ => ALPHA
 \u0023 => ALPHA

Notice that you need to use the Unicode character (UTF-8) for the hash symbol, since it is treated as comment in the text file.

According to all of the documentation, this should fix your issue and treat the # and @ symbols as alpha characters, which will provide the behavior you are looking for.