Removing Solr duplicate values into multivalued field

Question

My Solr index contains a multivalued field with duplicate values. How can I remove the duplicates ?

Is it possible to overwrite duplicate values into the multivalued field when indexing ?

Thanks

Answer 1

I was struggling to accomplish the same. This worked for me. Add the below processor to your solrconfig.xml

<updateRequestProcessorChain name="deduplicateMultiValued" default="true">
        <processor class="org.apache.solr.update.processor.UniqFieldsUpdateProcessorFactory">
            <lst name="fields">
                <str>multivaluedFieldXYZ</str>
            </lst>
        </processor>
        <processor class="solr.RunUpdateProcessorFactory" />
 </updateRequestProcessorChain>

Answer 2

Really late to the party, but the top answer did not work for me in Solr 6.0 for attempting to add a duplicate entry on a multivalued field. it was missing a processor right before UniqFieldsUpdateProcessorFactory. So adding something like this to my solrconfig.xml worked:

<updateRequestProcessorChain name="uniq-fields">
<processor class="org.apache.solr.update.processor.DistributedUpdateProcessorFactory"/>
<processor class="org.apache.solr.update.processor.UniqFieldsUpdateProcessorFactory">
  <str name="fieldName">YourFieldA</str>
  <str name="fieldName">yourFieldB</str>
</processor>
<processor class="solr.RunUpdateProcessorFactory" />

Where YourFieldA and YourFieldB are defined fields in your schema.xml. Note that you must also add this to the proper requestHandler ie:

  <requestHandler name="/update" class="solr.UpdateRequestHandler" >
<lst name="defaults">
  <str name="update.chain">uniq-fields</str>
</lst>

This will not only prevent duplicates from being added, but also remove all duplicates from your index upon update for the specified fields.

Answer 3

You would need to handle it on the Client side to remove the duplicate values.

You can customize the implementation like RemoveDuplicatesTokenFilterFactory (works for same text at same position) to filter out the tokens. Write an extension basically. OR

Also, If using the multivalued field for just faceting ,the value in faceted field is counted just once. So even if you add multiple same values, that would be reflected as a single value in the facet count entry. Have tested this. you too can confirm.

However, the duplicate values would cause the change in the lengthNorm and hence can have an effect on the scoring.