Menu
I am utilizing solr ExtractingRequestHandler to extract and index HTML content. My issue comes to the extracted links section that it produces. The extracted content returned has "rect" inserted where they do not exist in the HTML source.
I have my solrconfig cell configuration as follows:
<requestHandler name="/upate/extract"
startup="lazy"
class="solr.extraction.ExtractingRequestHandler" >
<lst name="defaults">
<str name="lowernames">true</str>
<!-- capture link hrefs but ignore div attributes -->
<str name="captureAttr">true</str>
<str name="fmap.div">ignored_</str>
</lst>
And my solr schema.xml with the following etnries:
<field name="content_type" type="string" indexed="true" stored="true" multiValued="true"/>
<field name="links" type="string" indexed="true" stored="true" multiValued="true"/>
<field name="meta" type="string" indexed="true" stored="true" multiValued="true"/>
<field name="content_encoding" type="string" indexed="false" stored="true" multiValued="false"/>
<field name="content" type="text_general" indexed="false" stored="true" multiValued="true"/>
I post the following HTML to sorl cell:
<!DOCTYPE html>
<html>
<body>
<h1>Heading1</h1><a href="http://www.google.com">Link to Google</a><a href=
"http://www.google.com">Link to Google2</a><a href="http://www.google.com">Link to
Google3</a><a href="http://www.google.com">Link to Google</a>
<p>Paragraph1</p>
</body>
</html>
Solr has the following indexed:
{
"meta": [
"Content-Encoding",
"ISO-8859-1",
"ignored_hbaseindexer_mime_type",
"text/html",
"Content-Type",
"text/html; charset=ISO-8859-1"
],
"links": [
"rect",
"http://www.google.com",
"rect",
"http://www.google.com",
"rect",
"http://www.google.com",
"rect",
"http://www.google.com"
],
"content_encoding": "ISO-8859-1",
"content_type": [
"text/html; charset=ISO-8859-1"
],
"content": [
" Heading1 Link to Google Link to Google2 Link to Google3 Link to Google Paragraph1 "
],
"id": "row69",
"_version_": 1461665607851180000
}
Notice the "rect" between every link. Why is solr cell or tika inserting these? I am not defining a tika config file to use. Do i need to configure tika?
342
Although an old Question, I also encountered this issue while indexing HTML documents via Solr 8.7.0.
<requestHandler name="/update/extract"
class="org.apache.solr.handler.extraction.ExtractingRequestHandler">
....
HTML:
<p>My website is <a href="https://buriedtruth.com/">BuriedTruth.com</a>.</p>
Result:
My website is rect https://buriedtruth.com/ BuriedTruth.com .
[ I am posting/indexing on the Linux command-line: solr restart; sleep 1; post -c gettingstarted /mnt/Vancouver/programming/datasci/solr/test/solr_test9.html; ]
I grepped (ripgrep: rg --color=always -w -e 'rect' . |less) the Solr code for that word, but found nothing, so the source of rect http... in indexed URLs eludes me.
My solution was to add a regex processor to mysolrconfig.xml:
<updateRequestProcessorChain name="add-unknown-fields-to-the-schema" default="${update.autoCreateFields:true}"
processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields">
<processor class="solr.LogUpdateProcessorFactory"/>
<processor class="solr.DistributedUpdateProcessorFactory"/>
<!-- ======================================== -->
<!-- https://lucene.apache.org/solr/7_4_0/solr-core/org/apache/solr/update/processor/RegexReplaceProcessorFactory.html -->
<!-- Solr bug? URLs parse as "rect https..." Managed-schema (Admin UI): defined p as text_general -->
<!-- but did not parse. Looking at content | title: text_general copied to string, so added -->
<!-- copyfield of p (text_general) as p_str ... regex below now works! -->
<processor class="solr.RegexReplaceProcessorFactory">
<str name="fieldName">content</str>
<str name="fieldName">title</str>
<str name="fieldName">p</str>
<!-- Case-sensitive (and only one pattern:replacement allowed, so use as many copies): -->
<!-- of this processor as needed: -->
<str name="pattern">rect http</str>
<str name="replacement">http</str>
<bool name="literalReplacement">true</bool>
</processor>
<!-- ======================================== -->
<!-- This needs to be last (may need to clear documents and reindex to see changes, e.g. Solr Admin UI): -->
<processor class="solr.RunUpdateProcessorFactory"/>
</updateRequestProcessorChain>
As alluded in my comments in that processor, I am extracting <p />-formatted HTML content to a p field (field: p | type: text_general).
That content did not parse with the RegexReplaceProcessorFactory processor.
In the Solr Admin UI I noted that title and content were copied as strings (e.g.: field: content | type: text_general | copied to: content_str), so I made copy field (p >> p_str) that resolved the regex issue.
For completeness, here are the relevant parts of my solrconfig.xml related to HTML document indexing,
<lib dir="${solr.install.dir:../../..}/contrib/extraction/lib" regex=".*\.jar" />
<lib dir="${solr.install.dir:../../..}/dist/" regex="solr-cell-\d.*\.jar" />
<!-- https://lucene.472066.n3.nabble.com/Prons-an-Cons-of-Startup-Lazy-a-Handler-td4059111.html -->
<!-- startup="lazy" -->
<requestHandler name="/update/extract"
class="org.apache.solr.handler.extraction.ExtractingRequestHandler">
<lst name="defaults">
<str name="lowernames">true</str>
<str name="uprefix">ignored_</str>
<str name="capture">div</str>
<str name="fmap.div">div</str>
<str name="capture">p</str>
<str name="fmap.p">p</str>
</lst>
</requestHandler>
... noting again that I added fields to the managed-schema via the Solr Admin UI.
Result:
My website is https://buriedtruth.com/ BuriedTruth.com .
<field name="p" type="text_general" uninvertible="true" indexed="true" stored="true"/>
<copyField source="p" dest="p_str"/>
See also:
re: <requestHandler name="/update/extract"...:
Solr 8.6.3 could not index html file
https://lucene.apache.org/solr/guide/8_6/uploading-data-with-solr-cell-using-apache-tika.html#configuring-the-extractingrequesthandler-in-solrconfig-xml
My answer here (which deals with pecularities associated with the updateRequestProcessorChain />, above) when switching from Solr's managed-schema to the classic schema.xml
102
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
