Crawl PDF documents using nutch

Question

I have to crawl PDF documents too from given URL... suggest any tool/API to crawl PDF docs also... now I am using nutch to crawl but I couldn't crawl PDF from given URL...should I use any plugin to crawl PDF in nutch?

seed.txt --> http://nutch.apache.org regex-urlfilter.txt--->+^http://([a-z0-9]*.)*nutch.apache.org/

Thanks in Advance

Answer 1

1. Edit regex-urlfilter.txt and remove any occurence of "pdf"
2. Edit suffix-urlfilter.txt and remove any occurence of "pdf"
3. Edit nutch-site.xml, add "parse-tika" and "parse-html" in the plugin.includes section. this should look like this

this answer came from here . I have tested it when working on Nutch

<property>


<name>plugin.includes</name>
  <value>protocol-http|urlfilter-regex|parse-(html|tika|text)|index-(basic|anchor)|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
  <description>
    ...
  </description>
</property>