Optimizing/Customizing Sharepoint Search Crawling

Question

With SharePoint Server 2007, there is also a Search Feature and a Crawler. However, the Crawler is somewhat limited in that it only supports Basic Auth when crawling external sites and that there is no way to tell it to ignore no-index,no-follow attributes.

Now, there is a site i'd like to index, unfortunately this site uses it's own Authentication System, and it uses no-index,no-follow on the pages. As I control that site, i can remove the Attributes, but it's a PITA to do so. Also, it does not solve the Authentication issue.

So I just wonder if it's possible to extend Sharepoint's Crawler somehow?

Answer 1

The limitation of MOSS crawling sites with different forms authentication should have been addressed in MOSS SP1. :

http://www.microsoft.com/downloads/details.aspx?FamilyID=ad59175c-ad6a-4027-8c2f-db25322f791b&displaylang=en

Here's a link to a post which describes how to get the hotfix for pre-SP1 MOSS to enable the crawling of sites with forms authentication:

http://blogs.microsoft.co.il/blogs/adir_ron/archive/2007/10/11/moss-search-for-sso-form-based-authentication-sites.aspx

Hope that helps!