Search fails after upgrade to TFS 2018 Update 2

Question

After performing an upgrade of a TFS server to 2018 Update 2 the search and indexing seems to be broken on one of our environments.

Any search gives "We encountered an unexpected error when processing your request" and I have work through all the troubleshooting docs to clean and reindex all collections. Also completely reinstalled the search package to the separate server we run for search and indexing to make sure we got the correct version running.

In the event logs on the TFS App Server a large number of these exceptions are logged:

Events (81277) completed with status FailedAndRetry. Event 81277 completed with message 'BeginBulkIndex-PushEventNotification: The operation did not complete successfully because of exception Microsoft.VisualStudio.Services.Search.Common.FeederException: Lots of files rejected by Elasticsearch, failing this job. Failure Reason : Microsoft.VisualStudio.Services.Search.Common.SearchPlatformException: ES Exception: [HTTP Status Code: [200] BULK_API_ERROR: [ index returned 404 _index: codesearchshared_1_0 _type: SourceNoDedupeFileContractV3 _version: 0 error: Type: type_missing_exception Reason: "type[SourceNoDedupeFileContractV3] missing"

And another exception type also logged a lot of times indicating failure to index work items:

Microsoft.VisualStudio.Services.Search.Common.SearchPlatformException: ES Exception: [HTTP Status Code: [200] BULK_API_ERROR: [ update returned 404 _index: workitemsearchshared_0_2 _type: workItemContract _version: 0 error: Type: type_missing_exception Reason: "type[workItemContract] missing" update returned 404 _index: workitemsearchshared_0_2 _type: workItemContract _version: 0 error: Type: type_missing_exception Reason: "type[workItemContract] missing"

The exceptions seems to indicate that some type registrations are missing like the workItemContract and SourceNoDedupeFileContractV3 but I can not find any errors on the search server installation logs.

Anyone got some suggestions on how to solve this and get the Elastic Search back into a working state?

Answer 1

We resolved to situation by completely uninstalling and then reinstalling everything related to search.

1. Uninstalled all Code/Work/Wiki extensions from all collections from the extension management in web admin
2. Removed the TFS Search Service feature from the TFS Admin Console.
3. Uninstalled the elasticsearch service from the separate search server, using the PowerShell script .\Configure-TFSSearch.ps1 -operation remove

4. Restart TFS Job Agent service

5. Deleted old Search related databas content from ALL collection databases using

   DELETE FROM [Search].[tbl_IndexingUnit]

   DELETE FROM [Search].[tbl_IndexingUnitChangeEvent]

   DELETE FROM [Search].[tbl_IndexingUnitChangeEventArchive]

   DELETE FROM [Search].[tbl_JobYield]

   DELETE FROM [Search].[tbl_TreeStore]

   DELETE FROM [Search].[tbl_DisabledFiles]

   DELETE FROM [Search].[tbl_ItemLevelFailures]

   DELETE FROM [Search].[tbl_ResourceLockTable]

6. Restart TFS Job Agent service

7. Reboot Search server

8. Run Configure Search feature wizard from TFS Admin Console, using existing search server

9. Install search package according to instructions by PowerShell script

   .\Configure-TFSSearch.ps1 -Operation install -TFSSearchInstallPath D:\ES -TFSSearchIndexPath D:\ESDATA -Port 9200 -Verbose

10. Completed Search configuration wizard from TFS Admin Console enabling code search for all existing collections

11. Checked that services were running and tested searching from the web application, it works!