Sonarqube upgrade to 6.7: Unrecoverable indexation failures

Question

Currently we are in middle of upgrade process from v4.5.7 to v6.7.1.

SonarQube is successfully upgraded to v5.6.7 (including DB upgrade) and currently in the process of updating to v6.7.1.

New SonarQube version started on an empty DB (bundled H2 DB) worked fine. After which all the plugins and their versions are validated and updated accordingly.

Now Sonar service launching is failing with error Background initialization failed. Stopping SonarQube java.lang.IllegalStateException: Unrecoverable indexation failures.

I removed data/es5 folder and reattempted to start sonar but with no luck. Any assistance to resolve the issue is greatly appreciated.

Web log:

  INFO  web[][o.s.p.ProcessEntryPoint] Starting web
  INFO  web[][o.a.t.u.n.NioSelectorPool] Using a shared selector for servlet write/read
  INFO  web[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.index.reindex.ReindexPlugin]
  INFO  web[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.join.ParentJoinPlugin]
  INFO  web[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.percolator.PercolatorPlugin]
  INFO  web[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
  INFO  web[][o.s.s.e.EsClientProvider] Connected to local Elasticsearch: [127.0.0.1:9002]
  INFO  web[][o.s.s.p.LogServerVersion] SonarQube Server / 6.7.1.35068 / 426519346f51f7b980a76f9050f983110550509d
  INFO  web[][o.sonar.db.Database] Create JDBC data source for jdbc:oracle:thin:@host:****:instance
  INFO  web[][o.s.s.p.ServerFileSystemImpl] SonarQube home: /.../sonar/sonarqube-6.7.1
  INFO  web[][o.s.s.u.SystemPasscodeImpl] System authentication by passcode is disabled
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin Branch / 1.0.0.507 / e9d161eaa2993f79890dd5d08230f3f566e6be93
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin Checkmarx / 8.41.0-20161229-0945
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin Developer / 1.0.0.240 / 25c709aeb128caa14725b943422cca19def50493
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin Flex / 2.3 / 79167f762219bea1fd181ff5ef6ca4b34abd1675
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin GitHub / 1.4.2.1027 / dcfb02a5fa40c219d527563f5b8c793966523ca7
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin Governance / 2.0.2.3011 / 4756b2f9d98d28cf2762d91eba1b48839a1edbe1
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin Groovy / 1.5 / dcbcd1eeb8f61c8351e5dd1e51e11e52d186df35
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin LDAP / 2.2.0.608 / 79dc3fa4393a29667673c70182f3016288b548b7
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin License / 3.3.0.1341 / 6b63240af64c92afe9421ece35c99c70f1b97991
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin Pitest / 0.9 / 0c011514ce025e5524fdcaa0c917eacf9d40f312
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SVG Badges / 3.0.1
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarABAP / 3.5.0.1080 / c37baa20eb9d24a27961ffe25d95ae4b716a3193
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarC# / 6.5.0.3766 / 6011a1e9fb002edb044ae3092d92bcfb7d9d8a2d
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarCFamily / 5.0.0.9359 / 6001ee3bce59e4bb9a17e5edbe40b4b99bc2901a
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarCOBOL / 4.1.1.2663 / 846e542a08e9e472f0216da6c30e66a96bcbc821
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarJS / 3.2.0.5506 / 86da09565f891121df248835665f5d8ed98772b6
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarJava / 4.15.0.12310 / 572454b93016ec73a53fe0e07b2ffdc356d21ba9
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarPHP / 2.11.0.2485 / 741861a29e5f9a26c6c99c06268facb6c4f4a882
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarPLI / 1.7.0.1117 / 2e07afcb9ecb6a2e96c980b48f7195b82ffdd014
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarPLSQL / 3.2.0.1753 / 455f471cfde991b2cd1a86a939ff410bf3006059
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarPython / 1.8.0.1496 / 3fe3bc4d0273a5721ea2fb368dc45b1bb82fede3
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarQube :: Plugins :: SCM :: Git / 1.3.0.869 / 4da53e3f9e55f4f2e5796625cb0c5768ed152079
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarQube :: Plugins :: SCM :: SVN / 1.6.0.860 / 2111fdbd1dddda4ad6d4ed6486fd0b18c1010d3b
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarRPG / 2.2.0.1005 / 146ae32dade755799da4fa9655e865d8757fddfa
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarSwift / 3.1.0.2067 / 22db8593d10c835675860598372d41ba8d5f0b8e
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarTS / 1.1.0.1079 / 042c9e65239a47d92d305f9767f730b3cc1e5ed3
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarTSQL / 1.1.0.2177 / e8811dfecb71d328c4deca89297bc100b270313c
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarVB / 4.2.0.248 / 75e2175c7464b814699abd6719b964c027717683
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarVB6 / 2.3.0.992 / 8c9e2016cb2e325b2fdeba4d303181b33bbbda4b
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarXML / 1.4.3.1027 / 39588245cecf538bb27be4e496ff303b0143d20b
  INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin Web / 2.5.0.476 / 636872f5d37fa7a440fe07d08d504e1a881225e5
  INFO  web[][c.c.sonar.CxPlugin] Checkmarx plugin loaded
  INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.ws.WebServiceFilter@6cdba960 [pattern=UrlPattern{inclusions=[/api/system/migrate_db/*, ...], exclusions=[/api/properties*, ...]}]
  INFO  web[][o.s.s.a.EmbeddedTomcat] HTTP connector enabled on port ****
  INFO  web[][o.s.s.p.UpdateCenterClient] Update center: https://update.sonarsource.org/update-center.properties (no proxy)
  INFO  web[][c.q.p.s.b.f.PreferredFontProvider] SVGImageGenerator will be using font 'Verdana' in order to compute SVG badges width.
  INFO  web[][c.q.p.s.b.w.SVGImageGenerator] SVGImageGenerator is now ready.
  INFO  web[][c.q.p.s.b.w.g.QualityGateBadgeGenerator] QualityGateBadgeGenerator is now ready.
  INFO  web[][c.q.p.s.b.w.m.MeasureBadgeGenerator] MeasureBadgeGenerator is now ready.
  INFO  web[][c.q.p.s.b.w.c.CeActivityBadgeGenerator] CeActivityBadgeGenerator is now ready.
  WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/editions/show_license
  WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/views/definition
  INFO  web[][o.s.s.s.LogServerId] Server ID: *********
  WARN  web[][o.s.s.u.DeprecatedViews] Widget 'Checkmarx' (checkmarx-project-widget) is ignored. See org.sonar.api.web.page.PageDefinition to define pages.
  WARN  web[][o.s.s.u.DeprecatedViews] Page 'Checkmarx' (/cx_config/index) is ignored. See org.sonar.api.web.page.PageDefinition to define pages.
  INFO  web[][org.sonar.INFO] Security realm: LDAP
  INFO  web[][org.sonar.INFO] Security realm started
  INFO  web[][o.s.s.t.TelemetryDaemon] Sharing of SonarQube statistics is enabled.
  INFO  web[][o.s.s.n.NotificationDaemon] Notification service started (delay 60 sec.)
  INFO  web[][o.s.s.s.GeneratePluginIndex] Generate scanner plugin index
  INFO  web[][o.s.s.s.RegisterPlugins] Register plugins
  INFO  web[][o.s.s.s.RegisterMetrics] Register metrics
  INFO  web[][o.s.s.r.RegisterRules] Register rules
  INFO  web[][o.s.s.q.BuiltInQProfileRepositoryImpl] Load quality profiles
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Register quality profiles
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile cpp/Sonar way
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile c/Sonar way
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile rpg/Sonar way
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile js/Sonar way Recommended
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile js/Sonar way
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile py/Sonar way
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile pli/Sonar way
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile tsql/Sonar way
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile vb/Sonar way
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile plsql/Sonar way
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile cs/Sonar way
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile java/Sonar way
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile cobol/Sonar way
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile web/Sonar way
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile flex/Sonar way
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile xml/Sonar way
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile php/Sonar way
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile php/PSR-2
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile php/Drupal
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile vbnet/Sonar way
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile abap/Sonar way
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile grvy/Sonar way
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile objc/Sonar way
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile swift/Sonar way
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile ts/Sonar way
  INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile ts/Sonar way recommended
  INFO  web[][o.s.s.s.RegisterPermissionTemplates] Register permission templates
  INFO  web[][o.s.s.s.RenameDeprecatedPropertyKeys] Rename deprecated property keys
  INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.ws.WebServiceFilter@1c655890 [pattern=UrlPattern{inclusions=[/api/measures/component/*, ...], exclusions=[/api/properties*, ...]}]
  INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.ws.DeprecatedPropertiesWsFilter@4f45e5bd [pattern=UrlPattern{inclusions=[/api/properties/*], exclusions=[]}]
  INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.ws.WebServiceReroutingFilter@29719313 [pattern=UrlPattern{inclusions=[/api/components/bulk_update_key, ...], exclusions=[]}]
  INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.InitFilter@694be25 [pattern=UrlPattern{inclusions=[/sessions/init/*], exclusions=[]}]
  INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.OAuth2CallbackFilter@61014c56 [pattern=UrlPattern{inclusions=[/oauth2/callback/*], exclusions=[]}]
  INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.ws.LoginAction@44a4512f [pattern=UrlPattern{inclusions=[/api/authentication/login], exclusions=[]}]
  INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.ws.LogoutAction@67e2e76 [pattern=UrlPattern{inclusions=[/api/authentication/logout], exclusions=[]}]
  INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.ws.ValidateAction@12d61b62 [pattern=UrlPattern{inclusions=[/api/authentication/validate], exclusions=[]}]
  INFO  web[][o.s.s.e.IndexerStartupTask] Indexing of type [components/component] ...
  INFO  web[][o.s.s.es.BulkIndexer] 201463 requests processed (3357 items/sec)
  INFO  web[][o.s.s.es.BulkIndexer] 357701 requests processed (2603 items/sec)
  INFO  web[][o.s.s.es.BulkIndexer] 506390 requests processed (2478 items/sec)
  INFO  web[][o.s.s.es.BulkIndexer] 636193 requests processed (2163 items/sec)
  INFO  web[][o.s.s.es.BulkIndexer] 766613 requests processed (2173 items/sec)
  INFO  web[][o.s.s.es.BulkIndexer] 899827 requests processed (2220 items/sec)
  INFO  web[][o.s.s.es.BulkIndexer] 899827 requests processed (0 items/sec)
  INFO  web[][o.s.s.es.BulkIndexer] 899827 requests processed (0 items/sec)
  INFO  web[][o.s.s.es.BulkIndexer] 899827 requests processed (0 items/sec)
  INFO  web[][o.s.s.es.BulkIndexer] 899827 requests processed (0 items/sec)
  INFO  web[][o.s.s.es.BulkIndexer] 899827 requests processed (0 items/sec)
  INFO  web[][o.s.s.es.BulkIndexer] 899827 requests processed (0 items/sec)
  INFO  web[][o.s.s.es.BulkIndexer] 899827 requests processed (0 items/sec)
  INFO  web[][o.s.s.e.IndexerStartupTask] Indexing of type [components/component] done | time=815372ms
  INFO  web[][o.s.s.e.IndexerStartupTask] Indexing of type [views/view] ...
  INFO  web[][o.s.s.e.IndexerStartupTask] Indexing of type [views/view] done | time=3122ms
  INFO  web[][o.s.s.e.IndexerStartupTask] Indexing of type [issues/issue] ...
  INFO  web[][o.s.s.es.BulkIndexer] 758897 requests processed (12648 items/sec)
  INFO  web[][o.s.s.es.BulkIndexer] 1395232 requests processed (10605 items/sec)
  INFO  web[][o.s.s.es.BulkIndexer] 1989212 requests processed (9899 items/sec)
  INFO  web[][o.s.s.es.BulkIndexer] 2567812 requests processed (9643 items/sec)
  INFO  web[][o.s.s.es.BulkIndexer] 3159399 requests processed (9859 items/sec)
  INFO  web[][o.s.s.es.BulkIndexer] 3320485 requests processed (2684 items/sec)
  INFO  web[][o.s.s.es.BulkIndexer] 3320485 requests processed (0 items/sec)
  ERROR web[][o.s.s.p.Platform] Background initialization failed. Stopping SonarQube
  java.lang.IllegalStateException: Unrecoverable indexation failures
at org.sonar.server.es.IndexingListener$1.onFinish(IndexingListener.java:39)
at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:117)
at org.sonar.server.issue.index.IssueIndexer.doIndex(IssueIndexer.java:247)
at org.sonar.server.issue.index.IssueIndexer.indexOnStartup(IssueIndexer.java:95)
at org.sonar.server.es.IndexerStartupTask.indexUninitializedTypes(IndexerStartupTask.java:68)
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
at org.sonar.server.es.IndexerStartupTask.execute(IndexerStartupTask.java:55)
at java.util.Optional.ifPresent(Optional.java:159)
at org.sonar.server.platform.platformlevel.PlatformLevelStartup$1.doPrivileged(PlatformLevelStartup.java:84)
at org.sonar.server.user.DoPrivileged.execute(DoPrivileged.java:45)
at org.sonar.server.platform.platformlevel.PlatformLevelStartup.start(PlatformLevelStartup.java:80)
at org.sonar.server.platform.Platform.executeStartupTasks(Platform.java:196)
at org.sonar.server.platform.Platform.access$400(Platform.java:46)
at org.sonar.server.platform.Platform$1.lambda$doRun$1(Platform.java:121)
at org.sonar.server.platform.Platform$AutoStarterRunnable.runIfNotAborted(Platform.java:371)
at org.sonar.server.platform.Platform$1.doRun(Platform.java:121)
at org.sonar.server.platform.Platform$AutoStarterRunnable.run(Platform.java:355)
at java.lang.Thread.run(Thread.java:745)
  INFO  web[][o.s.p.StopWatcher] Stopping process

Answer 1

Finally, our issue got resolved. System configuration seem to be the rootcause of this issue. We were running this on a server with approx. 25GB free space and ending up with indexation error. Using the server with more free space helped to upgrade Sonarqube to v6.7.1 successfully.