Failed to start Sonar Webservice

Question

I am working on Sonarqube 5.1. It used to work fine earlier, later i got this no space left on device error.So i have deleted some unwanted files and restarted the service. Now i can see that the service status is running, but i cannot access the webpage(since sonar webservice is not started) I have checked the logs and i find the below error.

2015.08.03 10:58:56 INFO   es[o.elasticsearch.node]  [sonar-1438592314778] started
2015.08.03 10:59:03 INFO   es[o.e.gateway]  [sonar-1438592314778] recovered [6] indices into cluster_state
2015.08.03 11:00:44 WARN   es[o.e.indices.cluster]  [sonar-1438592314778] [sourcelines][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [sourcelines][0] failed to recover shard
at     org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:287) ~[elasticsearch-1.4.4.jar:na]
at  org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132) ~[elasticsearch-1.4.4.jar:na]
at  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:114 2) [na:1.8.0_40]
at  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_40]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_40]
Caused by: org.elasticsearch.index.translog.TranslogCorruptedException: translog corruption while reading from stream
at  org.elasticsearch.index.translog.ChecksummedTranslogStream.read(ChecksummedTranslogStream.java:70) ~[elasticsearch-1.4.4.jar:na]
at  org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:257) ~[elasticsearch-1.4.4.jar:na]
... 4 common frames omitted
Caused by: org.elasticsearch.ElasticsearchException: failed to read [sourceline][22c2d1d1-8e73-47c0-b9fa-f8f5cc96b93f_796]
at org.elasticsearch.index.translog.Translog$Index.readFrom(Translog.java:520) ~[elasticsearch-1.4.4.jar:na]
at  org.elasticsearch.index.translog.ChecksummedTranslogStream.read(ChecksummedTranslogStream.java:68) ~[elasticsearch-1.4.4.jar:na]
... 5 common frames omitted
Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: No version type match [105]
at org.elasticsearch.index.VersionType.fromValue(VersionType.java:307) ~[elasticsearch-1.4.4.jar:na]
at org.elasticsearch.index.translog.Translog$Index.readFrom(Translog.java:517) ~[elasticsearch-1.4.4.jar:na]
... 6 common frames omitted
2015.08.03 11:00:44 WARN   es[o.e.c.action.shard]  [sonar-1438592314778] [sourcelines][0] sending failed shard for [sourcelines][0], node[5eRgYcVUTNCNLjhhzdiHGA], [P], s[INITIALIZING], indexUUID [JaH5lnRnRVOFD95Nw28W5Q], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[sourcelines][0] failed to recover shard]; nested: TranslogCorruptedException[translog corruption while reading from stream]; nested: ElasticsearchException[failed to read [sourceline][22c2d1d1-8e73-47c0-b9fa-f8f5cc96b93f_796]]; nested: ElasticsearchIllegalArgumentException[No version type match [105]]; ]]
2015.08.03 11:00:44 WARN   es[o.e.c.action.shard]  [sonar-1438592314778] [sourcelines][0] received shard failed for [sourcelines][0], node[5eRgYcVUTNCNLjhhzdiHGA], [P], s[INITIALIZING], indexUUID [JaH5lnRnRVOFD95Nw28W5Q], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[sourcelines][0] failed to recover shard]; nested: TranslogCorruptedException[translog corruption while reading from stream]; nested: ElasticsearchException[failed to read [sourceline][22c2d1d1-8e73-47c0-b9fa-f8f5cc96b93f_796]]; nested: ElasticsearchIllegalArgumentException[No version type match [105]]; ]]

do i have to delete shards? if yes how to do that? how to recover the shard? is indexing wrong? where do i find [sourcelines] on my machine ? help me with this.

Thankyou in advance.

Answer 1

Your ES index seems to be corrupted.

All base information is stored in the DB. Do not delete or update it manually.

On the contrary, if you delete the ES indexes, they will simply be rebuilt at the next startup (which may take some time depending on the quantity of data you have).

To achieve that, delete the es directory in the data directory of your SQ instance.

Answer 2

We have faced this issues many times. We delete the es directory under $SONAR_HOME/data directory of SQ instance and restart SQ instance. It start the indexing again Note : Only problem you will face , during indexing , your SQ webserver will be not accessible. You can not browse your localhost:9000(it will keep loading unless indexing gets over). As our project is big , it takes couple of hours to complete the indexing.Once the complete indexing is done, you can browse your SQ webserver.