Why do I get "KeeperErrorCode = NoNode for /clusterstate.json" when I use SolrJ and SolrCloud?

Question

I've downloaded Solr 4.10.3, running the server with the provided setup, and I can insert documents into it using SolrJ (v. 4.0.0) by using HttpSolrServer without any problems. But I cannot insert any documents when I try to use CloudSolrServer. I've read the official documentation, and tried the following:

SolrServer solr;
String zkHostString = "localhost:9983/solr";
solr = new CloudSolrServer(zkHostString);

But I get the org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /clusterstate.json exception when I try solr.add(document);:

2015-03-31 11:00:18 INFO  HttpClientUtil:102 - Creating new http client, config:
2015-03-31 11:00:18 INFO  ZooKeeper:100 - Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
2015-03-31 11:00:18 INFO  ZooKeeper:100 - Client environment:host.name=emre-ubuntu
2015-03-31 11:00:18 INFO  ZooKeeper:100 - Client environment:java.version=1.8.0_25
2015-03-31 11:00:18 INFO  ZooKeeper:100 - Client environment:java.vendor=Oracle Corporation
2015-03-31 11:00:18 INFO  ZooKeeper:100 - Client environment:java.home=/usr/lib/jvm/java-8-oracle/jre
2015-03-31 11:00:18 INFO  ZooKeeper:100 - Client environment:java.class.path=/usr/lib/jvm/java-8-oracle/jre/lib/plugin.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jfxswt.jar:/usr/lib/jvm/java-8-oracle/jre/lib/deploy.jar:/usr/lib/jvm/java-8-oracle/jre/lib/javaws.jar:/usr/lib/jvm/java-8-oracle/jre/lib/management-agent.jar:/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jce.jar:/usr/lib/jvm/java-8-oracle/jre/lib/charsets.jar:/usr/lib/jvm/java-8-oracle/jre/lib/resources.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jfr.jar:/usr/lib/jvm/java-8-oracle/jre/lib/jsse.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/jfxrt.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/nashorn.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/sunpkcs11.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/dnsns.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/cldrdata.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/sunec.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/sunjce_provider.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/localedata.jar:/usr/lib/jvm/java-8-oracle/jre/lib/ext/zipfs.jar:/home/emre/code/belga-big-content/belga-spark-modules/target/classes:/home/emre/.m2/repository/com/sun/jersey/jersey-core/1.9.1/jersey-core-1.9.1.jar:/home/emre/.m2/repository/org/apache/hadoop/hadoop-client/2.4.0/hadoop-client-2.4.0.jar:/home/emre/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.4.0/hadoop-hdfs-2.4.0.jar:/home/emre/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-app/2.4.0/hadoop-mapreduce-client-app-2.4.0.jar:/home/emre/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-common/2.4.0/hadoop-mapreduce-client-common-2.4.0.jar:/home/emre/.m2/repository/org/apache/hadoop/hadoop-yarn-client/2.4.0/hadoop-yarn-client-2.4.0.jar:/home/emre/.m2/repository/com/sun/jersey/jersey-client/1.9/jersey-client-1.9.jar:/home/emre/.m2/repository/org/apache/hadoop/hadoop-yarn-server-common/2.4.0/hadoop-yarn-server-common-2.4.0.jar:/home/emre/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-shuffle/2.4.0/hadoop-mapreduce-client-shuffle-2.4.0.jar:/home/emre/.m2/repository/org/apache/hadoop/hadoop-yarn-api/2.4.0/hadoop-yarn-api-2.4.0.jar:/home/emre/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.4.0/hadoop-mapreduce-client-jobclient-2.4.0.jar:/home/emre/.m2/repository/org/apache/hadoop/hadoop-annotations/2.4.0/hadoop-annotations-2.4.0.jar:/home/emre/.m2/repository/org/jsoup/jsoup/1.8.1/jsoup-1.8.1.jar:/home/emre/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/home/emre/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar:/home/emre/.m2/repository/commons-codec/commons-codec/1.4/commons-codec-1.4.jar:/home/emre/.m2/repository/commons-io/commons-io/2.4/commons-io-2.4.jar:/home/emre/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar:/home/emre/.m2/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar:/home/emre/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar:/home/emre/.m2/repository/com/sun/jersey/jersey-json/1.9/jersey-json-1.9.jar:/home/emre/.m2/repository/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar:/home/emre/.m2/repository/com/sun/xml/bind/jaxb-impl/2.2.3-1/jaxb-impl-2.2.3-1.jar:/home/emre/.m2/repository/org/codehaus/jackson/jackson-jaxrs/1.8.3/jackson-jaxrs-1.8.3.jar:/home/emre/.m2/repository/org/codehaus/jackson/jackson-xc/1.8.3/jackson-xc-1.8.3.jar:/home/emre/.m2/repository/com/sun/jersey/jersey-server/1.9/jersey-server-1.9.jar:/home/emre/.m2/repository/asm/asm/3.1/asm-3.1.jar:/home/emre/.m2/repository/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar:/home/emre/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/home/emre/.m2/repository/org/apache/httpcomponents/httpcore/4.1.2/httpcore-4.1.2.jar:/home/emre/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/home/emre/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar:/home/emre/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.8.8/jackson-core-asl-1.8.8.jar:/home/emre/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.8.8/jackson-mapper-asl-1.8.8.jar:/home/emre/.m2/repository/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar:/home/emre/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/home/emre/.m2/repository/org/apache/zookeeper/zookeeper/3.4.5/zookeeper-3.4.5.jar:/home/emre/.m2/repository/org/apache/commons/commons-compress/1.4.1/commons-compress-1.4.1.jar:/home/emre/.m2/repository/org/tukaani/xz/1.0/xz-1.0.jar:/home/emre/.m2/repository/org/apache/hadoop/hadoop-mapreduce-client-core/2.4.0/hadoop-mapreduce-client-core-2.4.0.jar:/home/emre/.m2/repository/org/apache/hadoop/hadoop-yarn-common/2.4.0/hadoop-yarn-common-2.4.0.jar:/home/emre/.m2/repository/javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.2.jar:/home/emre/.m2/repository/javax/xml/stream/stax-api/1.0-2/stax-api-1.0-2.jar:/home/emre/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/home/emre/.m2/repository/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar:/home/emre/.m2/repository/com/google/inject/guice/3.0/guice-3.0.jar:/home/emre/.m2/repository/javax/inject/javax.inject/1/javax.inject-1.jar:/home/emre/.m2/repository/aopalliance/aopalliance/1.0/aopalliance-1.0.jar:/home/emre/.m2/repository/com/sun/jersey/contribs/jersey-guice/1.9/jersey-guice-1.9.jar:/home/emre/.m2/repository/com/google/inject/extensions/guice-servlet/3.0/guice-servlet-3.0.jar:/home/emre/.m2/repository/io/netty/netty/3.6.2.Final/netty-3.6.2.Final.jar:/home/emre/.m2/repository/json-mapreduce/json-mapreduce/1.0-SNAPSHOT/json-mapreduce-1.0-SNAPSHOT.jar:/home/emre/.m2/repository/org/apache/avro/avro-mapred/1.7.7/avro-mapred-1.7.7.jar:/home/emre/.m2/repository/org/apache/avro/avro-ipc/1.7.7/avro-ipc-1.7.7.jar:/home/emre/.m2/repository/org/apache/velocity/velocity/1.7/velocity-1.7.jar:/home/emre/.m2/repository/org/mortbay/jetty/servlet-api/2.5-20081211/servlet-api-2.5-20081211.jar:/home/emre/.m2/repository/org/apache/avro/avro-ipc/1.7.7/avro-ipc-1.7.7-tests.jar:/home/emre/.m2/repository/org/apache/avro/avro/1.7.7/avro-1.7.7.jar:/home/emre/.m2/repository/com/thoughtworks/paranamer/paranamer/2.3/paranamer-2.3.jar:/home/emre/.m2/repository/org/xerial/snappy/snappy-java/1.0.5/snappy-java-1.0.5.jar:/home/emre/.m2/repository/com/google/guava/guava/16.0/guava-16.0.jar:/home/emre/.m2/repository/org/slf4j/slf4j-log4j12/1.7.7/slf4j-log4j12-1.7.7.jar:/home/emre/.m2/repository/org/apache/solr/solr-solrj/4.0.0/solr-solrj-4.0.0.jar:/home/emre/.m2/repository/org/apache/httpcomponents/httpclient/4.1.3/httpclient-4.1.3.jar:/home/emre/.m2/repository/org/apache/httpcomponents/httpmime/4.1.3/httpmime-4.1.3.jar:/home/emre/.m2/repository/org/codehaus/woodstox/wstx-asl/3.2.7/wstx-asl-3.2.7.jar:/home/emre/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.5.0/jackson-databind-2.5.0.jar:/home/emre/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.5.0/jackson-annotations-2.5.0.jar:/home/emre/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.5.0/jackson-core-2.5.0.jar:/home/emre/.m2/repository/net/spy/spymemcached/2.11.6/spymemcached-2.11.6.jar:/home/emre/bin/idea-IC-139.659.2/lib/idea_rt.jar
2015-03-31 11:00:18 INFO  ZooKeeper:100 - Client environment:java.library.path=/home/emre/bin/idea-IC-139.659.2/bin::/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2015-03-31 11:00:18 INFO  ZooKeeper:100 - Client environment:java.io.tmpdir=/tmp
2015-03-31 11:00:18 INFO  ZooKeeper:100 - Client environment:java.compiler=<NA>
2015-03-31 11:00:18 INFO  ZooKeeper:100 - Client environment:os.name=Linux
2015-03-31 11:00:18 INFO  ZooKeeper:100 - Client environment:os.arch=amd64
2015-03-31 11:00:18 INFO  ZooKeeper:100 - Client environment:os.version=3.13.0-43-generic
2015-03-31 11:00:18 INFO  ZooKeeper:100 - Client environment:user.name=emre
2015-03-31 11:00:18 INFO  ZooKeeper:100 - Client environment:user.home=/home/emre
2015-03-31 11:00:18 INFO  ZooKeeper:100 - Client environment:user.dir=/home/emre/code/belga-big-content/belga-spark-modules
2015-03-31 11:00:18 INFO  ZooKeeper:438 - Initiating client connection, connectString=localhost:9983/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@7181ae3f
2015-03-31 11:00:18 INFO  ConnectionManager:156 - Waiting for client to connect to ZooKeeper
2015-03-31 11:00:18 INFO  ClientCnxn:966 - Opening socket connection to server localhost/127.0.0.1:9983. Will not attempt to authenticate using SASL (unknown error)
2015-03-31 11:00:18 INFO  ClientCnxn:849 - Socket connection established to localhost/127.0.0.1:9983, initiating session
2015-03-31 11:00:18 INFO  ClientCnxn:1207 - Session establishment complete on server localhost/127.0.0.1:9983, sessionid = 0x14c6eeee46d0009, negotiated timeout = 10000
2015-03-31 11:00:18 INFO  ConnectionManager:71 - Watcher org.apache.solr.common.cloud.ConnectionManager@7181ae3f name:ZooKeeperConnection Watcher:localhost:9983/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
2015-03-31 11:00:18 INFO  ConnectionManager:174 - Client is connected to ZooKeeper
2015-03-31 11:00:18 INFO  SolrZkClient:370 - makePath: /clusterstate.json
Exception in thread "main" org.apache.solr.common.cloud.ZooKeeperException: 
    at org.apache.solr.client.solrj.impl.CloudSolrServer.connect(CloudSolrServer.java:149)
    at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:165)
    at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
    at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
    at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
    at be.belga.bigcontent.util.SolrClient.addNewsItem(SolrClient.java:183)
    at be.belga.bigcontent.util.SolrClient.main(SolrClient.java:198)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /clusterstate.json
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
    at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
    at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
    at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:399)
    at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:63)
    at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:396)
    at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:353)
    at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:340)
    at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:327)
    at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:94)
    at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:84)
    at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:176)
    at org.apache.solr.client.solrj.impl.CloudSolrServer.connect(CloudSolrServer.java:142)
    ... 11 more

This is how I start my local Solr server:

$ solr start -e cloud -noprompt -a "-Dsolr.clustering.enabled=true"
Welcome to the SolrCloud example!


Starting up 2 Solr nodes for your example SolrCloud cluster.

Starting up SolrCloud node1 on port 8983 using command:

solr start -cloud -d node1 -p 8983   -a -Dsolr.clustering.enabled=true


Waiting to see Solr listening on port 8983 [\]  
Started Solr server on port 8983 (pid=3623). Happy searching!



Starting node2 on port 7574 using command:

solr start -cloud -d node2 -p 7574 -z localhost:9983  -a -Dsolr.clustering.enabled=true 


Waiting to see Solr listening on port 7574 [\]  
Started Solr server on port 7574 (pid=3855). Happy searching!


Deploying default Solr configuration files to embedded ZooKeeper using command:

/home/emre/bin/solr-4.10.3/example/scripts/cloud-scripts/zkcli.sh -zkhost localhost:9983 -cmd upconfig -confdir /home/emre/bin/solr-4.10.3/example/solr/collection1/conf -confname default

Successfully deployed the /home/emre/bin/solr-4.10.3/example/solr/collection1/conf configuration directory to ZooKeeper as default



Creating new collection gettingstarted with 2 shards and replication factor 2 using Collections API command:

http://localhost:8983/solr/admin/collections?action=CREATE&name=gettingstarted&replicationFactor=2&numShards=2&collection.configName=default&maxShardsPerNode=2&wt=json&indent=2

For more information about the Collections API, please see: https://cwiki.apache.org/confluence/display/solr/Collections+API

{
  "responseHeader":{
    "status":400,
    "QTime":61},
  "Operation createcollection caused exception:":"org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: collection already exists: gettingstarted",
  "exception":{
    "msg":"collection already exists: gettingstarted",
    "rspCode":400},
  "error":{
    "msg":"collection already exists: gettingstarted",
    "code":400}}


SolrCloud example running, please visit http://localhost:8983/solr

Answer 1

Apparently it was a matter of SolrJ version, the value of zkHostString, and using the class CloudSolrServer.

When I switched to SolrJ 4.10.3 and used the following section:

CloudSolrServer solr
String zkHostString = "localhost:9983";
solr = new CloudSolrServer(zkHostString);
solr.setDefaultCollection("collection1");

It worked fine and was able to insert a sample document into Solr for indexing.