logstash - Exception in thread ">output" org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [30s]

Question

Log stash is 100% a disaster for me. I am using LS 1.4.1 and ES 1.02 in the same machine.

Here is how I start logstash indexer:

/usr/local/share/logstash-1.4.1/bin/logstash -f /usr/local/share/logstash.indexer.config
input {
  redis {
    host => "redis.queue.do.development.sf.test.com"
    data_type => "list"
    key => "logstash"
    codec => json
  }
}

output {
        stdout { }
        elasticsearch {
                bind_host => "127.0.0.1"
                port => "9300"
        }
}

ES I set:

network.bind_host: 127.0.0.1
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["127.0.0.1:9300"]

And wow..this is what I get:

/usr/local/share/logstash-1.4.1/bin/logstash -f /usr/local/share/logstash.indexer.config
Using milestone 2 input plugin 'redis'. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.4.1/plugin-milestones {:level=>:warn}
log4j, [2014-05-29T12:02:29.545]  WARN: org.elasticsearch.discovery: [logstash-do-logstash-sf-development-20140527082230-866-2010] waited for 30s and no initial state was set by the discovery
Exception in thread ">output" org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [30s]
    at org.elasticsearch.action.support.master.TransportMasterNodeOperationAction$3.onTimeout(org/elasticsearch/action/support/master/TransportMasterNodeOperationAction.java:180)
    at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(org/elasticsearch/cluster/service/InternalClusterService.java:492)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java/util/concurrent/ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java/util/concurrent/ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(java/lang/Thread.java:744)

Answer 1

See http://logstash.net/docs/1.4.1/outputs/elasticsearch

VERSION NOTE: Your Elasticsearch cluster must be running Elasticsearch 1.1.1. If you use any other version of Elasticsearch, you should set protocol => http in this plugin.

So your problem is that logstash doesn't support the older ES version you are using without using an http transport.

Answer 2

Setting 'protocol => "http"' worked for me. I expected the EPEL repo to have complementary versions of logstash and elasticsearch, but ES is used for lots of stuff, thus is not tightly coupled with the logstash rpms.

Answer 3

For me, the problem wasn't with the versions of elasticsearch or logstash. I had just installed them and I was using the latest version of each (1.5.0 & 1.4.2 respectively).

Running the following worked for me as well:

logstash -e 'input { stdin { } } output { elasticsearch { protocol => "http" } }' 

But I wanted to get to the bottom of why I wasn't able to connect over the other protocols. Though the documentation doesn't say what the default protocol is, I was pretty sure I was either using transport or node for port 9300 by default because of the following output I got when I started elasticsearch

[2015-04-14 22:21:56,355][INFO ][node                     ] [Super-Nova] version[1.5.0], pid[10796], build[5448160/2015-03-23T14:30:58Z]
[2015-04-14 22:21:56,355][INFO ][node                     ] [Super-Nova] initializing ...
[2015-04-14 22:21:56,358][INFO ][plugins                  ] [Super-Nova] loaded [], sites []
[2015-04-14 22:21:58,186][INFO ][node                     ] [Super-Nova] initialized
[2015-04-14 22:21:58,187][INFO ][node                     ] [Super-Nova] starting ...
[2015-04-14 22:21:58,257][INFO ][transport                ] [Super-Nova] bound_address {inet[/127.0.0.1:9300]}, publish_address {inet[/127.0.0.1:9300]}
[2015-04-14 22:21:58,273][INFO ][discovery                ] [Super-Nova] elasticsearch/KPaTxb9vRnaNXBncN5KN7g
[2015-04-14 22:22:02,053][INFO ][cluster.service          ] [Super-Nova] new_master [Super-Nova][KPaTxb9vRnaNXBncN5KN7g][Azads-MBP-2][inet[/127.0.0.1:9300]], reason: zen-disco-join (elected_as_master)
[2015-04-14 22:22:02,069][INFO ][http                     ] [Super-Nova] bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[/127.0.0.1:9200]}
[2015-04-14 22:22:02,069][INFO ][node                     ] [Super-Nova] started

At first, I tried opening up port 9300 by following these instructions. That didn't change a thing, so most likely that port wasn't blocked.

Then I stumbled upon this github issue. There wasn't really a solution there that helped, but I did double check to make sure my elasticsearch cluster name was right by checking elasticsearch.yaml (This file is ususally stored where elasticsearch is installed. Run "which elasticsearch" to give you an idea where to look). Lo and behold, my elastisearch cluster.name had my name appended to it. Removing it so that the cluster name was just "elasticsearch" helped logstash discover my elasticsearch instance.