Elasticsearch Unreachable: Connection refused

Question

I have the ELK stack running inside docker containers inside a VM.
I can curl stuff to ES and it shows up in Kibana just fine.
I can read files with Logstash and output them to stdout.
But Logstash fails to send the data to ES

docker-compose.yml

version: '2'
services:
  elasticsearch:
    image: elasticsearch
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      ES_JAVA_OPTS: "-Xmx256m -Xms256m"
      xpack.security.enabled: "false"
      xpack.monitoring.enabled: "false"
      xpack.graph.enabled: "false"
      xpack.watcher.enabled: "false"
    networks:
      - elk

  logstash:
    image: docker.elastic.co/logstash/logstash:5.3.2
    volumes:
      - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml
      - ./logstash/pipeline:/usr/share/logstash/pipeline
      - ./data:/usr/share/data
    ports:
      - "5000:5000"
      - "9600:9600"
    networks:
      - elk
    depends_on:
      - elasticsearch

  kibana:
    # kibana config

networks:
  elk:
    driver: bridge

logstash.yml (enabling or disabling xpack doesnt seem to make a difference)

#log.level: debug
xpack.monitoring.elasticsearch.password: changeme
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.url: http://127.0.0.1:9200
xpack.monitoring.enabled: false

pipeline.conf

input {
# some files
}
filter{
# csv filter
}
output {
    if ([type] == "GPS") {
      stdout { }
      elasticsearch {
          hosts => [ "127.0.0.1:9200" ]
          index => "GPS"
          template_overwrite => true
          user => logstash_system
          password => changeme
      }
    }
}

The output of docker-compose up

elasticsearch_1  [2017-05-03T05:06:29,821][INFO ][o.e.n.Node               ] initialized
elasticsearch_1  [2017-05-03T05:06:29,821][INFO ][o.e.n.Node               ] [SzDlw_j] starting ...
elasticsearch_1  [2017-05-03T05:06:30,352][WARN ][i.n.u.i.MacAddressUtil   ] Failed to find a usable hardware address from the network interfaces; using random bytes: 74:70:60:88:e2:f8:41:36
elasticsearch_1  [2017-05-03T05:06:30,722][INFO ][o.e.t.TransportService   ] [SzDlw_j] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
elasticsearch_1  [2017-05-03T05:06:30,744][WARN ][o.e.b.BootstrapChecks    ] [SzDlw_j] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
elasticsearch_1  [2017-05-03T05:06:34,027][INFO ][o.e.c.s.ClusterService   ] [SzDlw_j] new_master {SzDlw_j}{SzDlw_jiQsWcXN00BZfZHQ}{ELB1oDR4SlW6atQIM88hfg}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
elasticsearch_1  [2017-05-03T05:06:34,176][INFO ][o.e.h.n.Netty4HttpServerTransport] [SzDlw_j] publish_address {172.18.0.2:9200}, bound_addresses {[::]:9200}
elasticsearch_1  [2017-05-03T05:06:34,179][INFO ][o.e.n.Node               ] [SzDlw_j] started
elasticsearch_1  [2017-05-03T05:06:34,832][INFO ][o.e.g.GatewayService     ] [SzDlw_j] recovered [1] indices into cluster_state
elasticsearch_1  [2017-05-03T05:06:36,519][INFO ][o.e.c.r.a.AllocationService] [SzDlw_j] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana][0]] ...]).
logstash_1       Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties
logstash_1       [2017-05-03T05:06:55,865][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://127.0.0.1:9200/]}}
logstash_1       [2017-05-03T05:06:55,887][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://127.0.0.1:9200/, :path=>"/"}
logstash_1       [2017-05-03T05:06:56,131][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x7b0aac02 URL:http://127.0.0.1:9200/>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
logstash_1       [2017-05-03T05:06:56,140][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
logstash_1       [2017-05-03T05:06:56,147][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused) {:url=>http://127.0.0.1:9200/, :error_message=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
logstash_1       [2017-05-03T05:06:56,148][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:287:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:273:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:363:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:272:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:280:in `get'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:83:in `get_version'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:16:in `get_es_version'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:20:in `get_es_major_version'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:7:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/common.rb:54:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-6.3.0-java/lib/logstash/outputs/elasticsearch/common.rb:21:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:8:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:41:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:257:in `register_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:268:in `register_plugins'", "org/jruby/RubyArray.java:1613:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:268:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:277:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:207:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:389:in `start_pipeline'"]}
logstash_1       [2017-05-03T05:06:56,149][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<URI::Generic:0x6d11738 URL://127.0.0.1:9200>]}
logstash_1       [2017-05-03T05:06:56,163][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>125}
logstash_1       [2017-05-03T05:06:56,703][INFO ][logstash.pipeline        ] Pipeline main started
logstash_1       2017-05-03T05:06:56.776Z b68ff5316e02 time,lat,lon,elevation,accuracy,bearing,speed
logstash_1       2017-05-03T05:06:56.778Z b68ff5316e02 2014-09-14T00:26:23Z,98.404222,99.999021,9.599976,44.000000,0.000000,0.000000
logstash_1       2017-05-03T05:06:56.779Z b68ff5316e02 2014-09-14T00:48:45Z,98.404297,99.999338,9.000000,35.000000,102.300003,0.500000
logstash_1       [2017-05-03T05:06:57,035][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
logstash_1       [2017-05-03T05:06:57,058][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
logstash_1       [2017-05-03T05:06:57,069][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>2}
logstash_1       [2017-05-03T05:06:59,085][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
logstash_1       [2017-05-03T05:06:59,086][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>4}
logstash_1       [2017-05-03T05:07:01,146][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://127.0.0.1:9200/, :path=>"/"}
logstash_1       [2017-05-03T05:07:01,158][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x7b0aac02 URL:http://127.0.0.1:9200/>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://127.0.0.1:9200/][Manticore::SocketException] Connection refused (Connection refused)"}

after that it keeps looping between "sending bulk request", "performing health check", "attempting to resurrect" and "unexpected pool error".
With log.level: debug every once in a while i also see Error, cannot retrieve cgroups information {:exception=>"Errno::ENOENT", :message=>"No such file or directory - /sys/fs/cgroup/cpuacct/system.slice/docker-<containerId>.scope/cpuacct.usage"}

Answer 1

One of docker's purposes is isolation, from the point of view of the containers, 127.0.0.1 only refers to themselves.

Here you have 3 containers:

• elasticsearch
• logstash
• kibana

You should modify your logstash configuration to put elasticsearch instead of 127.0.0.1, as it's the name it knows for the elasticsearch container on the elk bridge network you have defined