100% CPU usage after logstash install

Question

I followed this tutorial to install a software stack Logstash/ES/Kibana on my Ubuntu server. I changed logstash configuration to test everything locally before trying to ship logs. So I have a single node running ES/Kibana and Logstash configured as follow :

input {
  file {
    path => "/var/log/syslog"
    type => "syslog"
  }
}
output {
  elasticsearch { host => localhost }
}

Everything is working as intended from what I can see on Kibana, but I have a background process that eats 100% cpu. Top tells me it's a job in java running under logstash user. sudo service logstash stop does not stop the process from running. I've also tried to remove web service following this, without success.

Answer 1

Digital Ocean's tutorial uses nginx in front of Kibana and listens on port 80. logstash ships with logstash-web that also wants to listen to port 80.

Since Ubuntu uses upstart, trying to kill the java processes won't succeed as they will keep respawning according to /etc/init/logstash*.conf. The high CPU usage comes from the fact that logstash uses a lot of CPU time at startup and should calm down after a couple of seconds, but because it dies being unable to bind to port 80 and keeps respawning, it looks as if it's constantly using resources.

If you have the same problem as I did, look at logstash's PIDs and you will notice they change. You should also see Address already in use - bind - Address already in use at the end of /var/log/logstash/logstash.log.

So, we just need to disable logstash-web. On Ubuntu, this can be done with:

$ echo manual | sudo tee /etc/init/logstash-web.override

To stop logstash-web without rebooting, we use

$ sudo stop logstash-web