Why is Elasticsearch starting manually but not starting as a service on Ubuntu 16.04?

Question

I installed ELasticsearch 6 and JDK 1.8.0_161 on my ARM Odroid XU4 running Ubuntu 16.04 LTS. No errors.

When I attempt to start Elasticsearch as a service, I can't connect to localhost on port 9200 and the service status prints:

~ $ > sudo service elasticsearch status
● elasticsearch.service - LSB: Starts elasticsearch
   Loaded: loaded (/etc/init.d/elasticsearch; bad; vendor preset: enabled)
   Active: active (exited) since Mon 2018-01-22 10:43:18 UTC; 9min ago
     Docs: man:systemd-sysv-generator(8)

Jan 22 10:43:18 lego systemd[1]: Starting LSB: Starts elasticsearch...
Jan 22 10:43:18 lego systemd[1]: Started LSB: Starts elasticsearch.
Jan 22 10:44:07 lego systemd[1]: Started LSB: Starts elasticsearch.

No logs. No clue where to look. nofile limits are set to 65536 in /etc/security/limits.conf. I don't know where else to look for diagnostics.

Although, when I start Elasticsearch manually via sudo /usr/shared/elasticsearch/bin/elasticsearch, it works as expected:

~ $ > sudo /usr/share/elasticsearch/bin/elasticsearch
[2018-01-22 10:55:55,944][WARN ][bootstrap                ] jvm uses the client vm, make sure to run `java` with the server vm for best performance by adding `-server` to the command line
[2018-01-22 10:55:56,073][INFO ][node                     ] [Ariel] version[1.7.3], pid[1126], build[NA/NA]
[2018-01-22 10:55:56,074][INFO ][node                     ] [Ariel] initializing ...
[2018-01-22 10:55:56,521][INFO ][plugins                  ] [Ariel] loaded [], sites []
[2018-01-22 10:55:56,638][INFO ][env                      ] [Ariel] using [1] data paths, mounts [[/ (/dev/mmcblk0p2)]], net usable_space 
[54.4gb], net total_space [57.2gb], types [ext4]
[2018-01-22 10:56:01,853][INFO ][node                     ] [Ariel] initialized
[2018-01-22 10:56:01,854][INFO ][node                     ] [Ariel] starting ...
[2018-01-22 10:56:02,080][INFO ][transport                ] [Ariel] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.0.103:9300]}
[2018-01-22 10:56:02,125][INFO ][discovery                ] [Ariel] elasticsearch/FtFOljAORnevIAOAFabptg
[2018-01-22 10:56:05,933][INFO ][cluster.service          ] [Ariel] new_master [Ariel][FtFOljAORnevIAOAFabptg][lego]
[inet[/192.168.0.103:9300]], reason: zen-disco-join (elected_as_master)
[2018-01-22 10:56:05,987][INFO ][http                     ] [Ariel] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.0.103:9200]}
[2018-01-22 10:56:05,988][INFO ][node                     ] [Ariel] started
[2018-01-22 10:56:06,014][INFO ][gateway                  ] [Ariel] 
recovered [0] indices into cluster_state

What important step am I missing? I followed everything here: https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html

EDIT: I tried setting START_DAEMON=true in /etc/default/elasticsearch, which is important indeed, but that did not do the trick either. Looking deeper into the values set inside /etc/default/elasticsearch, I found the problem in my case. See my final answer for a complete breakdown of the total solution.

Answer 1

As @Suaro states: our problem is related to what is discussed in this question with nearly identical symptoms and this thread that reiterates the START_DAEMON solution in the first link. That is a good start, but for me that didn't offer a comprehensive fix. I looked deeper at permissions, heap size, and ownership to find a final solution.

These are the steps for Elasticsearch 6 on Ubuntu 16.04 installed per the Elasticsearch debian installation instructions as of Jan 2018:

1. Set START_DAEMON=true in /etc/default/elasticsearch and restart service.
2. If the system has 2GB of RAM (true in my case), set ES_HEAP_SIZE=1g
3. Check the permissions of the elasticsearch directory in /usr/share/elasticsearch. Chances are that root owns these, which is not ideal. If you don't know already, running any service as root exposes your infrastructure to exploitation by attackers.
4. The temptation in #3 is to set ES_USER=root and ES_GROUP=root which will solve your problem. Elasticsearch will start as a service (even though their product documentation alleges that ES won't operate as root). DON'T DO THAT.

5. Instead, check that the elasticsearch user exists locally and that the group of the same name exists too.

   $ cut -d: -f1 /etc/passwd
   $ cut -d: -f1 /etc/group
   

6. Then, change ownership of all elasticsearch folders and resources to the elasticsearch user and group.

   $ ~ $ > ll /usr/share/elasticsearch/
   total 8.0K
   drwxr-xr-x 2 elasticsearch 4.0K Jan 22 10:02 bin/
   lrwxrwxrwx 1 elasticsearch   18 Dec 24  2015 config -> /etc/elasticsearch/
   lrwxrwxrwx 1 elasticsearch   22 Dec 24  2015 data -> /var/lib/elasticsearch/
   lrwxrwxrwx 1 elasticsearch   22 Dec 24  2015 logs -> /var/log/elasticsearch/
   drwxr-xr-x 2 elasticsearch 4.0K Dec 24  2015 plugins/
   
   # ^-- Take note that the symlinked directories need to be adjusted too
   
   $ sudo chown -R elasticsearch:elasticsearch /usr/share/elasticsearch
   $ sudo chown -R elasticsearch:elasticsearch /etc/elasticsearch/
   $ sudo chown -R elasticsearch:elasticsearch /var/lib/elasticsearch/
   $ sudo chown -R elasticsearch:elasticsearch /var/log/elasticsearch/
   

7. Then, set the values ES_USER=elasticsearch and ES_GROUP=elasticsearch in /etc/default/elasticsearch if they aren't set that way already (in case you gave in to temptation per #4).

8. Try it again...

   ~ $ > sudo service elasticsearch status
   ● elasticsearch.service - LSB: Starts elasticsearch
      Loaded: loaded (/etc/init.d/elasticsearch; bad; vendor preset: enabled)
      Active: active (exited) since Mon 2018-01-22 20:51:29 UTC; 2min 25s ago
        Docs: man:systemd-sysv-generator(8)
     Process: 3133 ExecStop=/etc/init.d/elasticsearch stop (code=exited, status=0/SUCCESS)
     Process: 3209 ExecStart=/etc/init.d/elasticsearch start (code=exited, status=0/SUCCESS)
   
   ~ $ > sudo systemctl restart elasticsearch.service
   
   ~ $ > sudo service elasticsearch status
   ● elasticsearch.service - LSB: Starts elasticsearch
      Loaded: loaded (/etc/init.d/elasticsearch; bad; vendor preset: enabled)
      Active: active (running) since Mon 2018-01-22 20:54:05 UTC; 2s ago
        Docs: man:systemd-sysv-generator(8)
     Process: 3306 ExecStop=/etc/init.d/elasticsearch stop (code=exited, status=0/SUCCESS)
     Process: 3340 ExecStart=/etc/init.d/elasticsearch start (code=exited, status=0/SUCCESS)
      CGroup: /system.slice/elasticsearch.service
              └─3391 /usr/lib/jvm/java-8-openjdk-armhf/bin/java -Xms1g -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.pidfil
   
   Jan 22 20:54:04 lego systemd[1]: Starting LSB: Starts elasticsearch...
   Jan 22 20:54:05 lego elasticsearch[3340]:  * Starting Elasticsearch Server
   Jan 22 20:54:05 lego elasticsearch[3340]:    ...done.
   Jan 22 20:54:05 lego systemd[1]: Started LSB: Starts elasticsearch.
   Jan 22 20:51:29 lego systemd[1]: Starting LSB: Starts elasticsearch...
   

Et Violà!

~ $ > curl -XGET 'localhost:9200/?pretty'
{
  "status" : 200,
  "name" : "Svarog",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "1.7.3",
    "build_hash" : "NA",
    "build_timestamp" : "NA",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}