Puppetserver service fails to start

Question

I have a Vagrant CentOS VM running with ps.memory = 2048 RAM allocated.

When I try to start the puppetserver service:

$ puppet --version
4.4.0
$ sudo puppet resource service puppetserver ensure=running
Error: Could not start Service[puppetserver]: Execution of '/bin/systemctl start puppetserver' returned 1: Job for puppetserver.service failed. See 'systemctl status puppetserver.service' and 'journalctl -xn' for details.
Error: /Service[puppetserver]/ensure: change from stopped to running failed: Could not start Service[puppetserver]: Execution of '/bin/systemctl start puppetserver' returned 1: Job for puppetserver.service failed. See 'systemctl status puppetserver.service' and 'journalctl -xn' for details.
service { 'puppetserver':
  ensure => 'stopped',
}
$ journalctl -xn
No journal files were found.
$ systemctl status puppetserver.service
puppetserver.service - puppetserver Service
   Loaded: loaded (/usr/lib/systemd/system/puppetserver.service; disabled)
  Process: 4708 ExecStartPre=/usr/bin/install --directory --owner=puppet --group=puppet --mode=775 /var/run/puppetlabs/puppetserver (code=exited, status=0/SUCCESS)
 Main PID: 4709 (java);         : 4710 (bash)
   CGroup: /system.slice/puppetserver.service
           ├─4709 /usr/bin/java -Xms1g -Xmx1g -XX:MaxPermSize=1g -XX:OnOutOfMemoryError=kill -9 %p -Djava.security.egd=/...
           └─control
             ├─4710 /bin/bash /opt/puppetlabs/server/apps/puppetserver/ezbake-functions.sh wait_for_app
             └─4755 sleep 1

My JAVA_ARGS from /etc/sysconfig/puppetserver:

JAVA_ARGS="-Xms1g -Xmx1g -XX:MaxPermSize=1g"

As requested, the puppetserver.service file:

$ cat /usr/lib/systemd/system/puppetserver.service
[Unit]
Description=puppetserver Service
After=syslog.target network.target

[Service]
Type=simple
EnvironmentFile=/etc/sysconfig/puppetserver
User=puppet
TimeoutStartSec=120
TimeoutStopSec=60
Restart=on-failure
StartLimitBurst=5

PermissionsStartOnly=true
ExecStartPre=/usr/bin/install --directory --owner=puppet --group=puppet --mode=775 /var/run/puppetlabs/puppetserver

ExecStart=/usr/bin/java $JAVA_ARGS \
          '-XX:OnOutOfMemoryError=kill -9 %%p' \
          -Djava.security.egd=/dev/urandom \
          -cp "${INSTALL_DIR}/puppet-server-release.jar" clojure.main \
          -m puppetlabs.trapperkeeper.main \
          --config "${CONFIG}" \
          -b "${BOOTSTRAP_CONFIG}" $@

KillMode=process

ExecStartPost=/bin/bash "${INSTALL_DIR}/ezbake-functions.sh" wait_for_app

SuccessExitStatus=143

StandardOutput=syslog

[Install]
WantedBy=multi-user.target

An attempt at running the ExecStartPost command by hand:

$ /usr/bin/java -Xms1g -Xmx1g -XX:MaxPermSize=1g -XX:OnOutOfMemoryError='kill -9 %%p' -Djava.security.egd=/dev/urandom -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg

RuntimeError: Got 2 failure(s) while initializing: File[/var/log/puppetlabs/puppetserver]: change from 0700 to 0750 failed: failed to set mode 0700 on /var/log/puppetlabs/puppetserver: Operation not permitted - No message available; File[/var/run/puppetlabs/puppetserver]: change from 0775 to 0755 failed: failed to set mode 0775 on /var/run/puppetlabs/puppetserver: Operation not permitted - No message available

So I tried again, but this time I changed some directory permissions, but still similar error (which doesn't make sense given I just changed the mode?):

$ sudo chown -R vagrant:vagrant /var/run/puppetlabs/
$ sudo chown -R vagrant:vagrant /var/log/puppetlabs/
$ sudo chmod -R 0755 /var/run/puppetlabs/

$ /usr/bin/java -Xms1g -Xmx1g -XX:MaxPermSize=1g -XX:OnOutOfMemoryError='kill -9 %%p' -Djava.security.egd=/dev/urandom -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg

OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=1g; support was removed in 8.0
RuntimeError: Got 1 failure(s) while initializing: File[/var/run/puppetlabs/puppetserver]: change from 0775 to 0755 failed: failed to set mode 0775 on /var/run/puppetlabs/puppetserver: Operation not permitted - No message available
                use at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/settings.rb:1007

What could be the issue?

Answer 1

Are you certain that it’s an OnOutOfMemory error? I ask because I found that the latest PuppetServer includes a newer version of logback, as shown by this message in /var/log/messages:

Mar 18 01:56:21 puppetserver java: Exception in thread "main" java.lang.AbstractMethodError: ch.qos.logback.core.net.SyslogAppenderBase.createOutputStream()Lch/qos/logback/core/net/SyslogOutputStream;
Mar 18 01:56:21 puppetserver java: at ch.qos.logback.core.net.SyslogAppenderBase.start(SyslogAppenderBase.java:62)
Mar 18 01:56:21 puppetserver java: at ch.qos.logback.classic.net.SyslogAppender.start(SyslogAppender.java:48)

If you see, this replace “classic.net.Syslog” in logback.xml with "core.net.Syslog"

sed -i_old -e 's/classic.net.Syslog/core.net.Syslog/' /etc/puppetlabs/puppetserver/logback.xml

If that's not the problem, please post your logfiles.

Answer 2

You have provided log as

OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=1g; support was removed in 8.0

So, it is clear that you are using jdk 8 which removed permgen space.

The Permanent Generation (PermGen) space has completely been removed and is kind of replaced by a new space called Metaspace. The consequences of the PermGen removal is that obviously the PermSize and MaxPermSize JVM arguments are ignored and you will never get a java.lang.OutOfMemoryError: PermGen error.

1. So please remove the -XX:MaxPermSize=1g portion from JAVA_ARGS of location /etc/sysconfig/puppetserver

JAVA_ARGS="-Xms1g -Xmx1g"

So

2. And then,

your command will be look like below:

$ /usr/bin/java -Xms1g -Xmx1g -XX:OnOutOfMemoryError='kill -9 %%p' -Djava.security.egd=/dev/urandom -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg

or

$ /usr/bin/java -Xms1g -Xmx1g -Djava.security.egd=/dev/urandom -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg

Please try this 2 commands. I hope both will run successfully. For caution purpose, I have added both.

N.B: There is no need to change the file permission. keep them as before you using.

Otherwise, if you don't want to change anything, you can downgrade java 8 to java 7

Related Link:

1. PermGen elimination in JDK 8
2. Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize
3. JAVA 8 XX:PERMSIZE AND XX:MAXPERMSIZE DISAPPEARING
4. JDK 8 Milestones
5. JEP 122: Remove the Permanent Generation

Answer 3

@lollercoster, you're starting your service with:

sudo puppet resource service puppetserver ensure=running

But your next command is not telling which user it is

/usr/bin/java -Xms1g -Xmx1g -XX:MaxPermSize=1g -XX:OnOutOfMemoryError='kill -9 %%p' -Djava.security.egd=/dev/urandom -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg

Please run whoami directly before your run the above command:

whoami
/usr/bin/java -Xms1g -Xmx1g -XX:MaxPermSize=1g -XX:OnOutOfMemoryError='kill -9 %%p' -Djava.security.egd=/dev/urandom -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg

Why this commands? I am pretty confident that your issue is a permission/usercontext issue and following commands make it more worse:

$ sudo chown -R vagrant:vagrant /var/run/puppetlabs/
$ sudo chown -R vagrant:vagrant /var/log/puppetlabs/
$ sudo chmod -R 0755 /var/run/puppetlabs/

with the above cases you have to do a sudo on the right user context to have the files writable, but it depends if the other data is writable. So you can give it a try, but I would use the puppet user created for the service

User=puppet

So your puppet service runs as user puppet, but your logfiles are under vagrant user control, and not writable for puppet user.

So I would also try to switch back to:

$ sudo chown -R puppet /var/run/puppetlabs/
$ sudo chown -R puppet /var/log/puppetlabs/
$ sudo chmod -R 0755 /var/run/puppetlabs/

Java

Also I suggest you NOT to mess around with the JVM heap parameters, except you know what you do. Since you do not need to enforce a lower and a max bound on Java VM heap since JDK5. I would only limit to the upper bound

-Xmx1g

The heap and the real allocated mem will be managed well by the javaVM. As JVM needs more it will step by step increase the heap and keep the needed size (no downsizing to a lower value will occur). So you will have a better used RAM on your machine.

Also please switch to the Oracle JVM. I am facing often issues with security and Compatibility with OpenJDK. So my first test is to run it on the latest needed JDK of Oracle. In your case Oracle JDK8. But please try first the permission things I mentioned before.

Good luck, and please keep me updated.