Docker containers shut down after systemd start

Question

For some reason when using systemd unit files my docker containers start but get shut down instantly. I have tried finding logs but can not see any indication on why this is happening. Is there someone that knows how to solve this / find the logs that show what is happening?

Note: When starting them manually after boot with docker start containername then it works (also when using systemctl start nginx)

After some more digging I found this error: could not find udev device: No such device it could have something to do with this?

Unit Service file:

[Unit]
Description=nginx-container
Requires=docker.service
After=docker.service

[Service]
Restart=always
RestartSec=2
StartLimitInterval=3600
StartLimitBurst=5
TimeoutStartSec=5
ExecStartPre=-/usr/bin/docker kill nginx
ExecStartPre=-/usr/bin/docker rm nginx
ExecStart=/usr/bin/docker run -i -d -t --restart=no --name nginx  -p 80:80 -v /projects/frontend/data/nginx/:/var/www -v /projects/frontend: nginx
ExecStop=/usr/bin/docker stop -t 2 nginx

[Install]
WantedBy=multi-user.target

Journalctl output:

May 28 11:18:15 frontend dockerd[462]: time="2015-05-28T11:18:15Z" level=info msg="-job start(d757f83d4a13f876140ae008da943e8c5c3a0765c1fe5bc4a4e2599b70c30626) = OK (0)"
May 28 11:18:15 frontend dockerd[462]: time="2015-05-28T11:18:15Z" level=info msg="POST /v1.18/containers/nginx/stop?t=2"
May 28 11:18:15 frontend dockerd[462]: time="2015-05-28T11:18:15Z" level=info msg="+job stop(nginx)"

Docker logs: empty (docker logs nginx)

Systemctl output: (systemctl status nginx, nginx.service)

● nginx.service - nginx-container
   Loaded: loaded (/etc/systemd/system/multi-user.target.wants/nginx.service)
   Active: failed (Result: start-limit) since Thu 2015-05-28 11:18:20 UTC; 12min ago
  Process: 3378 ExecStop=/usr/bin/docker stop -t 2 nginx (code=exited, status=0/SUCCESS)
  Process: 3281 ExecStart=/usr/bin/docker run -i -d -t --restart=no --name nginx -p 80:80 -v /projects/frontend/data/nginx/:/var/www -v /projects/frontend:/nginx (code=exited, status=0/SUCCESS)
  Process: 3258 ExecStartPre=/usr/bin/docker rm nginx (code=exited, status=0/SUCCESS)
  Process: 3246 ExecStartPre=/usr/bin/docker kill nginx (code=exited, status=0/SUCCESS)
 Main PID: 3281 (code=exited, status=0/SUCCESS)

May 28 11:18:20,frontend systemd[1]: nginx.service holdoff time over, scheduling restart.
May 28 11:18:20 frontend systemd[1]: start request repeated too quickly for nginx.service
May 28 11:18:20 frontend systemd[1]: Failed to start nginx-container.
May 28 11:18:20 frontend systemd[1]: Unit nginx.service entered failed state.
May 28 11:18:20 frontend systemd[1]: nginx.service failed.

Answer 1

Because you have not specified a Type in your systemd unit file, systemd is using the default, simple. From systemd.service:

If set to simple (the default if neither Type= nor BusName=, but ExecStart= are specified), it is expected that the process configured with ExecStart= is the main process of the service.

This means that if the process started by ExecStart exits, systemd will assume your service has exited and will clean everything up.

Because you are running the docker client with -d, it exits immediately...thus, systemd cleans up the service.

Typically, when starting containers with systemd, you would not use the -d flag. This means that the client will continue running, and will allow systemd to collect any output produced by your application.

That said, there are fundamental problems in starting Docker containers with systemd. Because of the way Docker operates, there really is no way for systemd to monitor the status of your container. All it can really do is track the status of the docker client, which is not the same thing (the client can exit/crash/etc without impacting your container). This isn't just relevant to systemd; any sort of process supervisor (upstart, runit, supervisor, etc) will have the same problem.