How to setup log-driver gcplogs while using gcloud compute instance update-container or force it to use default driver?

Question

Question

How to send logs from docker container standard output, to GCP Stackdriver, if this container was deployed using

gcloud compute instances update-container

command?

Most recent finding

gcloud compute instances update-container VM_INSTANCE --container-image IMG

• Spins off a container that doesn't respect default log-driver (set in /etc/docker/daemon.json)
• gcloud .. update-container uses "docker run" on VM_INSTANCE and allows to provide ARGS and COMMAND to "docker run" but no OPTIONS where I could put --log-driver=gcplogs

Context

My metadata "user-data" section:

#cloud-config
write_files:
  - path: /etc/docker/daemon.json
    content: '{"log-driver":"gcplogs"}'

runcmd:
  - systemctl restart docker

Gives a proper daemon.json file:

p**a@f**i ~ $ cat /etc/docker/daemon.json
{"log-driver":"gcplogs"}p**a@f**i ~ $

I have one container deployed with gcloud update-container command:

p**a@f**i ~ $ docker container ls
CONTAINER ID        IMAGE                                                                                   COMMAND                  CREATED             STATUS              PORTS               NAMES
425be3c47e2d        eu.gcr.io/a**i/f**e:9fa641db8491def826eef4f8f288ddbd62f2be1b   "yarn start --enviro…"   6 minutes ago       Up 3 minutes                            klt--xuyd

Double check that config was applied:

sudo journalctl -u docker.service

Apr 26 23:51:10 f**i systemd[1]: Started Docker Application Container Engine.
Apr 26 23:51:10 f**i dockerd[1464]: time="2019-04-26T23:51:10.036117615Z" level=info msg="API listen on /var/run/docker.sock"
Apr 26 23:30:21 f**i dockerd[711]: time="2019-04-26T23:30:21.903908969Z" level=debug msg="Using default logging driver gcplogs"

Then when I try to check what is the logger type it's still json-file:

p**a@f**i ~ $ docker inspect -f '{{.HostConfig.LogConfig.Type}}' klt--xuyd
json-file

Logs in stackdriver are missing. My filter value is:

resource.type="gce_instance"

All I can see are logs related with Compute Engine starting and stoping instance.

If I stop a docker container and run it with

docker run -d --log-driver=gcplogs --log-opt gcp-log-cmd=true IMG

Then checking of logger type gives a result:

docker inspect -f '{{.HostConfig.LogConfig.Type}}' klt--xuyd

gcplogs

And logs are visible in GCP Stackdriver.

But I can not execute docker run... as this VM is part of my CI/CD environment. Container is updated with

gcloud compute instances update-container ${VM_INSTANCE_NAME} --container-image IMG

container_vm: cos-stable-74-11895-86-0

I expect logs to be visible without necessity of stoping and starting container since it's set as a default.

Answer 1

I found this tutorial that can help you to configure Google Cloud logging driver on COS images.

Seems that installing the Stackdriver Agent is not an option on COS images as described here.

However, seems that you need to follow the configuration indicated in this link to make the configurations persistent across reboots.