Logs not getting sent to AWS Cloudwatch when docker in detached or foreground

Question

When I run the docker script in interactive mode it works. I can see the logs in the console and also in AWS CloudWatch Logs. The below docker script runs in interactive mode and I have added the awslogs configuration so the logs go into cloudwatch. docker awslogs configuration

docker run --rm -i -t  --log-driver awslogs \
    --log-opt awslogs-region=us-east-1 \
    --log-opt awslogs-group=falcoint \
    --log-opt awslogs-create-group=true \
    --privileged \
    -v /dev:/host/dev \
    -v /proc:/host/proc:ro \
    -v /boot:/host/boot:ro \
    -v /lib/modules:/host/lib/modules:ro \
    -v /usr:/host/usr:ro \
    -v /etc:/host/etc:ro \
    falcosecurity/falco:latest

But once I run in -d detached mode none of the logs go to aws cloudwatch

docker run --rm -d --log-driver awslogs \
    --log-opt awslogs-region=us-east-1 \
    --log-opt awslogs-group=falcoint \
    --log-opt awslogs-create-group=true \
    --privileged \
    -v /dev:/host/dev \
    -v /proc:/host/proc:ro \
    -v /boot:/host/boot:ro \
    -v /lib/modules:/host/lib/modules:ro \
    -v /usr:/host/usr:ro \
    -v /etc:/host/etc:ro \
    falcosecurity/falco:latest

When I run the same script in foreground mode i.e. no -it or -d then also no logs are sent to the cloudwatch. But all the data is buffered and sent when the falco docker is stopped.

docker run --rm --log-driver awslogs \
    --log-opt awslogs-region=us-east-1 \
    --log-opt awslogs-group=falcoint \
    --log-opt awslogs-create-group=true \
    --privileged \
    -v /dev:/host/dev \
    -v /proc:/host/proc:ro \
    -v /boot:/host/boot:ro \
    -v /lib/modules:/host/lib/modules:ro \
    -v /usr:/host/usr:ro \
    -v /etc:/host/etc:ro \
    falcosecurity/falco:latest

When the falco docker is stopped it dumps the following to the log. Ideally the logs with "Error File created below..." should have come to CloudWatch Logs without having to stop the container.

2020-06-04T02:33:44+0000: SIGINT received, exiting...
Syscall event drop monitoring:
   - event drop detected: 0 occurrences
   - num times actions taken: 0
2020-06-04T02:32:32.495581404+0000: Notice A shell was spawned in a container with an attached terminal (user=root <NA> (id=01ca7b2306b5) shell=sh parent=runc cmdline=sh terminal=34816 container_id=01ca7b2306b5 image=<NA>)
2020-06-04T02:33:00.014981252+0000: Error File created below /dev by untrusted program (user=root command=touch /dev/rootkit2 file=/dev/rootkit2 container_id=01ca7b2306b5 image=<NA>)
2020-06-04T02:33:30.226554205+0000: Error File created below /dev by untrusted program (user=root command=touch /dev/rootkit3 file=/dev/rootkit3 container_id=01ca7b2306b5 image=<NA>)
Events detected: 3
Rule counts by severity:
   ERROR: 2
   NOTICE: 1
Triggered rules by rule name:
   Terminal shell in container: 1
   Create files below dev: 2

To repoduce the issues run one of the above and another terminal run

docker run -it node:8-alpine sh

then log into container and run

touch /dev/rootkit

UPDATE:

I noticed that when I run the docker with -d -t Logs to go to aws Cloudwatch logs. Any idea why this is happening?

Answer 1

By default, Docker uses a json-file driver, which writes JSON-formatted logs to a container-specific file on the host where the container is running. Refer this docker logging

Giving -t option assigns pseudo tty through which main process of docker outputs logs to the virtual terminal. And aws log-driver looks for tty to capture logs. Look at this how -t option works and specifically answer number 3 in this post.