Docker daemon memory leak due to logs from long running process

Question

I have the following setup:

1. Perl service running in a container and writing logs out to STDERR
2. logspout to ship those logs out to a remote server for archiving

in a 600 MB RAM machine.

I also truncate the logs periodically at:

/var/lib/docker/containers/CID/CID-json.log 

as suggested here to avoid 100% disk scenarios.

Problem

Docker daemon starts of with low memory usage, 1% initially and slowly increases to 40% after 2 days of running the container.

Reference

Docker daemon memory leak has been talked about in this issue and this issue. But both of them are closed now saying merged at a commit. Am running the latest major version of docker (Docker version 1.4.0, build 4595d4f), but still face a monotonically increasing memory usage issue.

EDIT: I did this experiment: Just run a bash process in the container, print out a lot of lines to STDERR, docker daemon process's memory usage accelerates very quickly

Does docker do some log buffering and doesn't release memory even if underlying log file (/var/lib/docker/containers/CID/CID-json.log) is cleared?

There's apparently no way to clear the logs. Will this commit solve this issue for long running tasks?

I don't know why docker daemon's memory usage keeps increasing. How do I debug this issue?

Answer 1

There is still at least one outstanding issue relating to memory leaks with logs: https://github.com/docker/docker/issues/9139