Restart docker when it consumes certain amount of memory

Question

I have multiple docker processes running on a machine, I'd like to monitor them and restart docker process once it passes certain memory threshold. Can I make docker restart it's process when it hits memory threshold? What's my options?

Thanks

Alex

Answer 1

You could make a shell script to monitor resources usage, and restart Docker daemon when it reaches your memory limit, but I think that's not actually a good approach.

Using this command you can see your containers ordered by memory usage. Find which container is using too much memory and try to find the reason because that's happening.

docker stats --no-stream --format "table {{.Name}}\t{{.Container}}\t{{.CPUPerc}}\t{{.MemUsage}}" | sort -k 4 -h

Also, if memory consumption of your containers is normal, but you want to limit it, you can limit resources assigned to each container. You can do this using option --memory in docker run.

For further information about memory limits, check this info in Docker docs: https://docs.docker.com/engine/admin/resource_constraints/

Hope this helps, good luck.

Edit: Answering your response, if your container runs out of memory, it will be automatically killed by the kernel. You can configure a memory limit using option --memory and set restart policy as --restart=always. This way, your container will be killed automatically by an OOM (out-of-memory) error, but it will be restarted since its restart policy is to keep restarting after any error.

Answer 2

Always remember that if you don't set --memory-swap, the container will no to restart at --memory value limit because Docker will use Swap after reach --memory limit, so, if you want to restart at --memory value, you need to set --memory-swap with the same value as --memory limit.

docker run --memory 50m --memory-swap 50m --rm -it progrium/stress --vm 1 --vm-bytes 62914560 --timeout 1s