mysql in docker container hangs

Question

Two mysql(5.6.20) instances in two docker containers (1.8.32), master and slave build semi-synchronous replication with each other, then users do some dml or ddl operating in master always。

after ten days or more, all the clients which connect to slave will hang

gdb -p/strace slave mysqld process hangs

pstack/perf top -p slave mysqld process show nothing

kill -9 will not kill the mysqld process

docker stop will not stop the docker container

what tools or methods can help locating the problem?

Answer 1

I had the same occur today. In my case, using docker compose to bring up mysql and a range of consumers, using the current "latest" mysql image from docker hub. (5.7.16-1debian8)

I've launched a number of these, and within a week I've seen a couple of instances where mysql has well over 100 threads, all the memory on the host is consumed, and the containers are hung. I can't stop anything, I can't even reboot. Only a power cycle of the VM recovers.

I'll try and monitor. I suspect it depends highly on infrastructure load (slow VM host results in slow queries backing up). The solution is more likely to be mysql tuning and a docker bug.