Diagnosing high CPU usage on Docker for Mac

Question

How do I diagnose the cause of Docker on MacOS, specifically com.docker.hyperkit using 100% of CPU?

Docker stats

Docker stats shows all the running containers have low CPU, memory, net IO and block IO.

iosnoop

iosnoop shows that com.docker.hyperkit performs about 50 writes per second totaling 500KB per second to the file Docker.qcow2. According to What is Docker.qcow2?, Docker.qcow2 is a sparse file that's the persistent storage for all Docker containers.

In my case the file isn't that sparse. The physical size matches the logical size.

dtrace (dtruss)

dtruss sudo dtruss -p $DOCKER_PID shows a large number of psynch_cvsignal and psynch_cvwait calls.

psynch_cvsignal(0x7F9946002408, 0x4EA701004EA70200, 0x4EA70100)          = 257 0 psynch_mutexdrop(0x7F9946002318, 0x5554700, 0x5554700)           = 0 0 psynch_mutexwait(0x7F9946002318, 0x5554702, 0x5554600)           = 89474819 0 psynch_cvsignal(0x10BF7B470, 0x4C8095004C809600, 0x4C809300)             = 257 0 psynch_cvwait(0x10BF7B470, 0x4C8095014C809600, 0x4C809300)               = 0 0 psynch_cvwait(0x10BF7B470, 0x4C8096014C809700, 0x4C809600)               = -1 Err#316 psynch_cvsignal(0x7F9946002408, 0x4EA702004EA70300, 0x4EA70200)          = 257 0 psynch_cvwait(0x7F9946002408, 0x4EA702014EA70300, 0x4EA70200)            = 0 0 psynch_cvsignal(0x10BF7B470, 0x4C8097004C809800, 0x4C809600)             = 257 0 psynch_cvwait(0x10BF7B470, 0x4C8097014C809800, 0x4C809600)               = 0 0 psynch_cvwait(0x10BF7B470, 0x4C8098014C809900, 0x4C809800)               = -1 Err#316 

Update: top on Docker host

From https://stackoverflow.com/a/58293240/30900:

docker run -it --rm --pid host busybox top 

The CPU usage on docker embedded host is ~3%. CPU usage on my MacBook was ~100%. So, the docker embedded host isn't causing the CPU usage spike.

Update: running dtrace scripts of most common stack traces

Stack traces from the dtrace scripts in the answer below: https://stackoverflow.com/a/58293035/30900.

These kernel stack traces look innocuous.

              AppleIntelLpssGspi`AppleIntelLpssGspi::regRead(unsigned int)+0x1f               AppleIntelLpssGspi`AppleIntelLpssGspi::transferMmioDuplexMulti(void*, void*, unsigned long long, unsigned int)+0x91               AppleIntelLpssSpiController`AppleIntelLpssSpiController::transferDataMmioDuplexMulti(void*, void*, unsigned int, unsigned int)+0xb2               AppleIntelLpssSpiController`AppleIntelLpssSpiController::_transferDataSubr(AppleInfoLpssSpiControllerTransferDataRequest*)+0x5bc               AppleIntelLpssSpiController`AppleIntelLpssSpiController::_transferData(AppleInfoLpssSpiControllerTransferDataRequest*)+0x24f               kernel`IOCommandGate::runAction(int (*)(OSObject*, void*, void*, void*, void*), void*, void*, void*, void*)+0x138               AppleIntelLpssSpiController`AppleIntelLpssSpiDevice::transferData(IOMemoryDescriptor*, void*, unsigned long long, unsigned long long, IOMemoryDescriptor*, void*, unsigned long long, unsigned long long, unsigned int, AppleIntelSPICompletion*)+0x151               AppleHSSPISupport`AppleHSSPIController::transferData(IOMemoryDescriptor*, void*, unsigned long long, unsigned long long, IOMemoryDescriptor*, void*, unsigned long long, unsigned long long, unsigned int, AppleIntelSPICompletion*)+0xcc               AppleHSSPISupport`AppleHSSPIController::doSPITransfer(bool, AppleHSSPITransferRetryReason*)+0x97               AppleHSSPISupport`AppleHSSPIController::InterruptOccurred(IOInterruptEventSource*, int)+0xf8               kernel`IOInterruptEventSource::checkForWork()+0x13c               kernel`IOWorkLoop::runEventSources()+0x1e2               kernel`IOWorkLoop::threadMain()+0x2c               kernel`call_continuation+0x2e                53                kernel`waitq_wakeup64_thread+0xa7               pthread`__psynch_cvsignal+0x495               pthread`_psynch_cvsignal+0x28               kernel`psynch_cvsignal+0x38               kernel`unix_syscall64+0x27d               kernel`hndl_unix_scall64+0x16                60                kernel`hndl_mdep_scall64+0x4               113                kernel`ml_set_interrupts_enabled+0x19               524                kernel`ml_set_interrupts_enabled+0x19               kernel`hndl_mdep_scall64+0x10              5890                kernel`machine_idle+0x2f8               kernel`call_continuation+0x2e             43395 

The most common stack traces in user space over 17 seconds clearly implicate com.docker.hyperkit. There 1365 stack traces in 17 seconds in which com.docker.hyperkit created threads which averages to 80 threads per second.

              com.docker.hyperkit`0x000000010cbd20db+0x19f9               com.docker.hyperkit`0x000000010cbdb98c+0x157               com.docker.hyperkit`0x000000010cbf6c2d+0x4bd               libsystem_pthread.dylib`_pthread_body+0x7e               libsystem_pthread.dylib`_pthread_start+0x42               libsystem_pthread.dylib`thread_start+0xd                19                Hypervisor`hv_vmx_vcpu_read_vmcs+0x1               com.docker.hyperkit`0x000000010cbd4c4f+0x2a               com.docker.hyperkit`0x000000010cbd20db+0x174a               com.docker.hyperkit`0x000000010cbdb98c+0x157               com.docker.hyperkit`0x000000010cbf6c2d+0x4bd               libsystem_pthread.dylib`_pthread_body+0x7e               libsystem_pthread.dylib`_pthread_start+0x42               libsystem_pthread.dylib`thread_start+0xd                22                Hypervisor`hv_vmx_vcpu_read_vmcs               com.docker.hyperkit`0x000000010cbdb98c+0x157               com.docker.hyperkit`0x000000010cbf6c2d+0x4bd               libsystem_pthread.dylib`_pthread_body+0x7e               libsystem_pthread.dylib`_pthread_start+0x42               libsystem_pthread.dylib`thread_start+0xd                34                com.docker.hyperkit`0x000000010cbd878d+0x36               com.docker.hyperkit`0x000000010cbd20db+0x42f               com.docker.hyperkit`0x000000010cbdb98c+0x157               com.docker.hyperkit`0x000000010cbf6c2d+0x4bd               libsystem_pthread.dylib`_pthread_body+0x7e               libsystem_pthread.dylib`_pthread_start+0x42               libsystem_pthread.dylib`thread_start+0xd                47                Hypervisor`hv_vcpu_run+0xd               com.docker.hyperkit`0x000000010cbd20db+0x6b6               com.docker.hyperkit`0x000000010cbdb98c+0x157               com.docker.hyperkit`0x000000010cbf6c2d+0x4bd               libsystem_pthread.dylib`_pthread_body+0x7e               libsystem_pthread.dylib`_pthread_start+0x42               libsystem_pthread.dylib`thread_start+0xd               135 

Related issues

Github - docker/for-mac: com.docker.hyperkit 100% cpu usage is back again #3499 . One comment suggests adding volume caching described here: https://www.docker.com/blog/user-guided-caching-in-docker-for-mac/. I tried this and got a small ~10% reduction in CPU usage.

Answer 1

I have the same problem. My CPU % went back down to normal after I removed all my volumes.

docker system prune --volumes 

I also manually removed some named volumes:

docker volume rm NameOfVolumeHere 

That doesn't solve the overall issue of not being able to use volumes with Docker for mac. Right now I'm just being careful about the amount of volumes I use and closing Docker desktop when not in use.