How to debug a deadlocked process written in golang

Question

I have a process written in golang that runs as a daemon.

After a few days it stops producing output with no apparent reason. I think it might be an internal deadlock. If I do a strace -p <PID> I can see the line

futex(0x9aaba0, FUTEX_WAIT, 0, NULL

and apparently that call never ends. I would like to use the already running process (since I don't know how/when to trigger the bug again) to debug the problem.

How can I see what goroutines are running and where they? How can I dump from the process whatever other information might be useful to debug the problem?

Answer 1

Here are some methods from Deadlocks: the dark side of concurrency

• Use SIGQUIT
  • This produces a nice backtrace on Unix based system
  • Send it with kill -UQIT pid
  • Or press CTRL-\
  • Does not work on Windows
• Use the http debug handler
  • Retrieve the backtrace with a web browser or curl
  • Work on all platforms

Answer 2

You can send any go program a QUIT signal, and it will exit with a full stack trace, showing the state of all goroutines, and how they are blocked.

As for strace, go programs are always multithreaded, so you always need to add the -f option