How to debug a multithreaded hung process in Linux?

Question

A multithreaded application hangs and it is not responding to any commands. I have tried following things without luck:

1. Attach a process to gdb (error: (gdb) attach 6026 Attaching to process 6026 ptrace: Operation not permitted.)
2. gstack (gstack just hangs like that)

Is there any good way to debug this process?

Answer 1

Thanks for all your response. The problem is at kernel level. we used echo t > /proc/sysrq-trigger, which logs the stack of all the running process in /var/log/messages. This stack trace helped to analyze the problem.

From the stack trace, file system posted some waited event on behalf of the application process to other process (which is in defunct state) and waiting for the response indefinitely. Which results in hung state.

Answer 2

Most likely somebody else already tracing this process. To find out who is doing it look at proc file system.

cat /proc/6026/status|grep TracerPid