Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Taking thread dumps in production

Tags:

I am analyzing the differences between approaches for taking thread dumps. Below are the couple of them I am researching on

  1. Defining a jmx bean which triggers jstack through Runtime.exec() on clicking a declared bean operation.

  2. Daemon thread executing "ManagementFactory.getThreadMXBean().dumpAllThreads(true, true)" repeatedly after a predefined interval.

Comparing the thread dump outputs between the two, I see the below disadvantages with approach 2

  1. Thread dumps logged with approach 2 cannot be parsed by open source thread dump analyzers like TDA
  2. The ouput does not include the native thread id which could be useful in analyzing high cpu issues (right?)
  3. Any more?

I would appreciate to get suggestions/inputs on

  1. Are there any disadvantages of executing jstack through Runtime.exec() in production code? any compatibility issues on various operating systems - windows, linux?

  2. Any other approach to take thread dumps?

Thank you.

Edit -

A combined approach of 1 and 2 seems to be the way to go. We can have a dedicated thread running in background and printing the thread dumps in the log file in a format understood by the thread dump analyzers. If any extra information is need (like say probably the native thread id) which is logged only by the jstack output, we do it manually as required.

like image 456
Andy Dufresne Avatar asked Jan 02 '13 07:01

Andy Dufresne


People also ask

When should we take thread dump?

You can do thread dumps whenever you see fit to analyse anything to do with thread lock contention, deadlock detection, system resource contention, ... This is why there are tools to facilitate thread dump whenever we see fit, not only after a JVM crash.

How do you perform a thread dump in performance testing?

The task on the left indicates the list of currently running processes. Click on the process for which you want the information, and select the thread tab to check the thread information in real time. Click the Thread Dump button on the top right corner to get the thread dump file.

When should I take heap dump and thread dump?

When ever we will get memory related issues that time we need to generate heap dump for finding root cause. Thread dump or Java core: It contains all threads information (waiting, run, hung, deadlock threads). Whenever we will get hung, dead lock issues that time we need to generate heap dump for finding root cause.

What is a thread dump?

A thread dump is a snapshot of the state of all threads that are part of the process. The state of each thread is presented with a so called stack trace, which shows the contents of a thread's stack.


2 Answers

You can use

jstack {pid} > stack-trace.log 

running as the user on the box where the process is running.

If you run this multiple times you can use a diff to see which threads are active more easily.


For analysing the stack traces I use the following sampled periodically in a dedicated thread.

 Map<Thread, StackTraceElement[]> allStackTraces = Thread.getAllStackTraces(); 

Using this information you can obtain the thread's id, run state and compare the stack traces.

like image 168
Peter Lawrey Avatar answered Sep 18 '22 03:09

Peter Lawrey


With Java 8 in picture, jcmd is the preferred approach.

jcmd <PID> Thread.print 

Following is the snippet from Oracle documentation :

The release of JDK 8 introduced Java Mission Control, Java Flight Recorder, and jcmd utility for diagnosing problems with JVM and Java applications. It is suggested to use the latest utility, jcmd instead of the previous jstack utility for enhanced diagnostics and reduced performance overhead.

However, shipping this with the application may be licensing implications which I am not sure.

like image 29
Arnab Biswas Avatar answered Sep 22 '22 03:09

Arnab Biswas