jvmti agent deadlock

Question

When I run my jvmti agent with a java program, it seems that jvm encounters a deadlock. In my jvmti agent, I create a single raw monitor in Agent_OnLoad() and enter that lock at the beginning of every callback function and exit that lock at the end of every callback functions. I don't know the reason of this deadlock. Is there any other possibilities of deadlocks in jvmti agents?

thanks.

Answer 1

Yes, there is a possibility of deadlocks in JVMTI Agent. The JVMTI Reference states:

The same thread may enter a monitor more then once. The thread must exit the monitor the same number of times as it is entered. If a monitor is entered during OnLoad (before attached threads exist) and has not exited when attached threads come into existence, the enter is considered to have occurred on the main thread.

It can be difficult to make a remote diagnosis in this case, but I would suggest you take a look at how the demo JVMTI applications handle the use of raw locks. The use dedicated functions enter_critical_section and exist_critical_section. Maybe that helps, otherwise try to find the callback which results in a deadlock by debugging the agent.