Java "Thread-2" without stack prevents termination

Question

I have a pretty complex java program, which doesn't terminate. The eclipse debugger is showing a thread that can be suspended, but has no stack trace. It is called "Thread-2".

The jstack -l output for this thread is:

"Thread-2" #17 prio=5 os_prio=0 tid=0x00007f1268002800 nid=0x3342 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
    - None

I added a breakpoint to Thread.start(), but I cannot find a thread called "Thread-2". The thread just appears after the two "AWT-Event-Queue" threads were created. I don't create any threads manually in my program.

After the main thread and all other threads exit, and the JFrame is disposed, the following threads still exist:

Thread [AWT-EventQueue-0] (Running) 
Thread [Thread-2] (Running) 
Thread [DestroyJavaVM] (Running)    

When suspending the VM, the following threads exist:

Daemon System Thread [Signal Dispatcher] (Suspended)    
Daemon System Thread [Finalizer] (Suspended)    
Daemon System Thread [Reference Handler] (Suspended)    
Daemon System Thread [Java2D Disposer] (Suspended)  
Daemon System Thread [AWT-XAWT] (Suspended) 
Thread [AWT-EventQueue-0] (Suspended)   
Thread [Thread-2] (Suspended)   
Thread [DestroyJavaVM] (Suspended)  

How can I get more information about this thread, or allow it to terminate?

EDIT 1:

According to the Dependency Hierarchy of the eclipse pom.xml view, I use the following third party libraries:

guava 17.0 [compile]
hamcrest-core 1.3 [test]
junit 4.11 [test]
log4j-api 2.0-beta9 [compile]
log4j-core 2.0-beta9 [compile]

EDIT 2:

Adding breakpoints to all constructors of the thread class, as suggested in https://stackoverflow.com/a/35128213/577485, I see that Thread-0 and Thread-1 are created by log4j, but not Thread-2. It still just appears as before and no breakpoint triggers when it is constructed.

EDIT 3:

Now it's getting creepy. Not even the stop() method works, when invoked on the thread. I added it to the code given in https://stackoverflow.com/a/35128149/577485. At least System.exit(int) still works. But as said in a comment, I don't want to use that.

EDIT 4:

Info about my system:

• I'm running the newest stable version of Ubuntu 15.10 Wily. I have the security, updates and backports repos enabled.
• My JVM version is:

java version "1.7.0_91" OpenJDK Runtime Environment (IcedTea 2.6.3) (7u91-2.6.3-0ubuntu0.15.10.1) OpenJDK 64-Bit Server VM (build 24.91-b01, mixed mode)

EDIT 5:

I executed the program with Java version jre-8u71-linux-x64 directly downloaded from java.com, but the error persists. jstack -l shows the same strange thread. Note that the program was still built with the older java version. Edit: After compiling it with java8u72 from oracle.com, I get the same behaviour.

EDIT 6:

I iterated over all the threads fields, here's the output. I cannot get any hint out of those fields, the thread doesn't even have a target.

name: [C@6b67034
priority: 5
threadQ: null
eetop: 140274638530560
single_step: false
daemon: false
stillborn: false
target: null
group: java.lang.ThreadGroup[name=main,maxpri=10]
contextClassLoader: null
inheritedAccessControlContext: java.security.AccessControlContext@0
threadInitNumber: 3
threadLocals: null
inheritableThreadLocals: null
stackSize: 0
nativeParkEventPointer: 0
tid: 17
threadSeqNumber: 20
threadStatus: 5
parkBlocker: null
blocker: null
blockerLock: java.lang.Object@16267862
MIN_PRIORITY: 1
NORM_PRIORITY: 5
MAX_PRIORITY: 10
EMPTY_STACK_TRACE: [Ljava.lang.StackTraceElement;@453da22c
SUBCLASS_IMPLEMENTATION_PERMISSION: ("java.lang.RuntimePermission" "enableContextClassLoaderOverride")
uncaughtExceptionHandler: null
defaultUncaughtExceptionHandler: null
threadLocalRandomSeed: 0
threadLocalRandomProbe: 0
threadLocalRandomSecondarySeed: 0

EDIT 7:

Added a watchpoint to the name field of Thread. It is only accessed by my analytics code, and seems to be never written...

EDIT 8:

jstack -F -m throws an error for my program:

Attaching to process ID 10973, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.71-b15
Exception in thread "main" java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at sun.tools.jstack.JStack.runJStackTool(JStack.java:140)
    at sun.tools.jstack.JStack.main(JStack.java:106)
Caused by: java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007ff68000c000 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread, or SurrogateLockerThread)
    at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:169)
    at sun.jvm.hotspot.runtime.Threads.first(Threads.java:153)
    at sun.jvm.hotspot.tools.PStack.initJFrameCache(PStack.java:200)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:71)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:58)
    at sun.jvm.hotspot.tools.PStack.run(PStack.java:53)
    at sun.jvm.hotspot.tools.JStack.run(JStack.java:66)
    at sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:260)
    at sun.jvm.hotspot.tools.Tool.start(Tool.java:223)
    at sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
    at sun.jvm.hotspot.tools.JStack.main(JStack.java:92)
    ... 6 more
Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match    for type of address 0x00007ff68000c000
    at sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
    at sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
    at sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:165)
    ... 16 more

The class name of the strange thread is java.lang.Thread.

I don't use any command line arguments for executing the program. Adding the -Dlog4j2.disable.jmx=true option gives the strange thread the name Thread-1.

I updated log4j to 2.5, and the strange thread now has name Thread-0, when the -Dlog4j2.disable.jmx=true option is given, and Thread-1, when it isn't.

EDIT 9:

Completely removed log4j, error persists. The thread is now called Thread-0.

---

Here's the project, if that helps.

Answer 1

I don't understand why the breakpoint on Thread.start() did not work, but you could also try intercepting the thread >>creation<< by setting a breakpoint on the Thread constructors, or on the (internal) Thread.init() method.

The fact that the thread has the name Thread-2 implies that it was created by one of the constructors that generates a default thread name. That suggests that it was not created by the JVM or standard Java class libraries. It also narrows down the constructors that could have been used to create it.

How can I get more information about this thread ...

I can't think of any way apart from setting breakpoints.

... or allow it to terminate?

If you can find where it is created, you should be able to use setDaemon(true) to mark it as a daemon thread. However, this needs to be done before the thread is started.

Another possibility would be to find the thread by traversing the ThreadGroup tree and then calling Thread.interrupt() on it. ( Thread.getAllStackTraces() is another way of tracking down the thread object. ) However, there is no guarantee that the thread will "respect" the interrupt and shut down.

Finally, you could just call System.exit(...).

---

UPDATE

I mentioned that the thread might not respect interrupt() and I'm not surprised that stop() doesn't work. (It is deprecated, and may not even be implemented on some platforms.)

However, if you have managed to implement code that actually finds the mystery thread, you could dig around to find either the Thread subclass, or the Runnable that it is instantiated with. If you can print out the fully qualified class name, that will give you a big clue as to where it comes from. (Assuming that you are still having no success with breakpoints, then you may need to use "nasty" reflection to extract the runnable from the thread's private target field.)