Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Kill a stuck thread on a running VM (JBoss Instance) in Java?

A bug in a third party library is causing an infinite loop in a worker thread on a JBoss instance of mine. Do you know of a way to kill this "stuck" thread without restarting the server? We'd like to be able to recover from this until a fix is deployed, preferably without having to restart.

I've seen a few people mention using Thread.interrupt() - if I were to code my own MBean, how would I get a handle to the thread in question in order to interrupt it?

Update: Wasn't able to solve using any of these methods. I did come across another thread about the same issue that had a link to why Thread.stop() is deprecated. Someone else has asked a similar question with similar results. It seems like more sophisticated containers should provide this kind of health mechanism, but I guess their hands are tied w/r/t the JVM.

like image 407
cwash Avatar asked Jan 21 '11 19:01

cwash


3 Answers

I had a similar bug (infinite loop) in a 3rd party lib. I ended up applying the fix myself (while waiting for the people from the 3rd party lib to fix their mess) and then I placed the modified .class in my .war, making sure it is loaded before the bogus .class (the bogus one being inside the bogus 3rd party .jar).

It is not nice but it works, see my question here:

Order of class loading from a .war file

What I mean is this: if you have to wait for the people responsible for the 3rd party bugged lib to fix their stuff, you can potentially be waiting a very long time. We couldn't afford that. We needed a fix ASAP. So we ended up applying a patch/hack to their code.

You could for example add a boolean check inside the infinite loop and then forcing the loop to exit when you want the bogus thread to "die".

Note that I haven't used the deprecated Thread stop() since ten years and I really didn't want to use it in the above case.

like image 77
SyntaxT3rr0r Avatar answered Oct 16 '22 04:10

SyntaxT3rr0r


I suppose the most difficult part is to identify the hanging thread. You provide no info about it, but perhaps you can build some rules around the thread's name or its current stack trace.

If you can identify the thread by its name, I would get all threads in the VM by getting my own thread group with Thread.currentThread().getThreadGroup(), then walk up the thread group hierarchy by calling getParent() on the thread group until it returns null. You now have the top level thread group. You can now fill a preallocated array with all threads using the enumerate(Thread[] list) method on the top level thread group.

If you need the stack traces anyway to identify the thread, you can also use the static utility method Map<Thread,StackTraceElement[]> Thread.getAllStackTraces() to get all threads. Computing the stack traces is however quite expensive, so this might not be the best solution if you don't actually need them.

After identifying the thread you must call the stop() method on it. Interrupting it won't help, unless the implementation of the running code actually evaluates the thread's interrupted flag and behaves as you expect it to. Not that the stop() method is deprecated and that using it may have many funny side effects. You can find more details in the API documentation.

like image 45
jarnbjo Avatar answered Oct 16 '22 03:10

jarnbjo


You could use the discouraged myThread.stop() method. But then it is very likely the Thread is still referenced there, so you should use some reflection magic to remove all references to this thread from the components holding it.

How to find the Thread? Use Thread.getThreadGroup() and ThreadGroup.getThreadGroup() to go up to the root ThreadGroup(), and then use the iterate() functions to go through all threads.

like image 1
Daniel Avatar answered Oct 16 '22 04:10

Daniel