IllegalMonitorStateException on awaitTermination function

Question

I'm having a problem with using threads in Java (I have little experience with threads in Java, but much in C++, so i understand basic concept of threads). I've used example code for threads in Java, and the code is next:

        ExecutorService executor = Executors.newFixedThreadPool(machines.size());

        for (Machine m : machines) {
            Runnable worker = new restartMachine(m.dataformachine());
            executor.execute(worker);
        }

        executor.shutdown();
        try {
            executor.awaitTermination(15, TimeUnit.MINUTES);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }

restartMachine() is restarting some remote machines, and machines are not connected in any way, data that are being passed to Runnable are IP address for given machine, and command that are executing then locally on that machine.

Error that I'm getting on execution of this piece of code is next:

java.lang.IllegalMonitorStateException
 at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:155)
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1260)
 at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:460)
 at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1471) 

Exception is thrown on calling of function awaitTermination() from code above. As I understand, and from various examples that I've seen there shouldn't be any problems with this code.

public boolean awaitTermination(long timeout, TimeUnit unit)
    throws InterruptedException {
    long nanos = unit.toNanos(timeout);
    final ReentrantLock mainLock = this.mainLock;
    mainLock.lock();
    try {
        for (;;) {
            if (runStateAtLeast(ctl.get(), TERMINATED))
                return true;
            if (nanos <= 0)
                return false;
            nanos = termination.awaitNanos(nanos);
        }
    } finally {
        mainLock.unlock();
    }
}

Trace indicate that error is in calling of function mainLock.unlock(); but as I understand it only main thread is going to execute that line, so I don't know why am I getting IllegalMonitorStateException, and there is no other code regarding threads in program (so I'm basically only using code from library)

I would appreciate any help, I know that there are many questions already answered regarding this problem (this exception), but I don't know what is the problem here.

Answer 1

This problem could be easily reproduced, if we will wrap your code inside some Thread and then call on him deprecated (just for demonstrating problem) method stop, e.g.:

  private void method() throws InterruptedException {
        Runnable runnable = new Runnable() {
            public void run() {
                ExecutorService executor = Executors.newFixedThreadPool(1);
                executor.execute(new Runnable() {
                    @Override
                    public void run() {
                        try {
                            Thread.sleep(10000L);
                        } catch (InterruptedException e) {
                            e.printStackTrace();
                        }
                    }
                });

                executor.shutdown();

                try {
                    executor.awaitTermination(3, TimeUnit.SECONDS);
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
            }
        };

        Thread thread = new Thread(runnable);
        thread.start();
        Thread.sleep(1000L);
        thread.stop();
    }

Running this code, we always get "desired" exception:

Exception in thread "Thread-0" java.lang.IllegalMonitorStateException
    at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:155)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1260)
    at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:460)
    at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1471)
    at q29431344.TestThreads$1.run(TestThreads.java:37)
    at java.lang.Thread.run(Thread.java:724)

What does it mean?

Without viewing full project code (of course, we are not asking it), hard to say with 100% waranty what did happened. But there are 2 possibilities:

1) Your restartMachine class has stopped machine on which this application was running itself. This caused to JVM stopping with such sequel

2) Some where in your application you run mentioned code in other thread, which somewhere was stopped in way which I have described or another one.

So, you have to analyze these ways and understand what could be more similiar to your situation.

UPD: Just another idea, 3) if you are running your application under Tomcat for example, this also can lead to such problem when Tomcat stops your application.

Answer 2

This is very peculiar, and probably not your fault:

The Javadoc of ReentrantLock.unlock says:

throws IllegalMonitorStateException if the current thread does not hold this lock

but the implementation of awaitTermination you have posted shows that the thread has successfully locked the very same object (through the final variable mainLock) previously. Therefore, there has been an intermediary unlock, or the ReentrantLock implementation has a bug (in its Java code, or native code, or possibly even the hard). Further analysis is necessary to discover which is the case. As you are currently the only one to be able to reproduce the problem, you are the only one that can perform that analysis effectively.

A reasonable first step would be to launch the application in debug mode, and set a breakpoint in AbstractOwnableSynchronizer.setExclusiveOwnerThread to verify whether there has been an intermediary unlock (and if so, from where). Should the presence of the breakpoint cause the problem to disappear (because it is timing sensitive), you might use a conditional breakpoint that never halts, but whose condition logs to System.out for your inspection, instead.

Update Thanks to the reproducer provided by Andremoniy in his answer, I was able to perform this analysis myself. I used the following expression in the conditional breakpoint to obtain the stack trace whenever the lock is aquired or released:

new RuntimeException(this + " is now owned by " + arg0).printStackTrace();
return false;

Here is relevant part of the log output for his code:

java.lang.RuntimeException: java.util.concurrent.locks.ReentrantLock$NonfairSync@a5e3519[State = 1, empty queue] is now owned by null
    at java.util.concurrent.locks.AbstractOwnableSynchronizer.setExclusiveOwnerThread(AbstractOwnableSynchronizer.java:74)
    at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:155)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.fullyRelease(AbstractQueuedSynchronizer.java:1723)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2069)
    at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1465)
    at stackoverflow.Test$1.run(Test.java:24)
    at java.lang.Thread.run(Thread.java:745)

...

java.util.concurrent.locks.ReentrantLock$NonfairSync@a5e3519[State = 0, empty queue] could not be released, as it is owned by null rather than Thread[Thread-0,5,main]

That is, the executor has released, but not reacquired, mainLock in awaitNanos, which is implemented as follows:

    public final long awaitNanos(long nanosTimeout)
            throws InterruptedException {
        if (Thread.interrupted())
            throw new InterruptedException();
        Node node = addConditionWaiter();
        int savedState = fullyRelease(node);
        final long deadline = System.nanoTime() + nanosTimeout;
        int interruptMode = 0;
        while (!isOnSyncQueue(node)) {
            if (nanosTimeout <= 0L) {
                transferAfterCancelledWait(node);
                break;
            }
            if (nanosTimeout >= spinForTimeoutThreshold)
                LockSupport.parkNanos(this, nanosTimeout);
            if ((interruptMode = checkInterruptWhileWaiting(node)) != 0)
                break;
            nanosTimeout = deadline - System.nanoTime();
        }
        if (acquireQueued(node, savedState) && interruptMode != THROW_IE)
            interruptMode = REINTERRUPT;
        if (node.nextWaiter != null)
            unlinkCancelledWaiters();
        if (interruptMode != 0)
            reportInterruptAfterWait(interruptMode);
        return deadline - System.nanoTime();
    }

As we can see from the absence of a finally block, the method is not exception safe, i.e. the lock not reacquired when an exception is thrown (such as the ThreadDeathException caused by Thread.stop()).

You might wish to report this bug to Oracle. However, since it only appears to manifest upon use of a deprecated api, and the impact is rather minor (wrong exception type is thrown), they might not fix it.