Java: Monitor the acquisition of Locks

Question

I am currently experimenting a lot with java's security mechanisms in order to understand how to best execute untrusted code in a sandbox. One of the things you want to protect against are infinite loops which is why ideally you want to run the untrusted code in its own thread. Now, of course, malicious code could, for example, do some heavy processing resulting in a hanging thread. To get rid of this thread essentially the only way is to use java's deprecated Thread.stop() mechanism. The main reason this is problematic is that all of the locks held by the thread are released which could result in corrupted objects.

The question is: with Java's SecurityManager and custom classloaders I am able to track, for example, what classes can be loaded and what system resources can be accessed. Is there a way to be informed (and to potentially prohibit) code from acquiring locks (for example, defining a callback that is informed before the current thread goes into a synchronized block).

Answer 1

If you are already using a custom classloader, you could inspect the bytecode of each class before loading it and detect it if contains an instruction that grabs a lock (monitorenter).

You should also consider that locks released with stop() are only a problem if they are acquired on shared objects that other code might potentially lock. If you can avoid such objects to be accessible in the "evil" thread, you're safe.