Asserting order of synchronization in Java

Question

In highly concurrent systems, it can be difficult to be confident that your usage of locks is correct. Specifically, deadlocks can result if locks are acquired in an order that was not expected while being acquired in the proper order in another thread.

There are tools (e.g. Coverity) which can do static analysis on a code base and look for "unusual" locking orders. I'd like to explore other options to meet my needs.

Are there any light-weight* tools for instrumenting Java code which can detect cases where locks are being acquired in an order other than expected? I am okay with explicitly calling out locking orders via comments / annotations.

Free and/or open-source solutions preferred. Please also comment if there are non-instrumentation approaches to this problem.

* For my purposes, light-weight means...

• If it is instrumentation, I can still run my program with the same ballpark performance. 30-50% degradation is acceptable, I suppose.
• I don't have to spend half the day interacting with the tool just to get an "okay" out of it. Ideally I should only notice that I'm using it when there's a problem.
• If it is instrumentation, it should be easy to disable for production environments.
• It shouldn't clutter my code at every synchronize statement. As previously mentioned, I'm okay with explicitly commenting/annotating the objects or classes of objects which get locked with relative orderings.

Answer 1

I have not used AspectJ so cannot vouch for how easy it is to use. I have used ASM to create a custom code profiler, this was about 2 days work. The effort to instrument synchronization should be similar. AspectJ should be quicker and easier once you are up to speed with aspects.

I have implemented deadlock detecting trace for our c++ based server. Here is how I did it:

• When ever acquiring or releasing a lock I traced:
  • <time> <tid> <lockid> <acquiring|releasing> <location in code>
• This extra trace affected performance quite drastically and was not usable in production.
• So when a possible deadlock was discovered in production I used the log file to figure out what was happening around the deadlock. Then reproduced this functionality in a test environment with my tracing turned on.
• Then I ran a script on the log file to see if deadlock was possible and how. I used an awk script, using this algoritm:
  • Foreach line
    • if acquiring
      • add lockid to list of current locks for this thread
      • add each pair of locks in this list to a set lock pairs for this thread. eg for list of Lock A -> Lock B -> Lock C generate the pairs (Lock A, Lock B), (Lock A, Lock C), (Lock B, Lock C)
    • if releasing
      • remove current lockid from tail of list for this thread
  • For each lock pair search all other threads for the reverse lock pairs, each match is a potential deadlock so print the pairs and threads affected
  • Instead of making the algorithm smarter I then desk checked that the lock acquisition to see if it was a real deadlock.

I did this after failing to find the cause of a deadlock for a number of days, it took a few more days to implement and a few hours to find the deadlock.

If you are considering this approach in Java things to consider are:

• Do you only use synchronized to protect your critical sections? Are you using the classes in java.lang.concurrent? (these might require special handling/instrumentation)
• How easy is it to print the code location with aspects/ASM? I used __FILE__ and __LINE__ in c++. ASM will give you the class name, method name and signature.
• You cannot instrument the locks used to protect your tracing/logging.
• You can streamline your instrumentation if you use a log file per thread and thread local storage for the file object.
• How do you uniquely identify objects you synchronize on? Maybe toString() and System.identityHashCode() would be enough, but might require more. I used the address of the object in C++.

Answer 2

You can use AspectJ, which is relatively easy to learn and will allow you to setup your own customized and simplified way of monitoring your threads and any locks they access.