Terracotta Performance and Tips

Question

I am just learning how to use Terracotta after discovering it about a month ago. It is a very cool technology.

Basically what I am trying to do:

My root (System of Record) is a ConcurrentHashMap.

The main Instrumented Class is a "JavaBean" with 30 or so fields that I want to exist in the HashMap.

There will be about 20000 of these JavaBeans that exist in the Hashmap.

Each bean has (at least) 5 fields that will be updated every 5 seconds.

(The reason I am using Terracotta for this is because these JavaBeans need to be accessible across JVMs and nodes.)

Anyone with more experience than me with TC have any tips? Performance is key.

Any examples other similar applications?

Answer 1

You might find that batching several changes under one lock scope will perform better. Each synchronized block/method forms a write transaction (assuming you use a write lock) that must be sent to the server (and possibly back out to other nodes). By changing a bunch of fields, possibly on a bunch of objects under one lock, you reduce the overhead of creating a transaction. Something to play with at least.

Partitioning is also a key way to improve performance. Changes only need to be sent to nodes that are actually using an object. So if you can partition which nodes usually touch specific objects that reduces the number of changes that have to be sent around the cluster, which improves performance.

unnutz's suggestions about using CHM or CSM are good ones. CHM allows greater concurrency (as each internal segment can be locked and used concurrently) - make sure to experiment with larger segment counts too. CSM has effectively one lock per entry so has effectively N partitions in an N-sized table. That can greatly reduce lock contention (at the cost of managing more internal lock objects). Changes coming soon for CSM will make the lock mgmt cost much lower.

Generally we find a good strategy is:

1. Build a performance test (should be multi-threaded and multi-node and similar to your app (or your actual app!)
2. Tune objects - look at your clustered object graph in the dev-console to find objects that don't need to be clustered at all - sometimes this happens accidentally (remove or cut the cluster with a transient field). Sometimes you might be clustering a Date where a long would do. Small change but that's one object per map entry and that might make a difference.
3. Tune locks - use the lock profiler in the dev-console to find hot locks or locks that are too narrow or too wide. The clustered stats recorder can help look at transaction size as well.
4. Tune GC and DGC - tune JVM garbage collection, then tune Terracotta distributed GC by turning on changing the frequency of young gen gc.
5. Tune TC server - lots of very detailed tunings to do here, but usually not worth it till the stuff above is tuned.

Feel free to ask on the Terracotta forums as well - all of engineering, field engineering, product mgmt watch those and answer there.