Hazelcast CP Subsystem FencedLock memory usage

Question

Setup:

• Hazelcast: 3.12.3
• Spring Boot: 2.1.6
• Java: 1.8
• Args: -XX:+UseG1GC
• Xms7g -Xmx7g These are running in Docker: openjdk:8
• they run in dedicated vmware nodes CentOS 7.5.1804

I have 7 nodes in total that form a hazelcast cluster. 5 of them are CP members. The groups size is set to 3. Other notable config changes: setSessionTimeToLiveSeconds(200), setMissingCPMemberAutoRemovalSeconds(240).

Usage: I have 8 clients that try to acquire the fenced lock for a Long key, the interface is cached for 5 minutes. The keys are almost always different, but in some cases we receive many actions for the same key and we want to avoid async update for these keys. In the past I used IMap for this but I noticed that it not always respected the lock. Fenced lock seems to perform very well until it reaches a certain memory usage.

Issue: What I observed is that one cp member is used heavily and the G1 Old Gen heap is ever so slightly increases over time. In one day it reached 5GB. And even after I stop the requests for lock the memory is never cleared.

I tried to create two groups to utilize more nodes in the cluster, it did help, but the issue persists, the used heap on a few nodes increases over time and at point they become very slow.

Question: What am I doing wrong? Is this even a valid use case for this tool?

Answer 1

Lock objects do not expire on their own and Hazelcast does not automatically perform garbage collection on unused locks. If you no longer need a lock, you can destroy it using lock.destroy(), otherwise you run the risk of running into OOME.