java using 100% cpu

Question

I have a java app running on CentOS 6.0. It always runs in background via cron. Sometimes this app goes into wait state while using 100% cpu.

My java version is :

java version "1.6.0_17"
OpenJDK Runtime Environment (IcedTea6 1.7.4) (rhel-1.21.b17.el6-x86_64)
OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode)

Other symptoms are :

a. One thread of the process seems to be in a loop waiting for something. When traced using strace, it shows following o/p continuously :

futex(0x7fb8000ac728, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7fb8000ac754, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1340347489,> 822867000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)

b. It seems like the process has finished working, looking at the files it is using. Only few files are remaining. The output of 'ls /proc/pid/fd/ shows :

lr-x------ 1 root root 64 Jun 22 13:13 0 -> pipe:[77107601]
l-wx------ 1 root root 64 Jun 22 13:13 1 -> pipe:[77120162]
l-wx------ 1 root root 64 Jun 22 13:13 2 -> /var/log/mithi/mcs/agent_account_mailstore_exceed_limit.sh.log
lr-x------ 1 root root 64 Jun 22 13:13 3 -> /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/rt.jar

Has anybody faced similar situation? Any clues or references would be very helpful.

More specifically, are there any known problems in running openjdk based Java processes in background on CentOS 6?

Now I'm able to simulate the problem with a very simple infinite loop, given below

#!/bin/bash

while [ 1 ]
do
    /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/bin/java -version &
    sleep 1s
done

When this script is run for about 3 - 4 hours, I find one or two java processes which are hung or in infinite loop with identical symptom i.e

futex(0x7fb8000ac754, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1340347489,> 822867000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)

This is happening on multiprocessor systems only, not on single processor systems. It can also be simulated on RHEL6, not only CentOS6.

Answer 1

There are many possible reasons. Those, that I can think of:

1. Your process' memory usage is very close to the maximum heap size, causing notorious Full GCs. Enable GC logs with -Xloggc:/path/to/logFile.log -XX:+PrintGCDetails options and then analyze it using tools like GCViewer or HPJmeter.

2. Your process is actually doing something (like a infinite loop), and you can check it by doing a few thread dumps and analyzing them. You can do that with VisualVM and Thread Dump Analyzer.

Answer 2

The problem was solved after upgrading the kernel to kernel-2.6.32-220 which is the CentOS 6.2 kernel.