Java thread leaks when calling back from native thread via JNI

Question

Summary: I am seeing Java thread leaks when calling back into Java from native code on a natively-created thread.

(Update 11 Feb 2014: We raised this as a support request with Oracle. It has now been confirmed by Oracle on Java 7 update 45. It only affects 64-bit Linux (and possibly Mac) platforms: 32-bit Linux is unaffected).

(Update 29 April 2014: Oracle have a fix for this issue, and it will be released in Java 7 update 80).

I have an application consisting of a Java layer and a native library. The Java layer calls into the native library via JNI: this then causes a new native thread to start running, which calls back into Java. Because the new native thread is not attached to the JVM, it needs to be attached prior to doing the callback, then detached afterwards. The usual way to do this seems to be to bracket the code that calls back into Java with AttachCurrentThread / DetachCurrentThread calls. This works fine, but for our application (which calls back into Java very frequently) the overhead of attaching and detaching every time is significant.

There is an optimization described in several places (like here and here) that recommends using mechanisms based on thread local storage to eliminate this problem: essentially every time the native callback is triggered, the thread is tested to see if it is already attached to the JVM: if not, it is attached to the JVM and the thread local storage mechanism is used to automatically detach the thread when it exits. I have implemented this, but although the attach and detach appears to be occurring correctly, this causes a leak of threads on the Java side. I believe I am doing everything correctly and am struggling to see what might be wrong. I have been bashing my head on this for a while now and I would be very grateful for any insights.

I have recreated the problem in cut-down form. Below is the code for the native layer. What we have here is a wrapper that encapsulates the process of returning a JNIEnv pointer for the current thread, using the POSIX thread-local storage mechanism to automatically detach the thread if it wasn't already attached. There is a callback class that acts as a proxy for the Java callback method. (I have used callback to a static Java method in order to eliminate the extra complication of creating and deleting global object references to the Java object, which are irrelevant to this problem). Finally there is a JNI method that when called, constructs a callback, and creates a new native thread and waits for it to complete. This newly created thread calls the callback once then exits.

#include <jni.h> #include <iostream> #include <pthread.h>   using namespace std;   /// Class to automatically handle getting thread-specific JNIEnv instance, /// and detaching it when no longer required class JEnvWrapper {  public:      static JEnvWrapper &getInstance()     {         static JEnvWrapper wrapper;         return wrapper;     }      JNIEnv* getEnv(JavaVM *jvm)     {         JNIEnv *env = 0;         jint result = jvm->GetEnv((void **) &env, JNI_VERSION_1_6);         if (result != JNI_OK)         {             result = jvm->AttachCurrentThread((void **) &env, NULL);             if (result != JNI_OK)             {                 cout << "Failed to attach current thread " << pthread_self() << endl;             }             else             {                 cout << "Successfully attached native thread " << pthread_self() << endl;             }              // ...and register for detach when thread exits             int result = pthread_setspecific(key, (void *) env);             if (result != 0)             {                 cout << "Problem registering for detach" << endl;             }             else             {                 cout << "Successfully registered for detach" << endl;             }         }          return env;     }  private:      JEnvWrapper()     {         // Initialize the key         pthread_once(&key_once, make_key);     }      static void make_key()     {         pthread_key_create(&key, detachThread);     }       static void detachThread(void *p)     {         if (p != 0)         {             JavaVM *jvm = 0;             JNIEnv *env = (JNIEnv *) p;             env->GetJavaVM(&jvm);             jint result = jvm->DetachCurrentThread();             if (result != JNI_OK)             {                 cout << "Failed to detach current thread " << pthread_self() << endl;             }             else             {                 cout << "Successfully detached native thread " << pthread_self() << endl;             }          }     }       static pthread_key_t key;     static pthread_once_t key_once; };  pthread_key_t JEnvWrapper::key; pthread_once_t JEnvWrapper::key_once = PTHREAD_ONCE_INIT;    class Callback {  public:      Callback(JNIEnv *env, jobject callback_object)     {         cout << "Constructing callback" << endl;         const char *method_name = "javaCallback";         const char *method_sig = "(J)V";          env->GetJavaVM(&m_jvm);          m_callback_class = env->GetObjectClass(callback_object);         m_methodID = env->GetStaticMethodID(m_callback_class, method_name, method_sig);         if (m_methodID == 0)         {             cout << "Couldn't get method id" << endl;         }     }      ~Callback()     {         cout << "Deleting callback" << endl;     }      void callback()     {         JNIEnv *env = JEnvWrapper::getInstance().getEnv(m_jvm);         env->CallStaticVoidMethod(m_callback_class, m_methodID, (jlong) pthread_self());     }  private:      jclass m_callback_class;     jmethodID m_methodID;     JavaVM *m_jvm; };    void *do_callback(void *p) {     Callback *callback = (Callback *) p;     callback->callback();     pthread_exit(NULL); }     extern "C" {  JNIEXPORT void JNICALL Java_com_test_callback_CallbackTest_CallbackMultiThread(JNIEnv *env, jobject obj) {     Callback callback(env, obj);     pthread_t thread;     pthread_attr_t attr;     void *status;     int rc;      pthread_attr_init(&attr);     pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_JOINABLE);     rc = pthread_create(&thread, &attr, do_callback, (void *) &callback);     pthread_attr_destroy(&attr);     if (rc)     {         cout << "Error creating thread: " << rc << endl;     }     else     {         rc = pthread_join(thread, &status);         if (rc)         {             cout << "Error returning from join " << rc << endl;         }     } } 

The Java code is very simple: it just repeatedly calls the native method in a loop:

package com.test.callback;  public class CallbackTest {      static     {         System.loadLibrary("Native");     }      public void runTest_MultiThreaded(int trials)     {         for (int trial = 0; trial < trials; trial++)         {             // Call back from this thread             CallbackMultiThread();              try {                 Thread.sleep(200);             } catch (InterruptedException e) {                 // TODO Auto-generated catch block                 e.printStackTrace();             }         }     }      static void javaCallback(long nativeThread)     {         System.out.println("Java callback: native thread: " + nativeThread + ", java thread: " + Thread.currentThread().getName() + ", " + Thread.activeCount() + " active threads");     }      native void CallbackMultiThread();   } 

Below is some sample output from this test: you can see that although the native layer is reporting that the native thread is successfully being attached and detached, every time the callback is triggered a new Java thread is created:

Constructing callback Successfully attached native thread 140503373506304 Successfully registered for detach Java callback: native thread: 140503373506304, java thread: Thread-67, 69 active threads Successfully detached native thread 140503373506304 Deleting callback Constructing callback Successfully attached native thread 140503373506304 Successfully registered for detach Java callback: native thread: 140503373506304, java thread: Thread-68, 70 active threads Successfully detached native thread 140503373506304 Deleting callback Constructing callback Successfully attached native thread 140503373506304 Successfully registered for detach Java callback: native thread: 140503373506304, java thread: Thread-69, 71 active threads Successfully detached native thread 140503373506304 Deleting callback Constructing callback Successfully attached native thread 140503373506304 Successfully registered for detach Java callback: native thread: 140503373506304, java thread: Thread-70, 72 active threads Successfully detached native thread 140503373506304 Deleting callback Constructing callback Successfully attached native thread 140503373506304 Successfully registered for detach Java callback: native thread: 140503373506304, java thread: Thread-71, 73 active threads Successfully detached native thread 140503373506304 Deleting callback 

Just to add: the development platform I am using is CentOS 6.3 (64-bit). The Java version is the Oracle distribution version 1.7.0_45, although the problem also shows with the OpenJDK distribution, versions 1.7 and 1.6.

Answer 1

Oracle have fixed this issue with the JVM, and it will be released in Java 7 update 80.

Well if you aren't going to accept your own answer maybe you'll accept this one. At least it won't be pulling in as much traffic for a zero answers question any more.