How to handle :java.util.concurrent.TimeoutException: android.os.BinderProxy.finalize() timed out after 10 seconds errors?

Question

Full disclosure - I'm the author of the previously mentioned talk in TLV DroidCon.

I had a chance to examine this issue across many Android applications, and discuss it with other developers who encountered it - and we all got to the same point: this issue cannot be avoided, only minimized.

I took a closer look at the default implementation of the Android Garbage collector code, to understand better why this exception is thrown and on what could be the possible causes. I even found a possible root cause during experimentation.

The root of the problem is at the point a device "Goes to Sleep" for a while - this means that the OS has decided to lower the battery consumption by stopping most User Land processes for a while, and turning Screen off, reducing CPU cycles, etc. The way this is done - is on a Linux system level where the processes are Paused mid run. This can happen at any time during normal Application execution, but it will stop at a Native system call, as the context switching is done on the kernel level. So - this is where the Dalvik GC joins the story.

The Dalvik GC code (as implemented in the Dalvik project in the AOSP site) is not a complicated piece of code. The basic way it work is covered in my DroidCon slides. What I did not cover is the basic GC loop - at the point where the collector has a list of Objects to finalize (and destroy). The loop logic at the base can be simplified like this:

1. take starting_timestamp,
2. remove object for list of objects to release,
3. release object - finalize() and call native destroy() if required,
4. take end_timestamp,
5. calculate (end_timestamp - starting_timestamp) and compare against a hard coded timeout value of 10 seconds,
6. if timeout has reached - throw the java.util.concurrent.TimeoutException and kill the process.

Now consider the following scenario:

Application runs along doing its thing.

This is not a user facing application, it runs in the background.

During this background operation, objects are created, used and need to be collected to release memory.

Application does not bother with a WakeLock - as this will affect the battery adversely, and seems unnecessary.

This means the Application will invoke the GC from time to time.

Normally the GC runs is completed without a hitch.

Sometimes (very rarely) the system will decide to sleep in the middle of the GC run.

This will happen if you run your application long enough, and monitor the Dalvik memory logs closely.

Now - consider the timestamp logic of the basic GC loop - it is possible for the device to start the run, take a start_stamp, and go to sleep at the destroy() native call on a system object.

When it wakes up and resumes the run, the destroy() will finish, and the next end_stamp will be the time it took the destroy() call + the sleep time.

If the sleep time was long (more than 10 seconds), the java.util.concurrent.TimeoutException will be thrown.

I have seen this in the graphs generated from the analysis python script - for Android System Applications, not just my own monitored apps.

Collect enough logs and you will eventually see it.

Bottom line:

The issue cannot be avoided - you will encounter it if your app runs in the background.

You can mitigate by taking a WakeLock, and prevent the device from sleeping, but that is a different story altogether, and a new headache, and maybe another talk in another con.

You can minimize the problem by reducing GC calls - making the scenario less likely (tips are in the slides).

I have not yet had the chance to go over the Dalvik 2 (a.k.a ART) GC code - which boasts a new Generational Compacting feature, or performed any experiments on an Android Lollipop.

Added 7/5/2015:

After reviewing the Crash reports aggregation for this crash type, it looks like these crashes from version 5.0+ of Android OS (Lollipop with ART) only account for 0.5% of this crash type. This means that the ART GC changes has reduced the frequency of these crashes.

Added 6/1/2016:

Looks like the Android project has added a lot of info on how the GC works in Dalvik 2.0 (a.k.a ART).

You can read about it here - Debugging ART Garbage Collection.

It also discusses some tools to get information on the GC behavior for your app.

Sending a SIGQUIT to your app process will essentially cause an ANR, and dump the application state to a log file for analysis.

We see this constantly, all over our app, using Crashlytics. The crash usually happens way down in platform code. A small sampling:

android.database.CursorWindow.finalize() timed out after 10 seconds

java.util.regex.Matcher.finalize() timed out after 10 seconds

android.graphics.Bitmap$BitmapFinalizer.finalize() timed out after 10 seconds

org.apache.http.impl.conn.SingleClientConnManager.finalize() timed out after 10 seconds

java.util.concurrent.ThreadPoolExecutor.finalize() timed out after 10 seconds

android.os.BinderProxy.finalize() timed out after 10 seconds

android.graphics.Path.finalize() timed out after 10 seconds

The devices on which this happens are overwhelmingly (but not exclusively) devices manufactured by Samsung. That could just mean that most of our users are using Samsung devices; alternately it could indicate a problem with Samsung devices. I'm not really sure.

I suppose this doesn't really answer your questions, but I just wanted to reinforce that this seems quite common, and is not specific to your application.

I found some slides about this issue.

http://de.slideshare.net/DroidConTLV/android-crash-analysis-and-the-dalvik-garbage-collector-tools-and-tips

In this slides the author tells that it seems to be a problem with GC, if there are a lot of objects or huge objects in heap. The slide also include a reference to a sample app and a python script to analyze this issue.

https://github.com/oba2cat3/GCTest

https://github.com/oba2cat3/logcat2memorygraph

Furthermore I found a hint in comment #3 on this side: https://code.google.com/p/android/issues/detail?id=53418#c3

We solved the problem by stopping the FinalizerWatchdogDaemon.

public static void fix() {
    try {
        Class clazz = Class.forName("java.lang.Daemons$FinalizerWatchdogDaemon");

        Method method = clazz.getSuperclass().getDeclaredMethod("stop");
        method.setAccessible(true);

        Field field = clazz.getDeclaredField("INSTANCE");
        field.setAccessible(true);

        method.invoke(field.get(null));

    }
    catch (Throwable e) {
        e.printStackTrace();
    }
}

You can call the method in Application's lifecycle, like attachBaseContext(). For the same reason, you also can specific the phone's manufacture to fix the problem, it's up to you.

Here is an effective solution from didi to solve this problem, Since this bug is very common and difficult to find the cause, It looks more like a system problem, Why can't we ignore it directly？Of course we can ignore it, Here is the sample code:

final Thread.UncaughtExceptionHandler defaultUncaughtExceptionHandler = 
        Thread.getDefaultUncaughtExceptionHandler();
Thread.setDefaultUncaughtExceptionHandler(new Thread.UncaughtExceptionHandler() {
    @Override
    public void uncaughtException(Thread t, Throwable e) {
        if (t.getName().equals("FinalizerWatchdogDaemon") && e instanceof TimeoutException) {
        } else {
            defaultUncaughtExceptionHandler.uncaughtException(t, e);
        }
    }
});

By setting a special default uncaught exception handler, application can change the way in which uncaught exceptions are handled for those threads that would already accept whatever default behavior the system provided. When an uncaught TimeoutException is thrown from a thread named FinalizerWatchdogDaemon, this special handler will block the handler chain, the system handler will not be called, so crash will be avoided.

Through practice, no other bad effects were found. The GC system is still working, timeouts are alleviated as CPU usage decreases.

For more details see: https://mp.weixin.qq.com/s/uFcFYO2GtWWiblotem2bGg

Broadcast Receivers timeout after 10 seconds. Possibly your doing an asynchronous call (wrong) from a broadcast receiver and 4.3 actually detects it.