Workaround for Java bug which causes crash dump

Question

A program that I've developed is crashing the JVM occasionally due to this bug: http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8029516. Unfortunately the bug has not been resolved by Oracle and the bug report says that there are no known workarounds.

I've tried to modify the example code from the bug report by calling .register(sWatchService, eventKinds) in the KeyWatcher thread instead, by adding all pending register request to a list that I loop through in the KeyWatcher thread but it's still crashing. I'm guessing this just had the same effect as synchronizing on sWatchService (like the submitter of the bug report tried).

Can you think of any way to get around this?

Answer 1

From comments:

It appears that we have an issue with I/O cancellation when there is a pending ReadDirectoryChangesW outstanding.

The statement and example code indicate that the bug is triggered when:

1. There is a pending event that has not been consumed (it may or may not be visible to WatchService.poll() or WatchService.take())
2. WatchKey.cancel() is called on the key

This is a nasty bug with no universal workaround. The approach depends on the specifics of your application. Consider pooling watches to a single place so you don't need to call WatchKey.cancel(). If at one point the pool becomes too large, close the entire WatchService and start over. Something similar to.

public class FileWatcerService {
    static Kind<?>[] allEvents = new Kind<?>[] {
        StandardWatchEventKinds.ENTRY_CREATE,
        StandardWatchEventKinds.ENTRY_DELETE,
        StandardWatchEventKinds.ENTRY_MODIFY
    };

    WatchService ws;

    // Keep track of paths and registered listeners
    Map<String, List<FileChangeListener>> listeners = new ConcurrentHashMap<String, List<FileChangeListener>>();
    Map<WatchKey, String> keys = new ConcurrentHashMap<WatchKey, String>();

    boolean toStop = false;

    public interface FileChangeListener {
        void onChange();
    }

    public void addFileChangeListener(String path, FileChangeListener l) {
        if(!listeners.containsKey(path)) {
            listeners.put(path, new ArrayList<FileChangeListener>());
            keys.put(Paths.get(path).register(ws, allEvents), path);
        }
        listeners.get(path).add(l);
    }

    public void removeFileChangeListener(String path, FileChangeListener l) {
        if(listeners.containsKey(path))
            listeners.get(path).remove(l);
    }

    public void start() {
        ws = FileSystems.getDefault().newWatchService();
        new Thread(new Runnable() {
            public void run() {
                while(!toStop) {
                    WatchKey key = ws.take();
                    for(FileChangeListener l: listeners.get(keys.get(key)))
                        l.onChange();
                }
            }
        }).start();
    }

    public void stop() {
        toStop = true;
        ws.close();
    }
}

Answer 2

I've managed to create a workaround though it's somewhat ugly.

The bug is in JDK method WindowsWatchKey.invalidate() that releases native buffer while the subsequent calls may still access it. This one-liner fixes the problem by delaying buffer clean-up until GC.

Here is a compiled patch to JDK. In order to apply it add the following Java command-line flag:
-Xbootclasspath/p:jdk-8029516-patch.jar

If patching JDK is not an option in your case, there is still a workaround on the application level. It relies on the knowledge of Windows WatchService internal implementation.

public class JDK_8029516 {
    private static final Field bufferField = getField("sun.nio.fs.WindowsWatchService$WindowsWatchKey", "buffer");
    private static final Field cleanerField = getField("sun.nio.fs.NativeBuffer", "cleaner");
    private static final Cleaner dummyCleaner = Cleaner.create(Thread.class, new Thread());

    private static Field getField(String className, String fieldName) {
        try {
            Field f = Class.forName(className).getDeclaredField(fieldName);
            f.setAccessible(true);
            return f;
        } catch (Exception e) {
            throw new IllegalStateException(e);
        }
    }

    public static void patch(WatchKey key) {
        try {
            cleanerField.set(bufferField.get(key), dummyCleaner);
        } catch (IllegalAccessException e) {
            throw new IllegalStateException(e);
        }
    }
}

Call JDK_8029516.patch(watchKey) right after the key is registred, and it will prevent watchKey.cancel() from releasing the native buffer prematurely.

Answer 3

You might not be able to work around the problem itself but you could deal with the error and handle it. I don't know your specific situation but I could imagine the biggest issue is the crash of the whole JVM. Putting all in a try block does not work because you cannot catch a JVM crash.

Not knowing more about your project makes it difficult to suggest a good/acceptable solution, but maybe this could be an option: Do all the file watching stuff in a separate JVM process. From your main process start a new JVM (e.g. using ProcessBuilder.start()). When the process terminates (i.e. the newly started JVM crashes), restart it. Obviously you need to be able to recover, i.e. you need to keep track of what files to watch and you need to keep this data in your main process too.

Now the biggest remaining part is to implement some communication between the main process and the file watching process. This could be done using standard input/output of the file watching process or using a Socket/ServerSocket or some other mechanism.