JavaFX Minimizing Undecorated Stage

Question

I have an undecorated JavaFX Stage, and my own minimize, maximize & close buttons. But unfortunately clicking the taskbar icon in Windows 7 does not automatically minimize the stage - compared to the decorated behaviour.

Is there a way to minimize an undecorated stage with pure Java code, by clicking the taskbar icon? If not how can I do this with, say, JNA?

EDIT: OK, I've been trying to solve this with JNA, but having done next to none C/C++/JNA, I have a bit trouble setting this up. I'd be grateful if someone helped me to put the pieces together..

Here's my code so far:

public final class Utils {

   static {
    if (PlatformUtil.isWin7OrLater()) {
            Native.register("shell32");
            Native.register("user32");
        }
    }

    // Apparently, this is the event I am after
    public static final int WM_ACTIVATEAPP = 0x1C;


    public static void registerMinimizeHandler(Stage stage) {
       // Hacky way to get a pointer to JavaFX Window
       Pointer pointer = getWindowPointer(stage);

       WinDef.HWND hwnd = new WinDef.HWND(pointer);

       // Here's my minimize/activate handler
       WinUser.WindowProc windowProc = new MinimizeHandler(stage);

       Pointer magicPointer = ... set this to point to windowProc?

       // This.. apparently, re-sets the WndProc? But how do I get the "magicPointer" that is "attached" to the windowProc?
       User32.INSTANCE.SetWindowLong(hwnd, User32.GWL_WNDPROC, magicPointer);
    }
}

 private static class MinimizeHandler implements WinUser.WindowProc {

    private Stage stage;

    private MinimizeHandler(Stage stage) {
        this.stage = stage;
    }

    @Override
    public WinDef.LRESULT callback(WinDef.HWND hWnd, int uMsg, WinDef.WPARAM wParam, WinDef.LPARAM lParam) {
        if (uMsg == WM_ACTIVATEAPP) {
            System.out.println("ACTIVATE");
        }
        return User32.INSTANCE.DefWindowProc(hWnd, uMsg, wParam, lParam);
    }
}

private static Pointer getWindowPointer(Stage stage) {
    try {
        TKStage tkStage = stage.impl_getPeer();
        Method getPlatformWindow = tkStage.getClass().getDeclaredMethod("getPlatformWindow" );
        getPlatformWindow.setAccessible(true);
        Object platformWindow = getPlatformWindow.invoke(tkStage);
        Method getNativeHandle = platformWindow.getClass().getMethod( "getNativeHandle" );
        getNativeHandle.setAccessible(true);
        Object nativeHandle = getNativeHandle.invoke(platformWindow);
        return new Pointer((Long) nativeHandle);
    } catch (Throwable e) {
        System.err.println("Error getting Window Pointer");
        return null;
    }
}

EDIT 2: I eventually got further on with this one, but as soon as I re-set the WNDPROC, my undecorated window didn't respond to any events.. I'm offering a bounty of 100 reputation for a self-contained example with a working solution. Windows (7+) only is OK, I do not even know how this behaves on other platforms.

EDIT 3: Well, I kind of gave up with this one.. I got everything set up correctly, and received the events, but had problems figuring out the correct event to listen for..

Since there's been some interest in the question, if anyone wants to attempt to continue with this, here's my final code (it hopefully should "work" out-of-box):

public final class Utils {

    static interface ExtUser32 extends StdCallLibrary, User32 {
        ExtUser32 INSTANCE = (ExtUser32) Native.loadLibrary(
                        "user32",
                        ExtUser32.class,
                        W32APIOptions.DEFAULT_OPTIONS);

        WinDef.LRESULT CallWindowProcW(
                        Pointer lpWndProc,
                        Pointer hWnd,
                        int msg,
                        WinDef.WPARAM wParam,
                        WinDef.LPARAM lParam);

        int SetWindowLong(HWND hWnd, int nIndex, com.sun.jna.Callback wndProc) throws LastErrorException;
    }

    // Some possible event types
    public static final int WM_ACTIVATE = 0x0006;
    public static final int WM_ACTIVATEAPP = 0x1C;
    public static final int WM_NCACTIVATE = 0x0086;

    public static void registerMinimizeHandler(Stage stage) {
        Pointer pointer = getWindowPointer(stage);
        WinDef.HWND hwnd = new WinDef.HWND(pointer);
        long old = ExtUser32.INSTANCE.GetWindowLong(hwnd, User32.GWL_WNDPROC);
        MinimizeHandler handler = new MinimizeHandler(stage, old);
        ExtUser32.INSTANCE.SetWindowLong(hwnd, User32.GWL_WNDPROC, handler);
    }

    private static Pointer getWindowPointer(Stage stage) {
    try {
        TKStage tkStage = stage.impl_getPeer();
        Method getPlatformWindow = tkStage.getClass().getDeclaredMethod("getPlatformWindow" );
        getPlatformWindow.setAccessible(true);
        Object platformWindow = getPlatformWindow.invoke(tkStage);
        Method getNativeHandle = platformWindow.getClass().getMethod( "getNativeHandle" );
        getNativeHandle.setAccessible(true);
        Object nativeHandle = getNativeHandle.invoke(platformWindow);
        return new Pointer((Long) nativeHandle);
    } catch (Throwable e) {
        System.err.println("Error getting Window Pointer");
        return null;
    }
}

    private static class MinimizeHandler implements WinUser.WindowProc, StdCallLibrary.StdCallCallback {

        private Pointer mPrevWndProc32;

        private Stage stage;

        private MinimizeHandler(Stage stage, long oldPtr) {
            this.stage = stage;

            mPrevWndProc32 = new Pointer(oldPtr);

            // Set up an event pump to deliver messages for JavaFX to handle
            Thread thread = new Thread() {
                @Override
                public void run() {
                    int result;
                    WinUser.MSG msg = new WinUser.MSG();
                    while ((result = User32.INSTANCE.GetMessage(msg, null, 0, 0)) != 0) {
                        if (result == -1) {
                            System.err.println("error in get message");
                            break;
                        }
                        else {
                            System.out.println("got message: " + result);
                            User32.INSTANCE.TranslateMessage(msg);
                            User32.INSTANCE.DispatchMessage(msg);
                        }
                    }
                }
            };
            thread.start();
        }

        @Override
        public WinDef.LRESULT callback(WinDef.HWND hWnd, int uMsg, WinDef.WPARAM wParam, WinDef.LPARAM lParam) {

            if (uMsg == WM_ACTIVATEAPP) {
                // Window deactivated (wParam == 0)... Here's where I got stuck and gave up,
                // this is probably not the best event to listen to.. the app
                // does indeed get iconified now by pressing the task-bar button, but it
                // instantly restores afterwards..
                if (wParam.intValue() == 0) {
                    stage.setIconified(true);
                }
                return new WinDef.LRESULT(0);
            }

            // Let JavaFX handle other events
            return ExtUser32.INSTANCE.CallWindowProcW(
                            mPrevWndProc32,
                            hWnd.getPointer(),
                            uMsg,
                            wParam,
                            lParam);
        }
    }
}

Answer 1

You can just set the appropriate window style. It works in XP but should be ok in windows 7 32 bit. I think (but can't test) if you use 64 bit then change to the Ptr windows functions, ie. GetWindowLongPtr.

import com.sun.jna.Native;
import com.sun.jna.Pointer;
import com.sun.jna.platform.win32.User32;
import com.sun.jna.platform.win32.WinDef.HWND;
import com.sun.jna.platform.win32.WinUser;
import static com.sun.jna.platform.win32.WinUser.GWL_STYLE;
import javafx.application.Application;
import javafx.scene.Scene;
import javafx.scene.control.TextArea;
import javafx.scene.layout.VBox;
import javafx.stage.Stage;

public class JNATest extends Application {
    public static void main(String[] args) { launch(args); }

    @Override
    public void start(Stage stage) {
        TextArea ta = new TextArea("output\n");
        VBox root = new VBox(5,ta);
        Scene scene = new Scene(root,800,200);
        stage.setTitle("Find this window");
        stage.setScene(scene);
        stage.show();
        //gets this window (stage)
        long lhwnd = com.sun.glass.ui.Window.getWindows().get(0).getNativeWindow();
        Pointer lpVoid = new Pointer(lhwnd);
        //gets the foreground (focused) window
        final User32 user32 = User32.INSTANCE;
        char[] windowText = new char[512];
        HWND hwnd = user32.GetForegroundWindow();
        //see what the title is
        user32.GetWindowText(hwnd, windowText, 512);
        //user32.GetWindowText(new HWND(lpVoid), windowText, 512);//to use the hwnd from stage
        String text=(Native.toString(windowText));
        //see if it's the same pointer
        ta.appendText("HWND java:" + lpVoid + " HWND user32:"+hwnd+" text:"+text+"\n");
        //change the window style if it's the right title
        if (text.equals(stage.getTitle())){
            //the style to change 
            int WS_DLGFRAME = 0x00400000;//s/b long I think
            //not the same constant here??
            ta.appendText("windows api:"+WS_DLGFRAME+" JNA: "+WinUser.SM_CXDLGFRAME);
            int oldStyle = user32.GetWindowLong(hwnd, GWL_STYLE);
            int newStyle = oldStyle & ~0x00400000; //bitwise not WS_DLGFRAME means remove the style
            newStyle = newStyle & ~0x00040000;//WS_THICKFRAME   
            user32.SetWindowLong(hwnd, GWL_STYLE, newStyle);
        }
    }

}

My guess is you replace the last 3 lines with

            long oldStyleLong = user32.GetWindowLongPtr(hwnd, GWL_STYLE).longValue();
            long newStyleLong = oldStyleLong & ~ 0x00400000l;
            user32.SetWindowLongPtr(hwnd, GWL_STYLE, new BaseTSD.LONG_PTR(newStyleLong));

for 64 bit. I think I don't have those functions in my User32.dll, so I can't test it. There's lots of extraneous code in there, mainly for testing or teaching. Remove the unused lines once you figure out what you want to do.

ps. Don't add newStyle = newStyle & ~0x00020000;//WS_MINIMIZEBOX. That's one of the style flags JavaFX doesn't use for undecorated. That's why the minimize isn't available. Maybe if you try setting stage undecorated and adding (using |, not &~) the minimize box flag, you'll get the same result. There are tools to look up all the style flags from any window.

Here's the simplest amount of code that just changes an undecorated stage using the stage's HWND.

    public void start(Stage stage) {
        Scene scene = new Scene(new Pane(new Label("Hello World")));
        stage.initStyle(StageStyle.UNDECORATED);
        stage.setTitle("Find this window");
        stage.setScene(scene);
        stage.show();
        long lhwnd = com.sun.glass.ui.Window.getWindows().get(0).getNativeWindow();
        Pointer lpVoid = new Pointer(lhwnd);
        HWND hwnd = new HWND(lpVoid);
        final User32 user32 = User32.INSTANCE;
        int oldStyle = user32.GetWindowLong(hwnd, GWL_STYLE);
        System.out.println(Integer.toBinaryString(oldStyle));
        int newStyle = oldStyle | 0x00020000;//WS_MINIMIZEBOX
        System.out.println(Integer.toBinaryString(newStyle));
        user32.SetWindowLong(hwnd, GWL_STYLE, newStyle);
    }

It prints out the style flags before and after so you can look up what styles are set.

Answer 2

Two things to note here.

First, it doesn't look like these libraries are in the latest version of JNA, 5.50 as of now, adding from the Maven repository. I had to add the 4.2.1 library instead.

Second, you may encounter this exception, like I did on Windows 10 and Java 11: Error with package com.sun.glass.ui while learning Java Native Access

The solution is go to your VM options in your IDE (Run -> Edit Configurations..., in IntelliJ) and add this:

--add-exports
javafx.graphics/com.sun.glass.ui=ALL-UNNAMED

It should work after that.

I would like to see someone implement the native Windows animations for minimizing and un-minimizing an undecorated window, but I haven't searched too thoroughly yet to see if this has already been discussed. I'll update this if I come across a solution.

Edit:

Upon further research on the Windows animations, it looks like a solution could be hacked together, but I gave up at trying to implement this C# hack below. It seems to be more of an OS issue and not just JavaFX.

I was able to get the initial window to stay undecorated while minimizing and with the animation by modifying this in start():

int newStyle = oldStyle | 0x00020000 | 0x00C00000;

But, after minimizing and reopening, the Windows border appears oddly enough.

Then, I tried to use a ChangeListener to swap Windows styles when iconifying.

stage.iconifiedProperty().addListener(new ChangeListener<Boolean>() {

        @Override
        public void changed(ObservableValue<? extends Boolean> ov, Boolean t, Boolean t1) {
            if (t1.booleanValue() == true) {
                int newStyle = oldStyle | 0x00020000 | 0x00C00000;
                user32.SetWindowLong(hwnd, GWL_STYLE, newStyle);
            } else if (t1.booleanValue() == false) {
                int newStyle = oldStyle | 0x00020000;
                user32.SetWindowLong(hwnd, GWL_STYLE, newStyle);
            }
        }
    });

This successfully gets the windows un-minimize animation to work fine consistently, while leaving the (visible) stage borderless.

It looks like I can get minimization animations working once I find out the best way to re-apply:

int newStyle = oldStyle | 0x00020000 | 0x00C00000;
user32.SetWindowLong(hwnd, GWL_STYLE, newStyle);

just before the stage is iconified, and the border isn't visible to the user. Once implemented, this might work similarly to the C# solution in the first link below. Basically, what the above ChangeListener does in reverse.

Links to do with solving borderless/undecorated animations:

Use windows animations on borderless form

https://exceptionshub.com/borderless-window-using-areo-snap-shadow-minimize-animation-and-shake.html

http://pinvoke.net/default.aspx/Constants/Window%20styles.html

JavaFX Minimizing & Maximing undecorated stage with animations