ClipCursor succeeds, but effectively does nothing

Question

I'm writing a very simple program to clip the mouse to a specified window. It runs from the system tray without a visible window. Because there will be multiple instances of the same window, it uses EnumWindows() to iterate through every top-level window and compare their hwnd with GetForegroundWindow(). When true, it runs the standard ClipCursor() code. ClipCursor() returns TRUE, and, I've asserted that the RECT set by GetClipCursor() is the exact same as the RECT passed to ClipCursor(). Yet, the cursor is free to move anywhere on the screen.

I've checked that the values in the RECT are the exact values of the window, I've compiled the program in release mode and ran it with admin rights, still nothing. The code below is exactly what runs after we've found the HWND of the GetForegroundWindow():

// Get the window client area.
GetClientRect(hwnd, &rc);

// Convert the client area to screen coordinates.
POINT pt = { rc.left, rc.top };
POINT pt2 = { rc.right, rc.bottom };
ClientToScreen(hwnd, &pt);
ClientToScreen(hwnd, &pt2);
SetRect(&rc, pt.x, pt.y, pt2.x, pt2.y);

clipped = true;
ClipCursor(&rc);

RECT rect;
GetClipCursor(&rect);

assert(rect.bottom == rc.bottom);
assert(rect.left == rc.left);
assert(rect.right == rc.right);
assert(rect.top == rc.top);

I've removed a lot of the checks because they were getting annoying (I was using MessageBox()'s), but this code is certainly running when it's supposed to. The cursor just isn't getting clipped, and I can't fathom why.

Answer 1

Since the cursor is a shared resource, your attempt to clip it is overwritten by anybody else who calls ClipCursor to unclip the cursor. And a lot of operations automatically unclip the cursor (such as any focus change). It's considered poor form for a background window to change the cursor clip.

Answer 2

Well, it only took me a few days to find out that despite the shared nature of the Cursor resource, if a process is clipping a cursor in certain circumstances that I don't fully know of yet (maybe the process has to be the foreground application? or something like that... I'm not so sure.) then the OS automatically releases the cursor back to full clipping mode (or whatever you'd call it).

Anyway, the fix is to do a low level mouse hook and call the clipcursor from there. Here's some quick proof of concept code (tested and works, although I removed the irrelevant stuff like creating a window or setting up the systray etc):

// Some variables we'll use
bool clipped = false;   // Do we need to clip the mouse?
RECT rc;                // The clip rect
HHOOK hMouseHook;       // Low level mouse hook

// Low level mouse hook callback function
__declspec(dllexport) LRESULT CALLBACK MouseEvent(int nCode, WPARAM wParam, LPARAM lParam)
{
    // This part should be rewritten to make it not be a CPU-hog
    // But works as a proof of concept
    if ( clipped )
        ClipCursor(&rc);

    return CallNextHookEx(hMouseHook, nCode, wParam, lParam);
}

int WINAPI WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPSTR lpCmdLine, int nShowCmd)
{
    // .... Blah blah blah ....

    // Low level mouse hook
    hMouseHook = SetWindowsHookEx(WH_MOUSE_LL, (HOOKPROC)MouseEvent, hInstance, 0);

    // Only included to show that you set the hook before this,
    // And unhook after this.
    while(GetMessage(&msg, NULL, 0, 0) > 0)
    {
        TranslateMessage(&msg);
        DispatchMessage(&msg);
    }

    // Unhook the mouse
    UnhookWindowsHookEx(hMouseHook);

    return msg.wParam;
}