How to send mouse click event to a window in mac osx

Question

Could anyone give me any idea on how to send mouse click event to a hidden (not displaying in the foreground) window in mac osx? I'm trying to use pyobjc or pyautogui and really new to this scenario. Any keyword or idea would be appreciated. Thanks!

Answer 1

EDIT: There's an update at the end based on more investigation, but the short answer appears to be that this isn't generally possible in OSX.

Disclaimer: this isn't really an answer. I was about to post roughly the same question, but then I found this one. Rather than ask the same question, I thought I'd contribute with some comments about what I'd already found/tried. I'm new to contributing on SO, though, so I don't have enough reputation to just leave a comment, so I'm commenting here in an "answer". So it's not really an answer, but hopefully it helps you or someone else get a little closer.

Sort-of Answer:

AFAICT, the "right" way to do send a mouse event to just one application in OSX is to use the Core Graphics CGEventPostToPSN function.

Getting the "PSN" (Process Serial Number) isn't super straightforward, though, and all the methods people used to use to do so are deprecated. There's a replacement function called CGEventPostToPid, which uses the standard *nix process ID to specify the target application.

I've successfully posted keyboard events to background applications with this function, but not mouse events.

For example, this sends a character to whatever app you specify via PID:

pid = 1234  # get/input a real pid from somewhere for the target app.
type_a_key_down_event = Quartz.CGEventCreateKeyboardEvent(objc.NULL, 0, True)
type_a_key_up_event = Quartz.CGEventCreateKeyboardEvent(objc.NULL, 0, False)

Quartz.CGEventPostToPid(pid, type_a_key_down_event)
Quartz.CGEventPostToPid(pid, type_a_key_up_event)

(docs for the keyboard event creation: https://developer.apple.com/reference/coregraphics/1456564-cgeventcreatekeyboardevent?language=objc)

However, this does not send a click to the target app:

pid = 1234  # get/input a real pid from somewhere for the target app.
point = Quartz.CGPoint()
point.x = 100  # get a target x from somewhere
point.y = 100  # likewise, your target y from somewhere
left_mouse_down_event = Quartz.CGEventCreateMouseEvent(objc.NULL, Quartz.kCGEventLeftMouseDown, point, Quartz.kCGMouseButtonLeft)
left_mouse_up_event = Quartz.CGEventCreateMouseEvent(objc.NULL, Quartz.kCGEventLeftMouseUp, point, Quartz.kCGMouseButtonLeft)

Quartz.CGEventPostToPid(pid, left_mouse_down_event)
Quartz.CGEventPostToPid(pid, left_mouse_up_event)

Some SO answers for related questions have recommended that you use CGEventPost for the mouse events:

Quartz.CGEventPost(Quartz.kCGHIDEventTap, left_mouse_down_event)
Quartz.CGEventPost(Quartz.kCGHIDEventTap, left_mouse_up_event)

This what I think pyautogui is doing, because it successfully clicks the mouse at the requested coordinates, but it does so by moving and clicking the global mouse. Running that code will move the mouse to the point on the screen and perform a click on whatever's at that point, and then leave your mouse there. This is not what I want, and I don't think it's what you want.

What I want is the behavior described in CGEventPostToPSN or its more modern counterpart CGEventPostToPid: an input event posted just to the target application, whether it's in the foreground or not, which doesn't steal focus or change the actual mouse location.

The closest I've come is by adapting some code from Simulating mouse-down event on another window from CGWindowListCreate

That recommends using an NSEvent instead of a CGEvent. I've tried this with objective-C and with pyobjc, with limited success. The objective-C version I tried did nothing for me, so I won't post it. The pyobjc version allowed me to click on iTerm from Terminal or vice-versa, but I could not make the clicks go through to other apps. I tried Chrome, Console, and a few others with no luck at all.

Here's the partially working pyobjc code:

#! /usr/bin/env python
"""
CLI for sending mouse clicks to OSX apps.

Author: toejough
License: MIT
"""


# [ Imports ]
import Quartz
import fire
import AppKit


# [ API ]
def click(*, x, y, window_id, pid):
    """Click at x, y in the app with the pid."""
    point = Quartz.CGPoint()
    point.x = x
    point.y = y

    event_types = (
        AppKit.NSEventTypeMouseMoved,
        AppKit.NSEventTypeLeftMouseDown,
        AppKit.NSEventTypeLeftMouseUp,
    )

    for this_event_type in event_types:
        event = _create_ns_mouse_event(this_event_type, point=point, window_id=window_id)
        Quartz.CGEventPostToPid(pid, event)


# [ Internal ]
def _create_ns_mouse_event(event_type, *, point, window_id=None):
    """Create a mouse event."""
    create_ns_mouse_event = AppKit.NSEvent.mouseEventWithType_location_modifierFlags_timestamp_windowNumber_context_eventNumber_clickCount_pressure_
    ns_event = create_ns_mouse_event(
        event_type,  # Event type
        point,  # Window-specific coordinate
        0,  # Flags
        AppKit.NSProcessInfo.processInfo().systemUptime(),  # time of the event since system boot
        window_id,  # window ID the event is targeted to
        None,  # display graphics context.
        0,  # event number
        1,  # the number of mouse clicks associated with the event.
        0  # pressure applied to the input device, from 0.0 to 1.0
    )
    return ns_event.CGEvent()


# [ Script ]
if __name__ == "__main__":
    fire.Fire(click)

I tried playing with flags, event #'s, time, etc, with no luck. No matter how I changed those, the best I've been able to achieve is sending clicks from iTerm to Terminal or from Terminal to iTerm. Not sure what it is about terminal apps that makes them work and not other apps.

You can get the window_id from CGWindowListCopyWindowInfo and related functions documented at https://developer.apple.com/reference/coregraphics/quartz_window_services?language=objc

If someone has a better answer, a real answer, for all apps, please post and let us know. Or if you have an explanation/clue about why the clicks don't work, post that.

The only other possibly interesting bit of evidence I've collected is that whenever I run the above script, I get the following in my syslog:

4/21/17 4:16:08.284 PM launchservicesd[80]: SecTaskLoadEntitlements failed error=22
4/21/17 4:16:08.288 PM launchservicesd[80]: SecTaskLoadEntitlements failed error=22
4/21/17 4:16:08.295 PM tccd[21292]: SecTaskLoadEntitlements failed error=22

Might be red herring, though - I get those errors both for the successful case (sending clicks to different terminal app) and the failure case (sending clicks to a non-terminal app).

Hopefully this helps you or some other reader get closer to the real solution - if so, please post back!

UPDATE: I think the events work for terminal apps and not in general as a consequence of osx's built-in anti-focus-follows-mouse paradigm. You can send events, but unless the app is active (or allows FFM), the OS blocks it.

Apps that allow FFM (like iTerm and Terminal) can receive the ns_event variant above, and this is why I could make them work. Other apps will receive the mouse events, too (without moving the global mouse cursor!) as long as they are focused first.

You can see this for yourself by inserting a sleep into the above program, or at the terminal, and manually bring the target app to the front, and see the mouse events go through.

AFAICT, the only way to make the mouse events work generally would be to circumvent the OS's FFM filter, and I don't know of any way to do that. If someone finds out how, let me know!

More on FFM and OSX here.