NSMenuItem with custom view doesn't receive mouse events

Question

I'm working on a menu bar app, and I'm setting a custom view using NSMenuItem's view property.

The view displays ok, but I'm unable to receive any kind of mouse click events for menu items that have open submenus.

In this screenshot, I've added a button to each item. The 3 rightmost buttons function correctly, but the ones in the parent menus don't receive any click events at all.

I've tried a bunch of stuff, including:

• Trying to capture mouse events using the mouseUp and mouseDown methods
• Making the NSWindow for the custom view key when the mouse enters that view
• Adding global and local monitors for NSEvents

...but to no avail

Even without the approach of adding a button, I can't replicate the default behaviour of a standard NSMenuItem, as the target-action callback for the NSMenuItem doesn't get called if it has a custom view. (and I can't receive any click events to call it myself)

In theory this should be possible, because I am able to select menus that have open submenus using the default NSMenuItem (no custom view).

Is anybody able to help?

Thanks

Answer 1

I set up a test project like yours, with NSButtons as the view for the menu items, and saw the same behavior you were seeing. It is indeed intriguing. If you subclass NSApplication and override its -sendEvent: method, adding a log to see what events go through the mechanism, you find that -sendEvent: is never actually called when you click on any of the menu items, even the ones that do work. Isn't that weird? So the next thing to try is to subclass NSButton, add an override for -mouseDown:, and put a breakpoint there. Sure enough, the breakpoint is never hit for the item with the open submenu, but it is hit for the others. And when we do that, the backtrace is:

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
  * frame #0: 0x0000000100002fa0 menutest`MyButton.mouseDown(event=0x0000608000121900, self=0x0000600000140a50) at AppDelegate.swift:33
    frame #1: 0x000000010000303c menutest`@objc MyButton.mouseDown(with:) at AppDelegate.swift:0
    frame #2: 0x00007fffa9f6724f AppKit`-[NSWindow(NSEventRouting) _handleMouseDownEvent:isDelayedEvent:] + 6341
    frame #3: 0x00007fffa9f63a6c AppKit`-[NSWindow(NSEventRouting) _reallySendEvent:isDelayedEvent:] + 1942
    frame #4: 0x00007fffa9f62f0a AppKit`-[NSWindow(NSEventRouting) sendEvent:] + 541
    frame #5: 0x00007fffa9a2328d AppKit`-[NSCarbonWindow sendEvent:] + 118
    frame #6: 0x00007fffa9a20261 AppKit`NSMenuItemCarbonEventHandler + 10597
    frame #7: 0x00007fffab0acd85 HIToolbox`DispatchEventToHandlers(EventTargetRec*, OpaqueEventRef*, HandlerCallRec*) + 1708
    frame #8: 0x00007fffab0abff6 HIToolbox`SendEventToEventTargetInternal(OpaqueEventRef*, OpaqueEventTargetRef*, HandlerCallRec*) + 428
    frame #9: 0x00007fffab0c1d14 HIToolbox`SendEventToEventTarget + 40
    frame #10: 0x00007fffab0ea7df HIToolbox`ToolboxEventDispatcherHandler(OpaqueEventHandlerCallRef*, OpaqueEventRef*, void*) + 2503
    frame #11: 0x00007fffab0ad17a HIToolbox`DispatchEventToHandlers(EventTargetRec*, OpaqueEventRef*, HandlerCallRec*) + 2721
    frame #12: 0x00007fffab0abff6 HIToolbox`SendEventToEventTargetInternal(OpaqueEventRef*, OpaqueEventTargetRef*, HandlerCallRec*) + 428
    frame #13: 0x00007fffab0c1d14 HIToolbox`SendEventToEventTarget + 40
    frame #14: 0x00007fffab12e928 HIToolbox`IsUserStillTracking(MenuSelectData*, unsigned char*) + 1658
    frame #15: 0x00007fffab255dc4 HIToolbox`TrackMenuCommon(MenuSelectData&, unsigned char*, SelectionData*, MenuResult*, MenuResult*) + 1664
    frame #16: 0x00007fffab13a223 HIToolbox`MenuSelectCore(MenuData*, Point, double, unsigned int, OpaqueMenuRef**, unsigned short*) + 554
    frame #17: 0x00007fffab139f66 HIToolbox`_HandleMenuSelection2 + 460
    frame #18: 0x00007fffa97ee368 AppKit`_NSHandleCarbonMenuEvent + 239
    frame #19: 0x00007fffa9a68702 AppKit`_DPSEventHandledByCarbon + 54
    frame #20: 0x00007fffa9de90c5 AppKit`-[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 963
    frame #21: 0x00007fffa96623db AppKit`-[NSApplication run] + 926
    frame #22: 0x00007fffa962ce0e AppKit`NSApplicationMain + 1237
    frame #23: 0x00000001000035fd menutest`main at AppDelegate.swift:13
    frame #24: 0x00007fffc12fc235 libdyld.dylib`start + 1

As you can see, the events are not being dispatched through the Cocoa event dispatch mechanism because the menus are actually Carbon. That's right, many of those Carbon APIs and subsystems that were supposedly removed in the transition to 64-bit are actually still quite alive and well; they're just private API now. We can't use them in 64-bit mode, but Apple sure can, and the entire menu system is still implemented on top of the Carbon event model. Because it's okay for third-party developers to have to rewrite, say, Photoshop from scratch, but that menu handling code that somebody wrote in 1997 is way too valuable to just give up, I'm sure you agree.

Anyway, I did a little test by swizzling out -[NSCarbonWindow sendEvent:], the earliest Objective-C method in this backtrace (other than the very top-level stuff), to see if it was called at all when the submenu item was clicked, and it's not. So if I had to guess, I'd say the problem lies in the Carbon event handler. Well, this may be a bit of a pain in the rear end, but hey, no problem! We can work around this by dropping down to the Carbon level and installing our own Carbon event handler. All right, roll up your sleeves, let's do thi—

Oh, right.

We can't use those APIs in 64-bit mode.

Anyway, I sadly don't think there's going to be a way to get this to work short of using nasty hacks to use what are now private APIs like this guy did and risking future breakage (not to mention being instabanned from the App Store). Or doing something really crazy like monkeypatching one of those C functions in the backtrace, which is likely going to be even worse. This whole issue does seem worthy of a Radar report, though. Please file one with Apple and let them know about this problem, and maybe they'll fix it in some future release.

EDIT: There actually is a solution, sort of. Since a view attached to a menu item that doesn't have a submenu does receive the mouse events that you'd expect, you could forego setting submenu and just have your view catch mouseEntered: and mouseExited: events and display the menu yourself, thus simulating the submenu. Not the most ideal solution in the world, but it's something at least.