How to get an array of AXMenuItems from AXMenu?

Question

For my code I am attempting to get an array of AXMenuItems from an AXMenu (AXUIElementRef). The menu logs successfully, and here is my code:

NSArray *anArray = [NSRunningApplication runningApplicationsWithBundleIdentifier:@"com.apple.dock"];
AXUIElementRef anAXDockApp = AXUIElementCreateApplication([[anArray objectAtIndex:0] processIdentifier]);

CFTypeRef aChildren;
AXUIElementCopyAttributeValue(anAXDockApp, kAXChildrenAttribute, &aChildren);

SafeCFRelease(anAXDockApp);

CFTypeRef aMenu = CFArrayGetValueAtIndex(aChildren, 0);

NSLog(@"aMenu: %@", aMenu);

                    // Get menu items
CFTypeRef aMenuChildren;
AXUIElementCopyAttributeValue(aMenu, kAXVisibleChildrenAttribute, &aMenuChildren);

for (NSInteger i = 0; i < CFArrayGetCount(aMenuChildren); i++) {

    AXUIElementRef aMenuItem = [self copyAXUIElementFrom:aMenu role:kAXMenuItemRole atIndex:i];

    NSLog(@"aMenuItem: %@", aMenuItem); // logs (null)

    CFTypeRef aTitle;
    AXUIElementCopyAttributeValue(aMenuItem, kAXTitleAttribute, &aTitle);


    if ([(__bridge NSString *)aTitle isEqualToString:@"New Window"] || [(__bridge NSString *)aTitle isEqualToString:@"New Finder Window"]) /* Crashes here (i can see why)*/{

        AXUIElementPerformAction(aMenuItem, kAXPressAction);

        [NSThread sleepForTimeInterval:1];

        break;

    }

}

What is the correct way to get the list of AXMenuItems?

Screenshot of the Accessibility Inspector:

Answer 1

I have figured out an answer, using @Willeke answer of using AXUIElementCopyElementAtPosition() to get the menu. Since there were multiple dock orientations and hiding, I had to create enums in the .h file as it would be easier to read than 0, 1, or 2.

// .h
typedef enum {
    kDockPositionBottom,
    kDockPositionLeft,
    kDockPositionRight,
    kDockPositionUnknown
} DockPosition;

typedef enum {
    kDockAutohideOn,
    kDockAutohideOff
} DockAutoHideState;

Then, I added the methods to get these states in the .m

// .m
- (DockPosition)dockPosition
{
    NSRect screenRect = [[NSScreen mainScreen] frame];

    NSRect visibleRect = [[NSScreen mainScreen] visibleFrame];

    // Dont need to remove menubar height
    visibleRect.origin.y = 0;

    if (visibleRect.origin.x > screenRect.origin.x) {
        return kDockPositionLeft;
    } else if (visibleRect.size.width < screenRect.size.width) {
        return kDockPositionRight;
    } else if (visibleRect.size.height < screenRect.size.height) {
        return kDockPositionBottom;
    }
    return kDockPositionUnknown;
}

- (DockAutoHideState)dockHidden
{
    NSString *plistPath = [NSHomeDirectory() stringByAppendingPathComponent:@"Library/Preferences/com.apple.dock.plist"];
    NSDictionary *dockDict = [NSDictionary dictionaryWithContentsOfFile:plistPath];

    CFBooleanRef autohide = CFDictionaryGetValue((__bridge CFDictionaryRef)dockDict, @"autohide");
    if (CFBooleanGetValue(autohide) == true) {
        return kDockAutohideOn;
    }
    return kDockAutohideOff;
}

For the dock position unknown, I added in case it was not able to calculate it from the screen positions.

Then, I used a method to get the dock item from the menubar:

- (AXUIElementRef)getDockItemWithName:(NSString *)name
{
    NSArray *anArray = [NSRunningApplication runningApplicationsWithBundleIdentifier:@"com.apple.dock"];
    AXUIElementRef anAXDockApp = AXUIElementCreateApplication([[anArray objectAtIndex:0] processIdentifier]);

    AXUIElementRef aList = [self copyAXUIElementFrom:anAXDockApp role:kAXListRole atIndex:0];

    CFTypeRef aChildren;
    AXUIElementCopyAttributeValue(aList, kAXChildrenAttribute, &aChildren);
    NSInteger itemIndex = -1;

    for (NSInteger i = 0; i < CFArrayGetCount(aChildren); i++) {

        AXUIElementRef anElement = CFArrayGetValueAtIndex(aChildren, i);

        CFTypeRef aResult;

        AXUIElementCopyAttributeValue(anElement, kAXTitleAttribute, &aResult);

        if ([(__bridge NSString *)aResult isEqualToString:name]) {

            itemIndex = i;
        }
    }
    SafeCFRelease(aChildren);

    if (itemIndex == -1) return nil;

    // We have index now do something with it

    AXUIElementRef aReturnItem = [self copyAXUIElementFrom:aList role:kAXDockItemRole atIndex:itemIndex];
    SafeCFRelease(aList);
    return  aReturnItem;
}

This SafeCFRelease() method is a very simple method that checks if the passed value is not nil, then releases (had some issues earlier).

void SafeCFRelease( CFTypeRef cf )
{
    if (cf) CFRelease(cf);
}

And this method [copyAXUIElementFrom: role: atIndex:] is a method from @Willeke answer from another one of my questions:

- (AXUIElementRef)copyAXUIElementFrom:(AXUIElementRef)theContainer role:(CFStringRef)theRole atIndex:(NSInteger)theIndex {
    AXUIElementRef aResultElement = NULL;
    CFTypeRef aChildren;
    AXError anAXError = AXUIElementCopyAttributeValue(theContainer, kAXChildrenAttribute, &aChildren);
    if (anAXError == kAXErrorSuccess) {
        NSUInteger anIndex = -1;
        for (id anElement in (__bridge NSArray *)aChildren) {
            if (theRole) {
                CFTypeRef aRole;
                anAXError = AXUIElementCopyAttributeValue((__bridge AXUIElementRef)anElement, kAXRoleAttribute, &aRole);
                if (anAXError == kAXErrorSuccess) {
                    if (CFStringCompare(aRole, theRole, 0) == kCFCompareEqualTo)
                        anIndex++;
                    SafeCFRelease(aRole);
                }
            }
            else
                anIndex++;
            if (anIndex == theIndex) {
                aResultElement = (AXUIElementRef)CFRetain((__bridge CFTypeRef)(anElement));
                break;
            }
        }
        SafeCFRelease(aChildren);
    }
    return aResultElement;
}

Taking all this code, I put it into one of my methods:

// Check if in dock (otherwise cant do it)

                if ([self isAppOfNameInDock:[appDict objectForKey:@"AppName"]]) {

                    // Get dock item

                    AXUIElementRef aDockItem = [self getDockItemWithName:[appDict objectForKey:@"AppName"]];

                    AXUIElementPerformAction(aDockItem, kAXShowMenuAction);

                    [NSThread sleepForTimeInterval:0.5];

                    CGRect aRect;

                    CFTypeRef aPosition;
                    AXUIElementCopyAttributeValue(aDockItem, kAXPositionAttribute, &aPosition);
                    AXValueGetValue(aPosition, kAXValueCGPointType, &aRect.origin);
                    SafeCFRelease(aPosition);

                    CFTypeRef aSize;
                    AXUIElementCopyAttributeValue(aDockItem, kAXSizeAttribute, &aSize);
                    AXValueGetValue(aSize, kAXValueCGSizeType, &aRect.size);
                    SafeCFRelease(aSize);

                    SafeCFRelease(aDockItem);

                    CGPoint aMenuPoint;

                    if ([self dockHidden] == kDockAutohideOff) {
                        switch ([self dockPosition]) {
                            case kDockPositionRight:
                                aMenuPoint = CGPointMake(aRect.origin.x - 18, aRect.origin.y + (aRect.size.height / 2));
                                break;
                            case kDockPositionLeft:
                                aMenuPoint = CGPointMake(aRect.origin.x + aRect.size.width + 18, aRect.origin.y + (aRect.size.height / 2));
                                break;
                            case kDockPositionBottom:
                                aMenuPoint = CGPointMake(aRect.origin.x + (aRect.size.width / 2), aRect.origin.y - 18);
                                break;
                            case kDockPositionUnknown:
                                aMenuPoint = CGPointMake(0, 0);
                                break;
                        }
                    } else {

                        NSRect screenFrame = [[NSScreen mainScreen] frame];

                        switch ([self dockPosition]) {
                            case kDockPositionRight:
                                aMenuPoint = CGPointMake(screenFrame.size.width - 18, aRect.origin.y + (aRect.size.height / 2));
                                break;
                            case kDockPositionLeft:
                                aMenuPoint = CGPointMake(screenFrame.origin.x + 18, aRect.origin.y + (aRect.size.height / 2));
                                break;
                            case kDockPositionBottom:
                                aMenuPoint = CGPointMake(aRect.origin.x + (aRect.size.width / 2), screenFrame.size.height - 18);
                                break;
                            case kDockPositionUnknown:
                                aMenuPoint = CGPointMake(0, 0);
                                break;
                        }
                    }

                    if ((aMenuPoint.x != 0) && (aMenuPoint.y != 0)) {

                        AXUIElementRef _systemWideElement = AXUIElementCreateSystemWide();

                        AXUIElementRef aMenu;
                        AXUIElementCopyElementAtPosition(_systemWideElement, aMenuPoint.x, aMenuPoint.y, &aMenu);

                        SafeCFRelease(_systemWideElement);

                        // Get menu items
                        CFTypeRef aMenuChildren;
                        AXUIElementCopyAttributeValue(aMenu, kAXVisibleChildrenAttribute, &aMenuChildren);

                        NSRunningApplication *app = [[NSRunningApplication runningApplicationsWithBundleIdentifier:[appDict objectForKey:@"BundleID"]] objectAtIndex:0];

                        for (NSInteger i = 0; i < CFArrayGetCount(aMenuChildren); i++) {

                            AXUIElementRef aMenuItem = [self copyAXUIElementFrom:aMenu role:kAXMenuItemRole atIndex:i];

                            CFTypeRef aTitle;
                            AXUIElementCopyAttributeValue(aMenuItem, kAXTitleAttribute, &aTitle);

                            // Supports chrome, safari, and finder
                            if ([(__bridge NSString *)aTitle isEqualToString:@"New Window"] || [(__bridge NSString *)aTitle isEqualToString:@"New Finder Window"]) {

                                AXUIElementPerformAction(aMenuItem, kAXPressAction);

                                NSInteger numberOfWindows = [self numberOfWindowsOpenFromApplicationWithPID:[app processIdentifier]];
                                // Wait until open
                                while ([self numberOfWindowsOpenFromApplicationWithPID:[app processIdentifier]] <= numberOfWindows) {
                                }
                                break;
                            }
                        }
                        SafeCFRelease(aMenu);
                        SafeCFRelease(aMenuChildren);
                    }
                }

This is pretty complicated, but it works. I probably can't explain it, but I have stress tested this code and it works quite well.

Answer 2

Answer to the question "How to get an array of AXMenuItems from AXMenu?": aMenuChildren is the list of menu items. To be sure you could filter by role kAXMenuItemRole.

Answer to the question "Why does it not work?": The menu isn't a menu. When you inspect the menu in Axccessibility Inspector, it displays a warning "Parent does not report element as one of its children". The Dock app has one child, a list of dock items.