AppleScript or Automator to click on menus in an application?

Question

I'm not sure if this is do-able via AppleScript and/or Automator…but I'd like to be able to:

a) launch an application (I know this can be done pretty easily by AppleScript or Automator)

b) once the application is launched, use AppleScript or Automator to select specific menu items.

e.g. I'd like to launch Excel 2008 (I have the home/student edition which doesn't come preconfigured for Automator) and then click the "File" menu and click on "open".

Any pointers on where to go/look for how to select menu items like this (or if it's even possible at all)?

You can "sort of" do this using Automator's Record function, but Record is very brittle.

I'd rather be able to use AppleScript to simply grab an "array" that contains every menu item for the application and then programmatically click on the 0th menu item in my array…etc. etc.

Is this possible?

TIA

Answer 1

tell application "###"

activate

end tell


tell application "System Events"

tell process "###"

click menu item "^^^" of menu "$$$" of menu bar 1

end tell

end tell

Put your application in for the ### and put your menu item in for the ^^^ and put your menu (file, edit, view, etc.) in for $$$. Capitalization matters.

Put it in applescript btw

EXAMPLE:

tell application "iTunes"

activate

end tell


tell application "System Events"

tell process "iTunes"

click menu item "as list" of menu "view" of menu bar 1

end tell

end tell

delete the double spaces EXCEPT BETWEEN END TELL AND TELL APPLICATION "SYSTEM EVENTS"

Answer 2

I frequently need to create a GUI script to access a menu item in applications whose AppleScript library doesn't provide direct access to the objects or functions the menu item represents. To make it easy to re-use the code, I use variables for the app, menu, & menu item names. Then I just need to change the variables at the top rather than pick out the names from the code body.

set targetApp to "app_name"
set theMenu to "menu_name"
set theItem to "menu_item_name"

tell application targetApp
    activate
    tell application "System Events"
        tell application process targetApp
            tell menu bar 1
                tell menu bar item theMenu
                    tell menu theMenu
                        click menu item theItem
                    end tell
                end tell
            end tell
        end tell
    end tell
end tell

SUB-MENUS

It gets a little more involved when there are sub- & sub-sub-menus involved, as a menu item with a sub-menu is both a menu item of its parent menu AND a menu parent of its sub-menus. Note that the text variable "theItem" is used to specify both a menu item AND a menu; the "targetApp" string variable is used to reference both an app & a process, so it saves having to edit 2 names in 2 places each when reusing the code. I use this script to run on voice commands for accessing menu items quickly rather than having to say, e.g., "click Edit Menu"... "click Transformations"... "click Make Upper Case"... I add another variable for the sub-menu item:

set targetApp to "app_name"
set theMenu to "menu_name"
set theItem to "menu_item_name"
set theSubItem to "sub_item_name"

tell application targetApp
    activate
    tell application "System Events"
        tell application process targetApp
            tell menu bar 1
                tell menu bar item theMenu
                    tell menu theMenu
                        tell menu item theItem
                            tell menu theItem
                                click menu item theSubItem
                            end tell
                        end tell
                    end tell
                end tell
            end tell
        end tell
    end tell
end tell

For example:

set targetApp to "TextEdit"
set theMenu to "Edit"
set theItem to "Transformations"
set theSubItem to "Make Upper Case"

tell application targetApp
    activate
    tell application "System Events"
        tell application process targetApp
            tell menu bar 1
                tell menu bar item theMenu
                    tell menu theMenu
                        tell menu item theItem
                            tell menu theItem
                                click menu item theSubItem
                            end tell
                        end tell
                    end tell
                end tell
            end tell
        end tell
    end tell
end tell

If there's another level of sub-menus, an additional variable (e.g., "theSubSubItem") would be required, & the line in the System Events process tell block would have another layer
...

                            tell menu item theItem
                                tell menu theSubItem
                                    click menu item theSubSubItem
                                end tell
                            end tell

...
As noted elsewhere in this thread it's recommended to address an application's objects & functions directly whenever they're included in the API, but addressing the GUI is useful as a last resort when the API doesn't provide direct access. The downside is a GUI script can get more cumbersome & may have to be revised with each application update.

Answer 3

Indeed, UI scripting is fragile and finicky, but you might take a look at Apple's AppleScript GUI Scripting page

Answer 4

UI scripting is a fickle beast. It's brittle and often sensitive to many things beyond your control such as unexpected dialogs (from other applications), changes in system font or language, user activity. Not to mention changes in an application's UI across versions.

It's much more robust to use talk directly to the application via its AppleScript API. This is exactly the purpose of AppleScript. You can always see the actions and data an application can provide via AppleScript via the "Open Dictionary..." menu item in AppleScript Editor.app's (select the application you want to script in the file dialog). Instead of simulating a mouse click on a menu, you call the action that menu item triggers directly via AppleScript.

Of course not all applications have a complete (or even any) AppleScript API. Excel, however, has excellent AppleScript support, if that's your target.

If you're really trying to do UI testing, I recommend you start with other options:

1. Test the model (analogous to scripting the app's model via AppleScript). In fact, if you include an AppleScript API in your app, you can automated testing of that API as well.
2. Use Google Toolbox for Mac's unit testing additions (or roll your own) to programatically send mouse or keyboard events to your app.
3. Use Instruments.app to record a UI sequence. The Instruments recording is solid (I haven't used Automator's Record feature), and you get all the other goodies of Instruments too.