How to click buttons in an application using AutoIt or Windows UI Automation

Question

Setup Environment:

I'm using vb.net to develop a Windows Form Application with .NET Framework 4.

My goal:

1. Open calculator.exe using Process.Start
2. Using all vb.net code, be able to click 5 + 5 =

I do not want to use SendKeys as a method.

After researching, this link provided a good start:

This tutorial (written in C#) is very similar to what I'm trying to do by using vb.net:

• How to click a button in another application

Could somebody provide a pointer on how to go about approaching this? I'd really appreciate it.

Answer 1

My Solution:

I tried two approaches:

• Windows UI Automation
• AutoIt

AutoIt was what I used because it was more reliable for my specific application.

However, Windows UI worked as well. Here are both solutions.

Steps if using Windows UI Automation:

• Identify Control IDs of buttons using Spy++
• Add references to UIAutomationClient and UIAutomationTypes
• Set aeDesktop as the root ae element and invoke button clicks

Imports System.Windows.Automation
Imports System.Threading
Imports System.Diagnostics

Public Class Form1

    Private aeDesktop As AutomationElement
    Private aeCalculator As AutomationElement

    Private ae5Btn As AutomationElement
    Private aeAddBtn As AutomationElement
    Private aeEqualsBtn As AutomationElement

    Private p As Process

    Private Sub Button1_Click(sender As System.Object, e As System.EventArgs) Handles Button1.Click

        Try

            'Set reference to the root ae element - the desktop
            aeDesktop = AutomationElement.RootElement

            'Launch Calculator application 
            p = Process.Start("C:\Windows\System32\calc.exe")

            '**********  Keep looping while waiting to get the reference to the "Calculator" on Desktop ************************************

            Dim numwaits As Integer = 0

            Do
                Debug.WriteLine("Looking for Calculator . . . ")
                aeCalculator = aeDesktop.FindFirst(TreeScope.Children, New PropertyCondition(AutomationElement.NameProperty, "Calculator"))
                numwaits += 1
                Thread.Sleep(100)

            Loop While aeCalculator Is Nothing AndAlso numwaits < 50

            If aeCalculator Is Nothing Then
                Throw New Exception("Failed to find Calculator")
            Else
                Debug.WriteLine("Found the Calculator Application!")
            End If

            '*********************************************************************************************************************************


            'NOTE: In spy++ Controlids are represented as hex (i.e. 00000087) - need to convert these to decimal (i.e. 135)

            '`5` btn
            '00000087 ---> 135
            Dim btn5hexID As String = "00000087"
            Dim btn5decimalID As String = Convert.ToInt32("00000087", 16).ToString

            '`+` btn
            '0000005D ---> 93
            Dim btnAddhexID As String = "0000005D"
            Dim btnAdddecimalID As String = Convert.ToInt32("0000005D", 16).ToString

            '`=` btn
            '00000079 ---> 121
            Dim btnEqualshexID As String = "00000079"
            Dim btnEqualsdecimalID As String = Convert.ToInt32("00000079", 16).ToString


            'Set reference for the `5` Button
            ae5Btn = aeCalculator.FindFirst(TreeScope.Descendants, New PropertyCondition(AutomationElement.AutomationIdProperty, btn5decimalID))

            'Set reference for the `+` Button
            aeAddBtn = aeCalculator.FindFirst(TreeScope.Descendants, New PropertyCondition(AutomationElement.AutomationIdProperty, btnAdddecimalID))

            'Set reference for the `=` Button
            aeEqualsBtn = aeCalculator.FindFirst(TreeScope.Descendants, New PropertyCondition(AutomationElement.AutomationIdProperty, btnEqualsdecimalID))


            'Manipulate calculator application by using invoke method to click on buttons
            Dim ipClick5Btn As InvokePattern = DirectCast(ae5Btn.GetCurrentPattern(InvokePattern.Pattern), InvokePattern)
            Dim ipClickAddBtn As InvokePattern = DirectCast(aeAddBtn.GetCurrentPattern(InvokePattern.Pattern), InvokePattern)
            Dim ipClickEqualsBtn As InvokePattern = DirectCast(aeEqualsBtn.GetCurrentPattern(InvokePattern.Pattern), InvokePattern)

            'Click 5
            ipClick5Btn.Invoke()

            'Click +
            ipClickAddBtn.Invoke()

            'Click 5
            ipClick5Btn.Invoke()

            'Click =
            ipClickEqualsBtn.Invoke()

            'Now calculator should display 10 as a result


            'Wait two seconds before closing
            Thread.Sleep(2000)

            'Exit Calculator
            p.CloseMainWindow()



        Catch ex As Exception

            'Handle any exceptions
            Debug.WriteLine("Fatal error: " & ex.Message)

        End Try

    End Sub

End Class

Steps if using AutoIt:

• Identify ClassnameNN of buttons
• Get a handle for calc.exe
• Use ControlClick function to click buttons

If using AutoIt choose the full installation and download the Script Editor. Paste the code in and it should work.

• Download AutoIt

;Open up Calculator
Run('calc.exe')

;Pause execution until Calculator becomes active window
WinWaitActive('Calculator')

;Get the handle for Calculator
$hWnd = WinGetHandle('Calculator')

;Using the `Finder Tool`, you can drag and drop it onto controls to see all information (i.e. Text, Class, Handle, etc.)

;`ClassnameNN: Button10` is the number 5
;`ClassnameNN: Button23` is the addition operator (+)
;`ClassnameNN: Button28` is the equals operator (=)

;***** simple operation will perform 5 + 5 = 10 **************

;click 5
ControlClick($hWnd, "", "[CLASSNN:Button10]")

;click +
ControlClick($hWnd, "", "[CLASSNN:Button23]")

;click 5
ControlClick($hWnd, "", "[CLASSNN:Button10]")

;click =
ControlClick($hWnd, "", "[CLASSNN:Button28]")

;calculator should now display 10 as a result

;************************************************************

;Wait 2 seconds to show result
Sleep(2000)

;Close Calculator
WinClose($hWnd)

Exit

Thank you for all the help and suggestions in the comments. It helped tremendously.