Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Web Automation Mouse Clicking

I have a project that involves opening multiple window clients and then simulating mouse clicks within each of those processes. I've been able to succesfully send a message to multiple instances of Notepad using the Win32 API and SendMessage. The code that works for me is as follows:

Process[] notepads = Process.GetProcessesByName("notepad");
            foreach (Process proc in notepads)
            {
                    IntPtr handle = proc.Handle;
                    IntPtr child = FindWindowEx(proc.MainWindowHandle, new IntPtr(0), "Edit", null);
                    if (child != null)
                    {
                        MessageBox.Show("Child was found, sending text");
                        SendMessage(child, 0x000C, 0, "test");
                    }
                }
            }

This sends "test" to every instance I have opened of Notepad, no matter how many. As you can see I'm iterating through every instance of a process, and simply looping the message. Wasn't hard at all...

The end goal isn't Notepad, rather the Microsoft extension Awesomium. I retrieved the window handle, and then the child (Awesomium) Class name, which is Chrome_RenderWidgetHostHWND. From there, I tried to send mouse events by refactoring the variable types in Sendmessage in order to assemble and lParam readable by the system. Here is that code:

[DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = false)]
        static extern IntPtr SendMessage(IntPtr hWnd, uint Msg, IntPtr wParam, IntPtr lParam);

Using some examples I found I assemble the lParam via this:

int x = 834;
int y = 493;
IntPtr lParam = (IntPtr)((y << 16) | x);
IntPtr wParam = IntPtr.Zero;

I am using the Global Mouse hooks to detect press events in all child windows, and when I hit my button to iterate through the process and click, nothing is sent at all. I know I'm doing something stupid here that's causing the mouse clicks to not be sent. In any case, here is the final code structure that I have. Any advice, or tips would be appreciated. Thanks.

  private void button2_Click(object sender, EventArgs e)
    {
        Process[] notepads = Process.GetProcessesByName("client");
        foreach (Process proc in notepads)
        {
                IntPtr handle = proc.Handle;
                string mine = Convert.ToString(proc);
                MessageBox.Show(mine);
                IntPtr child = FindWindowEx(proc.MainWindowHandle, new IntPtr(0), "Chrome_RenderWidgetHostHWND", null);
                if (child != null)
                {
                int x = 834;
                int y = 493;
                IntPtr lParam = (IntPtr)((y << 16) | x);
                IntPtr wParam = IntPtr.Zero;
                SendMessage(child, 0x201, wParam, lParam);
                SendMessage(child, 0x202, wParam, lParam);
                }
            }
        }
like image 533
user2578424 Avatar asked Jul 19 '13 17:07

user2578424


People also ask

What is XClick?

The XClick and the XMove commands allows you to send real user mouse clicks to web page elements. The advantage of this approach is that the web app reacts 100% the same way as if a real human user executes this click.


1 Answers

Or you could use SendInput instead, which will work correctly in the presence of mouse hooks; your solution (of posting messages directly) will not.

More importantly, if the process rejects activation (i.e., if the target returns 0 from WM_MOUSEACTIVATE), the WM_LBUTTONDOWN won't get sent.

like image 128
Eric Brown Avatar answered Oct 13 '22 16:10

Eric Brown