Human-like mouse movements via Selenium

Question

The Story:

One of the approaches to solve captchas, like Google ReCaptcha, is to try to imitate the human mouse actions: movements, hovering and clicks.

Some users reported that making mouse moves as B-spline curves worked for them.

The Question:

How to move the mouse to a particular element following the B-spline trajectory via Selenium?

---

Note that the regular browser.actions().mouseMove(elm).perform(); would "jump" to the element straight and far too quickly. My understanding is that it is a matter of slowing down the movement speed, "jumping" from point to point smoothly following the mathematical model for the B-spline trajectory.

^{We are using Protractor/JavaScript, but the question is really language-agnostic.Note that I'm not trying to solve the captcha, or contribute to the "captcha-solving making new evil bots violating terms of use here and there" space. I'm just curious and eager to obtain more skills in the test automation space.}

Answer 1

You can use scipy.interpolate to interpolate B-spline curves like you can see in this question.

Here I'll use one of the B-spline examples to get values to x and y:

import numpy as np import scipy.interpolate as si  # Curve base: points = [[0, 0], [0, 2], [2, 3], [4, 0], [6, 3], [8, 2], [8, 0]]; points = np.array(points)  x = points[:,0] y = points[:,1]   t = range(len(points)) ipl_t = np.linspace(0.0, len(points) - 1, 100)  x_tup = si.splrep(t, x, k=3) y_tup = si.splrep(t, y, k=3)  x_list = list(x_tup) xl = x.tolist() x_list[1] = xl + [0.0, 0.0, 0.0, 0.0]  y_list = list(y_tup) yl = y.tolist() y_list[1] = yl + [0.0, 0.0, 0.0, 0.0]  x_i = si.splev(ipl_t, x_list) # x interpolate values y_i = si.splev(ipl_t, y_list) # y interpolate values 

With values of x and y, you can move the mouse cursor with ActionChains:

from selenium import webdriver from selenium.webdriver.common.action_chains import ActionChains  url = "https://codepen.io/falldowngoboone/pen/PwzPYv" driver = webdriver.Chrome(executable_path="/home/selenium/chromedriver2.25") driver.get(url)  action =  ActionChains(driver);  startElement = driver.find_element_by_id('drawer')  # First, go to your start point or Element: action.move_to_element(startElement); action.perform();  for mouse_x, mouse_y in zip(x_i, y_i):     action.move_by_offset(mouse_x,mouse_y);     action.perform();     print(mouse_x, mouse_y)