How to perform Mouse Wheel scrolling over HTML5 Canvas in Selenium?

Question

I am working on a GWT Application (similar to Paint). In this, I have an HTML5 Canvas in which there is a functionality that scrolling a mousewheel up and down will zoom in and out of the canvas.

I have searched a lot but didn't find a workaround to fix this issue. Here's what did:

int PosX = 0;
int PosY = 10;
JavascriptExecutor executor = (JavascriptExecutor) getDriver();
String script = "document.getElementById('frontCanvas').scrollBy("
                                + PosX + "," + PosY + ")";
executor.executeScript(script); 

WebDriverWait wait = new WebDriverWait(getDriver(), 20);
wait.until(ExpectedConditions.javaScriptThrowsNoExceptions(script));

Now, this above code is working for another Angular application in which I am scrolling up and down a div element (which has a scrollbar) but it is not working on my canvas (which doesn't have a scrollbar) in the GWT application.

I am using Selenium 3.14.0 and run this code on the Chrome browser. Can anyone suggest what can be done to fix this issue?

Answer 1

HTML5 Canvas

The HTML element is used to draw graphics, on the fly, via JavaScript. The element is only a container for graphics. You must use JavaScript to actually draw the graphics. Canvas has several methods for drawing paths, boxes, circles, text, and adding images.

In general, to scroll the mousewheel up and down we could have opted for the Actions Class. But as per Automated Testing of HTML5 Canvas Applications with Selenium WebDriver it seems this API is not that reliable. In Firefox, every mouse down, mouse up, or mouse click happens at the center of the element. So the code above produces a mouse move event to the provided (x,y), then a mouse move event to the center of the Canvas, then a mouse down, mouse up, and click all at the center of the Canvas. That may be fine for a button, but is unworkable for a Canvas, where you want to be able to hover, click, etc. at a specific location. The situation is even worse in Safari, where it just produces an exception indicating that mouse move events aren't supported. Chrome, meanwhile, works fine.

Alternative

An work around would be to use the JavascriptExecutor Interface manually dispatching synthesized mouse events using JavaScript.

Taking a leaf out of @FlorentB.'s answer, to scroll a mousewheel up and down, you can emit the mouseover, mousemove and wheel events to the top element with a script injection and you can use the following solution:

• Code Block:

  package demo;
  
  import org.openqa.selenium.By;
  import org.openqa.selenium.JavascriptExecutor;
  import org.openqa.selenium.WebDriver;
  import org.openqa.selenium.WebDriverException;
  import org.openqa.selenium.WebElement;
  import org.openqa.selenium.chrome.ChromeDriver;
  import org.openqa.selenium.chrome.ChromeOptions;
  import org.openqa.selenium.support.ui.ExpectedConditions;
  import org.openqa.selenium.support.ui.WebDriverWait;
  
  public class Canvas {
  
      static WebDriver driver;
      public static void main(String[] args) {
  
          System.setProperty("webdriver.chrome.driver", "C:\\Utility\\BrowserDrivers\\chromedriver.exe");
          ChromeOptions options = new ChromeOptions();
          options.addArguments("start-maximized");
          options.addArguments("disable-infobars");
          options.addArguments("--disable-extensions");
          driver = new ChromeDriver(options);
          driver.get("https://www.google.co.uk/maps");
          WebElement elm = new WebDriverWait(driver, 20).until(ExpectedConditions.visibilityOfElementLocated(By.cssSelector("#scene > div.widget-scene > canvas")));
          // Mouse wheel UP or Zoom In 
          wheel_element(elm, -500, 0, 0);
          System.out.println("Mouse wheel UP or Zoom In through Wheel achieved !!!");
          // Mouse wheel DOWN or Zoom Out 
          wheel_element(elm, 120, 0, 0);
          System.out.println("Mouse wheel DOWN or Zoom Out through Wheel achieved !!!");
          System.out.println("Mouse Scroll through Wheel achieved !!!");
      }
  
      public static void wheel_element(WebElement element, int deltaY, int offsetX, int offsetY)
      {
          try{
                String script = "var element = arguments[0];"
                  +"var deltaY = arguments[1];"
                  +"var box = element.getBoundingClientRect();"
                  +"var clientX = box.left + (arguments[2] || box.width / 2);"
                  +"var clientY = box.top + (arguments[3] || box.height / 2);"
                  +"var target = element.ownerDocument.elementFromPoint(clientX, clientY);"
                  +"for (var e = target; e; e = e.parentElement) {"
                    +"if (e === element) {"
                  +"target.dispatchEvent(new MouseEvent('mouseover', {view: window, bubbles: true, cancelable: true, clientX: clientX, clientY: clientY}));"
                  +"target.dispatchEvent(new MouseEvent('mousemove', {view: window, bubbles: true, cancelable: true, clientX: clientX, clientY: clientY}));"
                  +"target.dispatchEvent(new WheelEvent('wheel',     {view: window, bubbles: true, cancelable: true, clientX: clientX, clientY: clientY, deltaY: deltaY}));"
                  +"return;"
                    +"}"
                  +"}";  
  
                WebElement parent = (WebElement) ((JavascriptExecutor) driver).executeScript("return arguments[0].parentNode;", element);
                ((JavascriptExecutor) driver).executeScript(script, parent, deltaY, offsetX, offsetY);
              }catch(WebDriverException e)
              {
              System.out.println("Exception caught in Catch block");
              }
      }
  
  }
  

• Console Output:

  Mouse wheel UP or Zoom In through Wheel achieved !!!
  Mouse wheel DOWN or Zoom Out through Wheel achieved !!!
  Mouse Scroll through Wheel achieved !!!
  

---

Reference

You can find a couple of relevant detailed discussion in:

• How to find the coordinates of the buttons on a canvas, and click on them after using Java and Selenium?