Java find image on screen

Question

Can you guys give me hints on how to find a image on screen. I mean, a simple pixel combination. For exmaple, it finds coordinates of 30x30 pixel white square.

Java robot class allows me to find color of certain pixel. But i need to opposite, I want my program to scan my screen and then tell me the coords of this little image. Well I could go through all pixels with Robot, but it should be faster than that. Much faster.

Any suggestions?

Answer 1

Actually, there's a much simpliler or more reliable solution to this. You can implement the Sikuli libraries inside your Java application to spot image elements on your screen and interact with them. It was meant to automate UI testing, but I think it can accommodate your needs pretty easily.

Sample application (source):

import java.net.MalformedURLException;
import java.net.URL;

import org.sikuli.api.*;
import org.sikuli.api.robot.Mouse;
import org.sikuli.api.robot.desktop.DesktopMouse;
import org.sikuli.api.visual.Canvas;
import org.sikuli.api.visual.DesktopCanvas;

import static org.sikuli.api.API.*;

public class HelloWorldExample {

     public static void main(String[] args) throws MalformedURLException {

           // Open the main page of Google Code in the default web browser
           browse(new URL("http://code.google.com"));

           // Create a screen region object that corresponds to the default monitor in full screen 
           ScreenRegion s = new DesktopScreenRegion();

           // Specify an image as the target to find on the screen
           URL imageURL = new URL("http://code.google.com/images/code_logo.gif");                
           Target imageTarget = new ImageTarget(imageURL);

           // Wait for the target to become visible on the screen for at most 5 seconds
           // Once the target is visible, it returns a screen region object corresponding
           // to the region occupied by this target
           ScreenRegion r = s.wait(imageTarget,5000);

           // Display "Hello World" next to the found target for 3 seconds
           Canvas canvas = new DesktopCanvas();
           canvas.addLabel(r, "Hello World").display(3);

           // Click the center of the found target
           Mouse mouse = new DesktopMouse();
           mouse.click(r.getCenter());
     }
}

Also see How to use Sikuli inside your Java programs for setup.

Answer 2

Well I could go through all pixels with Robot, but it should be faster than that. Much faster.

I'm afraid that that's precisely what you'll have to do.

If all pixels should be white, you could first take 30 pixel wide steps and if you find a white pixel, take say, 5 pixel steps, and then if these pixels are white too, examine the remaining pixels in the square.

Something like this:

.        .        .        .        .        .



.        ..........        .        .        .
         ...... 
         .  .  .  .

         .  .  .  .
.        .        .        ..........        .
                           ..........
                           ..........
                           ..........
                           ..........
.        .        .        ..........