How To Capture Text Using Mouse Pointer And Keyboard Shortcuts?

Question

i want to capture text from opened windows using mouse pointer and keyboard shortcut using C# or java ( like babylon ) , so what i need to know and how to implement it ?

what are the libraries i need to use ? or can i use winapi ?

Answer 1

Use a scripting language to create a draft of what you want to do.

You can use programs like AutoHotKey or AutoIt. Note, that thy include auto recorder, that gives you a basic draft. You can compile those scripts to executables, and call them from C# or Java using Shell Execute ( c#; java (exec) ) or run as new Process ( c#; java (process builder) ). Latter is preferred.

Here is an example of how to map a key 'pause', to a function that selects a text from screen, copy's it and pastes it in another place using AutoHotKey. Shift + left click is used on background to select all the text. Note, that this is simplest example and does not invoke window by its pointer and uses fixed locations (and work only for one resolution).

HotKeySet("{PAUSE}", "getInput")

While 1
    Sleep(100)
Wend


Func getInput()
    MouseClick("left",272,241,1)
    Sleep(100)
    MouseClick("left",272,241,1)
    Send("{SHIFTDOWN}")
    MouseClick("left",272,241,1)
    MouseClick("left",529,242,2)
    Send("{SHIFTUP}{CTRLDOWN}c{CTRLUP}")
    MouseClick("left",656,42,1)
    Sleep(100)  
    MouseClick("left",696,42,1)
    Send("{CTRLDOWN}a")
    Send("{DELETE}")
    Send("{CTRLDOWN}v{CTRLUP}")
    MouseClick("left",1178,44,1)
EndFunc

Using Java.

Java contains Robot class, to do this.

This class is used to generate native system input events for the purposes of test automation, self-running demos, and other applications where control of the mouse and keyboard is needed. The primary purpose of Robot is to facilitate automated testing of Java platform implementations.

Using the class to generate input events differs from posting events to the AWT event queue or AWT components in that the events are generated in the platform's native input queue. For example, Robot.mouseMove will actually move the mouse cursor instead of just generating mouse move events.

Note that some platforms require special privileges or extensions to access low-level input control. If the current platform configuration does not allow input control, an AWTException will be thrown when trying to construct Robot objects. For example, X-Window systems will throw the exception if the XTEST 2.2 standard extension is not supported (or not enabled) by the X server.

Applications that use Robot for purposes other than self-testing should handle these error conditions gracefully.

You can tailor how you use Robot yourself, but general way:

import java.awt.Robot;
import java.awt.event.InputEvent;
import java.awt.event.KeyEvent;

public class Tester {
    public static void doLeftMouseClick(Robot r, int x, int y) {
        r.mouseMove(x, y);
        r.mousePress(InputEvent.BUTTON1_MASK);
        r.mouseRelease(InputEvent.BUTTON1_MASK);
    }

    public static void doLeftMouseClickEvent(Robot r, int x, int y, int nr) {
        r.mouseMove(x, y);
        if (nr == 1)
            r.mousePress(InputEvent.BUTTON1_MASK);
        else
            r.mouseRelease(InputEvent.BUTTON1_MASK);
    }

    public static void main(String args[]) throws Exception {
        Robot r = new Robot();
        doLeftMouseClick(r, 272, 241);
        r.delay(1000);
        doLeftMouseClick(r, 272, 241);
        r.keyPress(KeyEvent.SHIFT_MASK);
        doLeftMouseClickEvent(r, 272, 241, 1);
        doLeftMouseClickEvent(r, 529, 242, 2);
        r.keyRelease(KeyEvent.SHIFT_MASK);
        r.keyPress(KeyEvent.CTRL_MASK);
        r.keyPress(KeyEvent.VK_C);
        r.keyRelease(KeyEvent.CTRL_MASK);
        // etc.
    }
}

More Robot examples at java2s: ( link )

1. Robot: createScreenCapture(Rectangle screenRect)
2. Robot: getPixelColor(int x, int y)
3. Robot: keyPress(int keycode)
4. Robot: keyRelease(int keycode)
5. Robot: mouseMove(int x, int y)
6. Robot: mousePress(int buttons)
7. Robot: mouseRelease(int buttons)
8. Robot: mouseWheel(int wheelAmt)

Using C#.

There are myriad of solutions. Just google "Test Automation c#" or "spy c#".

MSDN: SendKeys
MSDN: How to: Simulate Mouse and Keyboard Events in Code

You can use windows API, but it requires some tedious work. You don't want to do that, you really don't, but if you do, then definitely check out:

• http://www.pinvoke.net/default.aspx/user32.mouse_event
• http://www.pinvoke.net/default.aspx/user32.sendinput

I recommend you use inputsimulator. Example:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
// add reference to following
using WindowsInput;
using System.Windows.Forms;
using System.Drawing;

namespace ConsoleApplicationTester
{
    class Program
    {
        public static void doLeftMouseClick(int x, int y)
        {
            Cursor.Position = new System.Drawing.Point(x, y);
            InputSimulator.SimulateKeyPress(VirtualKeyCode.LBUTTON);
        }
        public static void doLeftMouseClickEvent(int x, int y, int nr)
        {
            Cursor.Position = new Point(x, y);
            if(nr==1)
                InputSimulator.SimulateKeyDown(VirtualKeyCode.LBUTTON);
            else
                InputSimulator.SimulateKeyUp(VirtualKeyCode.LBUTTON);
        }

        static void Main(string[] args){
            doLeftMouseClick( 272, 241);
            System.Threading.Thread.Sleep(100);
            doLeftMouseClick( 272, 241);
            InputSimulator.SimulateKeyDown(VirtualKeyCode.MENU);
            doLeftMouseClickEvent(272, 241, 1);
            doLeftMouseClickEvent(529, 242, 2);
            InputSimulator.SimulateKeyUp(VirtualKeyCode.MENU);
            InputSimulator.SimulateKeyDown(VirtualKeyCode.CONTROL);
            InputSimulator.SimulateKeyUp(VirtualKeyCode.VK_C);
            InputSimulator.SimulateKeyUp(VirtualKeyCode.CONTROL);
            // etc.          
        }
    }
}