how to build a bot for on-line flash game?

Question

there are a lot of flash games that ask u to do repetitive actions. an example would be FarmVille or getting statistics on on-line poker game.

What is the best way to create an interface between what you see on the screen, a bot's algorithm and clicking of the mouse.

so basically, when i'm playing a texas hold'em on face book and if i get AA i want the bot to click all-in. if i don't get AA i want to fold. this is just an example, not the actual strategy. or another examples would be routing tasks in farmville.

so basically. it has to grab a picture of the screen. recognize patterns and convert them into an input of the algorithm. the output algorithms would be clicking on some other (may be same) patterns.

any suggestions?

Answer 1

I have experience building a simple computer vision based bot for a online card game.

I used a crossplatform python library called autopy to handle automated mouse click and keyboard input. The bot capture screen every 0.5 sec, then convert the captured screen into a numpy array for analysis. Python has good image processing (PIL, OpenCV python binding) and machine learning utilities (scikit-learn). For simple image pattern recognition, the bot extract mean of pixel brightness over a image region to make decision. For complicated ones, OpenCV template matching and SVM classifier are being used.

The server caught my bot the first time, so I guess some there's bot detection on the server side. After I added more randomness and flexible decision making, the bot has bypassed serverside bot detection.

All took me a Saturday and I enjoyed Sunday.

Answer 2

Before attempting to parse screen images and produce mouse clicks (which would be exceedingly difficult), first take a look at the data actually being sent/received by the game between the browser and the server. Interact with the game as normal and use something like WireShark to capture the packets and see if it's something you can automate.

It's unlikely (since they specifically don't want people to do what you're trying to do), but it's possible that the communication with the server is just simple requests with parameters. (I know this was the case a long time ago with MySpace games, such as Dragon Wars and Mafia Wars.)

If the data is something you can parse and understand then you can skip the Flash interface entirely and just write your own.

Answer 3

I've made many bot using c# and a simple webBrowser control. Usually however i try to automate the "click" function on a element without the use of actually simulating a cursor and invoking a click.

Example:

webBrowser1.Document.GetElementById("elementid").InvokeMember("Click");

I find this much easier than actually simulating a cursor, however simulating a cursor can also achieved relatively easily , here's the code i usually make use of:

//Example of how to use it:
    Point controlLoc = this.PointToScreen(webbrowser1.Location);
    controlLoc.X = controlLoc.X + webbrowser1.Document.GetElementById("elementid").OffsetRectangle.Left;
    controlLoc.Y = controlLoc.Y + webbrowser1.Document.GetElementById("elementid").OffsetRectangle.Top;
    Cursor.Position = controlLoc;
    MouseSimulator.ClickLeftMouseButton();

public class MouseSimulator
 {
[DllImport("user32.dll", SetLastError = true)]
static extern uint SendInput(uint nInputs, ref INPUT pInputs, int cbSize);

[StructLayout(LayoutKind.Sequential)]
struct INPUT
{
    public SendInputEventType type;
    public MouseKeybdhardwareInputUnion mkhi;
}
[StructLayout(LayoutKind.Explicit)]
struct MouseKeybdhardwareInputUnion
{
    [FieldOffset(0)]
    public MouseInputData mi;

    [FieldOffset(0)]
    public KEYBDINPUT ki;

    [FieldOffset(0)]
    public HARDWAREINPUT hi;
}
[StructLayout(LayoutKind.Sequential)]
struct KEYBDINPUT
{
    public ushort wVk;
    public ushort wScan;
    public uint dwFlags;
    public uint time;
    public IntPtr dwExtraInfo;
}
[StructLayout(LayoutKind.Sequential)]
struct HARDWAREINPUT
{
    public int uMsg;
    public short wParamL;
    public short wParamH;
}
struct MouseInputData
{
    public int dx;
    public int dy;
    public uint mouseData;
    public MouseEventFlags dwFlags;
    public uint time;
    public IntPtr dwExtraInfo;
}
[Flags]
enum MouseEventFlags : uint
{
    MOUSEEVENTF_MOVE = 0x0001,
    MOUSEEVENTF_LEFTDOWN = 0x0002,
    MOUSEEVENTF_LEFTUP = 0x0004,
    MOUSEEVENTF_RIGHTDOWN = 0x0008,
    MOUSEEVENTF_RIGHTUP = 0x0010,
    MOUSEEVENTF_MIDDLEDOWN = 0x0020,
    MOUSEEVENTF_MIDDLEUP = 0x0040,
    MOUSEEVENTF_XDOWN = 0x0080,
    MOUSEEVENTF_XUP = 0x0100,
    MOUSEEVENTF_WHEEL = 0x0800,
    MOUSEEVENTF_VIRTUALDESK = 0x4000,
    MOUSEEVENTF_ABSOLUTE = 0x8000
}
enum SendInputEventType : int
{
    InputMouse,
    InputKeyboard,
    InputHardware
}

public static void ClickLeftMouseButton()
{

    INPUT mouseDownInput = new INPUT();
    mouseDownInput.type = SendInputEventType.InputMouse;
    mouseDownInput.mkhi.mi.dwFlags = MouseEventFlags.MOUSEEVENTF_LEFTDOWN;
    SendInput(1, ref mouseDownInput, Marshal.SizeOf(new INPUT()));

    INPUT mouseUpInput = new INPUT();
    mouseUpInput.type = SendInputEventType.InputMouse;
    mouseUpInput.mkhi.mi.dwFlags = MouseEventFlags.MOUSEEVENTF_LEFTUP;
    SendInput(1, ref mouseUpInput, Marshal.SizeOf(new INPUT()));
}

This will Left click on any element named "elementid" in a loaded webBrowser control. Notice the other MouseEventFlags that can easily be implemented:

Here's a example for right click:

public static void ClickRightMouseButton()
{
    INPUT mouseDownInput = new INPUT();
    mouseDownInput.type = SendInputEventType.InputMouse;
    mouseDownInput.mkhi.mi.dwFlags = MouseEventFlags.MOUSEEVENTF_RIGHTDOWN;
    SendInput(1, ref mouseDownInput, Marshal.SizeOf(new INPUT()));

    INPUT mouseUpInput = new INPUT();
    mouseUpInput.type = SendInputEventType.InputMouse;
    mouseUpInput.mkhi.mi.dwFlags = MouseEventFlags.MOUSEEVENTF_RIGHTUP;
    SendInput(1, ref mouseUpInput, Marshal.SizeOf(new INPUT()));
}