Remembering and replaying keystrokes in C# : synchronisation issues

Question

I am a speedrunner (somebody who likes to finish games in the fastest way possible) on PC games and I'd like to record my inputs while I play in order to automatically replay a run later. So I've created a little C# program to do that : basically, it starts a timer and each time I press/release a key, it saves the action (keyup/keydown), the key and at which millisecond I did that. Then, when I want to play it again, it launches a timer, and when it reaches a millisecond on which a keystroke happen, it reproduce it.

And it works ! Well... In fact, it nearly works : the keys are well-reproduced, but sometimes, they differs a little, leading to an unexpected death where I succeeded before.

Here is a video showing the problem : https://www.youtube.com/watch?v=4RPkcx68hpw&feature=youtu.be

The video on the top is the reproduced keys, the video on the bottom is the original play. Everything seems similar, until the 3rd room, where the original play hits the "spider" and make it turn back, while the reproduced keys doesn't touch it, so it distrubs the rest of the progression. Of course, this part of the game is 100% deterministic, so the same inputs would lead to the same results. By advancing the video frame by frame in my video editor, I clearly see a 2 frame gap when the character climb on the first crate, and this gap continues to grow.

Here is my (heavily commented) code :

KeysSaver.cs, the class that saves my inputs

class KeysSaver
{
    public static IntPtr KEYUP = (IntPtr)0x0101; // Code of the "key up" signal
    public static IntPtr KEYDOWN = (IntPtr)0x0100; // Code of the "key down" signal
    private Stopwatch watch; // Timer used to trace at which millisecond each key have been pressed
    private Dictionary<long, Dictionary<Keys, IntPtr>> savedKeys; // Recorded keys activity, indexed by the millisecond the have been pressed. The activity is indexed by the concerned key ("Keys" type) and is associated with the activity code (0x0101 for "key up", 0x0100 for "key down").
    private IntPtr hookId; // Hook used to listen to the keyboard

    private delegate IntPtr LowLevelKeyboardProc(int nCode, IntPtr wParam, IntPtr lParam); // Imported type : LowLevelKeyboardProc. Now we can use this type.


    /*
     * Constructor 
     */
    public KeysSaver()
    {
        this.savedKeys = new Dictionary<long, Dictionary<Keys, IntPtr>>();
        this.watch = new Stopwatch();
    }

    /*
     * method Start()
     * Description : starts to save the keyboard inputs.
     * See : https://msdn.microsoft.com/en-us/library/windows/desktop/ms644990%28v=vs.85%29.aspx
     */

    public void Start()
    {
        using (Process curProcess = Process.GetCurrentProcess())
        using (ProcessModule curModule = curProcess.MainModule) // Get the actual thread
        {
            // Installs a hook to the keyboard (the "13" params means "keyboard", see the link above for the codes), by saying "Hey, I want the function 'onActivity' being called at each activity. You can find this function in the actual thread (GetModuleHandle(curModule.ModuleName)), and you listen to the keyboard activity of ALL the treads (code : 0)
            this.hookId = SetWindowsHookEx(13, onActivity, GetModuleHandle(curModule.ModuleName), 0);
        }
        this.watch.Start(); // Starts the timer
    }

    /*
     * method Stop()
     * Description : stops to save the keyboard inputs.
     * Returns : the recorded keys activity since Start().
     */
    public Dictionary<long, Dictionary<Keys, IntPtr>> Stop()
    {
        this.watch.Stop(); // Stops the timer
        UnhookWindowsHookEx(this.hookId); //Uninstalls the hook of the keyboard (the one we installed in Start())
        return this.savedKeys;
    }

    /*
     * method onActivity()
     * Description : function called each time there is a keyboard activity (key up of key down). Saves the detected activity and the time at the moment it have been done.
     * @nCode : Validity code. If >= 0, we can use the information, otherwise we have to let it.
     * @wParam : Activity that have been detected (keyup or keydown). Must be compared to KeysSaver.KEYUP and KeysSaver.KEYDOWN to see what activity it is.
     * @lParam : (once read and casted) Key of the keyboard that have been triggered.
     * See : https://msdn.microsoft.com/en-us/library/windows/desktop/ms644985%28v=vs.85%29.aspx (for this function documentation)
     * See : https://msdn.microsoft.com/en-us/library/windows/desktop/ms644974%28v=vs.85%29.aspx (for CallNextHookEx documentation)
     */
    private IntPtr onActivity(int nCode, IntPtr wParam, IntPtr lParam)
    {
        if (nCode >= 0) //We check the validity of the informations. If >= 0, we can use them.
        {
            long time = this.watch.ElapsedMilliseconds; //Number of milliseconds elapsed since we called the Start() method
            int vkCode = Marshal.ReadInt32(lParam); //We read the value associated with the pointer (?)
            Keys key = (Keys)vkCode; //We convert the int to the Keys type
            if (!this.savedKeys.ContainsKey(time))
            {
                // If no key activity have been detected for this millisecond yet, we create the entry in the savedKeys Dictionnary
                this.savedKeys.Add(time, new Dictionary<Keys, IntPtr>());
            }
            this.savedKeys[time].Add(key, wParam); //Saves the key and the activity
        }
        return CallNextHookEx(IntPtr.Zero, nCode, wParam, lParam); //Bubbles the informations for others applications using similar hooks
    }

    // Importation of native libraries
    [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
    private static extern IntPtr SetWindowsHookEx(int idHook, LowLevelKeyboardProc lpfn, IntPtr hMod, uint dwThreadId);

    [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
    [return: MarshalAs(UnmanagedType.Bool)]
    private static extern bool UnhookWindowsHookEx(IntPtr hhk);

    [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
    private static extern IntPtr CallNextHookEx(IntPtr hhk, int nCode,
        IntPtr wParam, IntPtr lParam);

    [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
    private static extern IntPtr GetModuleHandle(string lpModuleName);
}

KeysPlayer.cs, the one which simulate key events.

class KeysPlayer
    {
        private Dictionary<long, Dictionary<Keys, IntPtr>> keysToPlay; // Keys to play, with the timing. See KeysSaver.savedKeys for more informations.
        private Dictionary<long, INPUT[]> playedKeys; // The inputs that will be played. This is a "translation" of keysToPlay, transforming Keys into Inputs.
        private Stopwatch watch; // Timer used to respect the strokes timing.
        private long currentFrame; // While playing, keeps the last keysToPlay frame that have been played.

        /*
         * Constructor 
         */
        public KeysPlayer(Dictionary<long, Dictionary<Keys, IntPtr>> keysToPlay)
        {
            this.keysToPlay = keysToPlay;
            this.playedKeys = new Dictionary<long, INPUT[]>();
            this.watch = new Stopwatch();
            this.currentFrame = 0;
            this.loadPlayedKeys(); //Load the keys that will be played.
        }

        /*
         * method Start()
         * Description : starts to play the keyboard inputs.
         */
        public void Start()
        {
            this.currentFrame = 0;  //currentFrame is 0 at the beginning.
            this.watch.Reset(); //Resets the timer
            this.watch.Start(); //Starts the timer (yeah, pretty obvious)
            IEnumerator<long> enumerator = this.playedKeys.Keys.GetEnumerator(); //The playedKeys enumerator. Used to jump from one frame to another.
            long t; //Will receive the elapsed milliseconds, to track desync.
            while (enumerator.MoveNext()) //Moves the pointer of the playedKeys dictionnary to the next entry (so, to the next frame).
            {
                Thread.Sleep((int)(enumerator.Current - this.currentFrame - 1)); //The thread sleeps until the millisecond before the next frame. For exemple, if there is an input at the 42th millisecond, the thread will sleep to the 41st millisecond. Seems optionnal, since we have a "while" that waits, but it allows to consume less ressources. Also, in a too long "while", the processor tends to "forget" the thread for a long time, resulting in desyncs.
                while (this.watch.ElapsedMilliseconds < enumerator.Current) { } //We wait until the very precise millisecond that we want
                t = this.watch.ElapsedMilliseconds; //We save the actual millisecond
                uint err = SendInput((UInt32)this.playedKeys[enumerator.Current].Length, this.playedKeys[enumerator.Current], Marshal.SizeOf(typeof(INPUT))); //Simulate the inputs of the actual frame
                if (t != enumerator.Current) // We compare the saved time with the supposed millisecond. If they are different, we have a desync, so we log some infos to track the bug.
                {
                    Console.WriteLine("DESYNC : " + t + "/" + enumerator.Current + " - Inputs : " + err);
                }
                this.currentFrame = enumerator.Current; //Updates the currentFrame to the frame we just played.
            }
        }

        /*
         * method Stop()
         * Description : stops to play the keyboard inputs.
         */
        public void Stop()
        {
            this.watch.Stop(); //Stops the timer.
        }

        /*
         * method loadPlayedKeys()
         * Description : Transforms the keysToPlay dictionnary into a sequence of inputs. Also, pre-load the inputs we need (loading takes a bit of time that could lead to desyncs).
         */
        private void loadPlayedKeys()
        {
            foreach (KeyValuePair<long, Dictionary<Keys, IntPtr>> kvp in this.keysToPlay)
            {
                List<INPUT> inputs = new List<INPUT>(); //For each recorded frame, creates a list of inputs
                foreach (KeyValuePair<Keys, IntPtr> kvp2 in kvp.Value)
                {
                    inputs.Add(this.loadKey(kvp2.Key, this.intPtrToFlags(kvp2.Value))); //Load the key that will be played and adds it to the list. 
                }
                this.playedKeys.Add(kvp.Key, inputs.ToArray());//Transforms the list into an array and adds it to the playedKeys "partition".
            }
        }

        /*
         * method intPtrToFlags()
         * Description : Translate the IntPtr which references the activity (keydown/keyup) into input flags.
         */
        private UInt32 intPtrToFlags(IntPtr activity)
        {
            if (activity == KeysSaver.KEYDOWN) //Todo : extended keys
            {
                return 0;
            }
            if (activity == KeysSaver.KEYUP)
            {
                return 0x0002;
            }
            return 0;
        }

        /*
         * method loadKey()
         * Description : Transforms the Key into a sendable input (using the above structures).
         */
        private INPUT loadKey(Keys key, UInt32 flags)
        {
            return new INPUT
            {
                Type = 1, //1 = "this is a keyboad event"
                Data =
                {
                    Keyboard = new KEYBDINPUT
                    {
                        KeyCode = (UInt16)key,
                        Scan = 0,
                        Flags = flags,
                        Time = 0,
                        ExtraInfo = IntPtr.Zero
                    }
                }

            };
        }

        // Importation of native libraries
        [DllImport("user32.dll", SetLastError = true)]
        public static extern UInt32 SendInput(UInt32 numberOfInputs, INPUT[] inputs, Int32 sizeOfInputStructure);

        [DllImport("kernel32.dll")]
        static extern uint GetLastError();

    }
}

All the structs used by SendInput (those are copied from the InputSimulator script) :

/*
     * Struct MOUSEINPUT
     * Mouse internal input struct
     * See : https://msdn.microsoft.com/en-us/library/windows/desktop/ms646273(v=vs.85).aspx
     */
    internal struct MOUSEINPUT
    {
        public Int32 X;
        public Int32 Y;
        public UInt32 MouseData;
        public UInt32 Flags;
        public UInt32 Time;
        public IntPtr ExtraInfo;
    }

    /*
     * Struct HARDWAREINPUT
     * Hardware internal input struct
     * See : https://msdn.microsoft.com/en-us/library/windows/desktop/ms646269(v=vs.85).aspx
     */
    internal struct HARDWAREINPUT
    {
        public UInt32 Msg;
        public UInt16 ParamL;
        public UInt16 ParamH;
    }

    /*
     * Struct KEYBDINPUT
     * Keyboard internal input struct (Yes, actually only this one is used, but we need the 2 others to properly send inputs)
     * See : https://msdn.microsoft.com/en-us/library/windows/desktop/ms646271(v=vs.85).aspx
     */
    internal struct KEYBDINPUT
    {
        public UInt16 KeyCode; //The keycode of the triggered key. See https://msdn.microsoft.com/en-us/library/windows/desktop/dd375731(v=vs.85).aspx
        public UInt16 Scan; //Unicode character in some keys (when flags are saying "hey, this is unicode"). Ununsed in our case.
        public UInt32 Flags; //Type of action (keyup or keydown). Specifies too if the key is a "special" key.
        public UInt32 Time; //Timestamp of the event. Ununsed in our case.
        public IntPtr ExtraInfo; //Extra information (yeah, it wasn't that hard to guess). Ununsed in our case.
    }

    /*
     * Struct MOUSEKEYBDHARDWAREINPUT
     * Union struct for key sending 
     * See : https://msdn.microsoft.com/en-us/library/windows/desktop/ms646270%28v=vs.85%29.aspx
     */

    [StructLayout(LayoutKind.Explicit)]
    internal struct MOUSEKEYBDHARDWAREINPUT
    {
        [FieldOffset(0)]
        public MOUSEINPUT Mouse;

        [FieldOffset(0)]
        public KEYBDINPUT Keyboard;

        [FieldOffset(0)]
        public HARDWAREINPUT Hardware;
    }

    /*
     * Struct INPUT
     * Input internal struct for key sending 
     * See : https://msdn.microsoft.com/en-us/library/windows/desktop/ms646270%28v=vs.85%29.aspx
     */

    internal struct INPUT
    {
        public UInt32 Type; //Type of the input (0 = Mouse, 1 = Keyboard, 2 = Hardware)
        public MOUSEKEYBDHARDWAREINPUT Data; //The union of "Mouse/Keyboard/Hardware". Only one is read, depending of the type.
    }

And my main form :

public partial class Taslagrad : Form
    {
        private KeysSaver k;
        private KeysPlayer p;

        //Initialisation 
        public Taslagrad()
        {
            InitializeComponent();
            this.k = new KeysSaver();
        }

        /*
         * method launchRecording()
         * Description : Starts to record the keys. Called when the "record" button is triggered.
         */
        private void launchRecording(object sender, EventArgs e)
        {
            this.k.Start(); //Starts to save the keys
            startButton.Text = "Stop"; //Updates the button
            startButton.Click -= launchRecording;
            startButton.Click += stopRecording;
        }

        /*
         * method stopRecording()
         * Description : Stops to record the keys and logs the recorded keys in the console. Called when the "record" button is triggered.
         */
        private void stopRecording(object sender, EventArgs e)
        {
            startButton.Text = "Record";//Updates the button
            startButton.Click += launchRecording;
            startButton.Click -= stopRecording;
            Dictionary<long, Dictionary<Keys, IntPtr>> keys = this.k.Stop(); //Gets the recorded keys
            foreach (KeyValuePair<long, Dictionary<Keys, IntPtr>> kvp in keys)
            {
                foreach (KeyValuePair<Keys, IntPtr> kvp2 in kvp.Value)
                {
                    //Displays the recorded keys in the console
                    if (kvp2.Value == KeysSaver.KEYDOWN)
                    {
                        Console.WriteLine(kvp.Key + " : (down)" + kvp2.Key);
                    }
                    if (kvp2.Value == KeysSaver.KEYUP)
                    {
                        Console.WriteLine(kvp.Key + " : (up)" + kvp2.Key);
                    }
                }
            }
            this.p = new KeysPlayer(keys); //Creates a new player and gives it the recorded keys.
        }

        /*
         * method launchPlaying()
         * Description : Starts to play the keys. Called when the "play" button is triggered.
         */
        private void launchPlaying(object sender, EventArgs e)
        {
            this.p.Start(); //Starts to play the keys.
        }
}

Of course, all my debugs seems to work properly : the recorder saves all the inputs (I tested by typing a long text) and when I compare the milliseconds at which the keys are recorded and at which they are played, I have no difference...

So is there a problem in the way I record/play ? Is the StopWatch not precise enough? Is there a more precise/effective way?

Answer 1

I happened to appreciate MetalFoxDoS's work. At first glance it was a little rough around the edges but the gist of the work is there.

The honest answer is yes, he was precision limited and running a bit more CPU % than necessary for NO-OP if there was a delay on users providing inputs. Sorting this out while taking an async approach has given me microsecond accuracy - far beyond anything a human could provide for accuracy.

I have continued to enhance this work in an attempt to refine it further as I appreciated the need to create "Framing" based off of delays between keypress events. I have also corrected a few mistakes/changes required from the past few years of gap.

https://github.com/houseofcat/WinKeyRecorder