Are there any tools/libraries (.Net/WPF) to measure and store UI navigation data for analysis?

Question

I want to measure and analyze user movements and gestures in the UI in order to refine the application user experience. I had imagined that feature-tracking libraries (like EQATEC or Preemptive's Runtime Intelligence) would allow this. However, this does not appear to be the case.

Ideally, I'd like to be able to instrument a UI and then capture mouse and keyboard navigation gestures to display via a heat-map.

My searches have come up empty. Does anything OSS or commercial exist here?

Answer 1

1. Download Sources Version 2 from this article on code project
2. Open the solution or only Gma.UserActivityMonitor project and blindly convert it to .NET 4.0

3. In HookManager.Callbacks.cs file make following changes.

   1. Add using System.Diagnostics;

   2. Replace

      s_MouseHookHandle = SetWindowsHookEx(
          WH_MOUSE_LL,
          s_MouseDelegate,
          Marshal.GetHINSTANCE(
              Assembly.GetExecutingAssembly().GetModules()[0]),
          0);
      

      With

      using (Process curProcess = Process.GetCurrentProcess())
      using (ProcessModule curModule = curProcess.MainModule)
      {
          s_MouseHookHandle = SetWindowsHookEx(
              WH_MOUSE_LL,
              s_MouseDelegate,
             GetModuleHandle(curModule.ModuleName), 0);
      }
      

   3. Replace

      s_KeyboardHookHandle = SetWindowsHookEx(
          WH_KEYBOARD_LL,
          s_KeyboardDelegate,
          Marshal.GetHINSTANCE(
              Assembly.GetExecutingAssembly().GetModules()[0]),
          0);
      

      With

      using (Process curProcess = Process.GetCurrentProcess())
      using (ProcessModule curModule = curProcess.MainModule)
      {
          s_KeyboardHookHandle = SetWindowsHookEx(
          WH_KEYBOARD_LL,
          s_KeyboardDelegate, 
          GetModuleHandle(curModule.ModuleName), 0);
      }
      

4. In HookManager.Windows.cs add following two lines anywhere in HookManager class's definition.

   [DllImport("kernel32.dll", CharSet = CharSet.Auto)]
   public static extern IntPtr GetModuleHandle(string lpModuleName);
   

5. Now you should be able to build this and keep it aside. Now starts the WPF part.

6. Create new WPF project with name WpfApplication1 preferably. Add reference to the project/assembly you built in step 1-5, add reference to System.Windows.Forms.

7. Now Replace the MainWindow.xaml with following XAML make sure to check class name and project-name, I just created WpfApplication1 for testing.

   <Window x:Class="WpfApplication1.MainWindow"
           xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
           xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
           Title="MainWindow" Height="350" Width="525">
       <Grid>
           <Grid.RowDefinitions>
               <RowDefinition Height="0.30*"/>
               <RowDefinition Height="0.70*"/>
           </Grid.RowDefinitions>
           <StackPanel Grid.Row="0" Orientation="Horizontal">
               <StackPanel Orientation="Vertical">
                   <CheckBox Name="MouseMove" Padding="5" Content="Mouse Move" Width="120" Height="30" Click="checkBoxOnMouseMove_CheckedChanged"/>
                   <CheckBox Name="MouseClick" Padding="5" Content="Mouse Click" Width="120" Height="30" Click="checkBoxOnMouseClick_CheckedChanged"/>
                   <CheckBox Name="MouseDown" Padding="5" Content="Mouse Down" Width="120" Height="30" Click="checkBoxOnMouseDown_CheckedChanged"/>
               </StackPanel>
               <StackPanel Orientation="Vertical">
                   <CheckBox Name="MouseUp" Padding="5" Content="Mouse Up" Width="120" Height="30" Click="checkBoxOnMouseUp_CheckedChanged"/>
                   <CheckBox Name="MouseDouble" Padding="5" Content="Mouse Double" Width="120" Height="30" Click="checkBoxMouseDoubleClick_CheckedChanged"/>
                   <CheckBox Name="MouseWheel" Padding="5" Content="Mouse Wheel" Width="120" Height="30" Click="checkBoxMouseWheel_CheckedChanged"/>
               </StackPanel>
               <StackPanel Orientation="Vertical">
                   <CheckBox Name="KeyDown" Padding="5" Content="Key Down" Width="120" Height="30" Click="checkBoxKeyDown_CheckedChanged"/>
                   <CheckBox Name="KeyPress" Padding="5" Content="Key Press" Width="120" Height="30" Click="checkBoxKeyPress_CheckedChanged"/>
                   <CheckBox Name="KeyUp" Padding="5" Content="Key Up" Width="120" Height="30" Click="checkBoxKeyUp_CheckedChanged"/>
               </StackPanel>
               <StackPanel Orientation="Vertical">
                   <TextBlock Name="labelMousePosition" Text="x={0:####}; y={1:####}"/>
                   <TextBlock Name="labelWheel" Text="Wheel={0:####}"/>
               </StackPanel>
           </StackPanel>
           <TextBlock Name="textBoxLog" Text="START" VerticalAlignment="Stretch" HorizontalAlignment="Stretch" Grid.Row="1" ScrollViewer.VerticalScrollBarVisibility="Visible"/>
       </Grid>
   </Window>
   

8. Now add following code to MainWindow class defined MainWindow.xaml.cs file.

   #region Check boxes to set or remove particular event handlers.
   
   private void checkBoxOnMouseMove_CheckedChanged(object sender, EventArgs e)
   {
       if ((bool)MouseMove.IsChecked)
       {
           HookManager.MouseMove += HookManager_MouseMove;
       }
       else
       {
           HookManager.MouseMove -= HookManager_MouseMove;
       }
   }
   
   private void checkBoxOnMouseClick_CheckedChanged(object sender, EventArgs e)
   {
       if ((bool)MouseClick.IsChecked)
       {
           HookManager.MouseClick += HookManager_MouseClick;
       }
       else
       {
           HookManager.MouseClick -= HookManager_MouseClick;
       }
   }
   
   private void checkBoxOnMouseUp_CheckedChanged(object sender, EventArgs e)
   {
       if ((bool)MouseUp.IsChecked)
       {
           HookManager.MouseUp += HookManager_MouseUp;
       }
       else
       {
           HookManager.MouseUp -= HookManager_MouseUp;
       }
   }
   
   private void checkBoxOnMouseDown_CheckedChanged(object sender, EventArgs e)
   {
       if ((bool)MouseDown.IsChecked)
       {
           HookManager.MouseDown += HookManager_MouseDown;
       }
       else
       {
           HookManager.MouseDown -= HookManager_MouseDown;
       }
   }
   
   private void checkBoxMouseDoubleClick_CheckedChanged(object sender, EventArgs e)
   {
       if ((bool)this.MouseDouble.IsChecked)
       {
           HookManager.MouseDoubleClick += HookManager_MouseDoubleClick;
       }
       else
       {
           HookManager.MouseDoubleClick -= HookManager_MouseDoubleClick;
       }
   }
   
   private void checkBoxMouseWheel_CheckedChanged(object sender, EventArgs e)
   {
       if ((bool)MouseWheel.IsChecked)
       {
           HookManager.MouseWheel += HookManager_MouseWheel;
       }
       else
       {
           HookManager.MouseWheel -= HookManager_MouseWheel;
       }
   }
   
   private void checkBoxKeyDown_CheckedChanged(object sender, EventArgs e)
   {
       if ((bool)KeyDown.IsChecked)
       {
           HookManager.KeyDown += HookManager_KeyDown;
       }
       else
       {
           HookManager.KeyDown -= HookManager_KeyDown;
       }
   }
   
   
   private void checkBoxKeyUp_CheckedChanged(object sender, EventArgs e)
   {
       if ((bool)KeyUp.IsChecked)
       {
           HookManager.KeyUp += HookManager_KeyUp;
       }
       else
       {
           HookManager.KeyUp -= HookManager_KeyUp;
       }
   }
   
   private void checkBoxKeyPress_CheckedChanged(object sender, EventArgs e)
   {
       if ((bool)KeyPress.IsChecked)
       {
           HookManager.KeyPress += HookManager_KeyPress;
       }
       else
       {
           HookManager.KeyPress -= HookManager_KeyPress;
       }
   }
   
   #endregion
   
   //##################################################################
   #region Event handlers of particular events. They will be activated when an appropriate check box is checked.
   
   private void HookManager_KeyDown(object sender, System.Windows.Forms.KeyEventArgs e)
   {
       textBoxLog.Text += (string.Format("KeyDown - {0}\n", e.KeyCode));
   
   }
   
   private void HookManager_KeyUp(object sender, System.Windows.Forms.KeyEventArgs e)
   {
       textBoxLog.Text += (string.Format("KeyUp - {0}\n", e.KeyCode));
   
   }
   
   
   private void HookManager_KeyPress(object sender, System.Windows.Forms.KeyPressEventArgs e)
   {
       textBoxLog.Text += (string.Format("KeyPress - {0}\n", e.KeyChar));
   
   }
   
   
   private void HookManager_MouseMove(object sender, System.Windows.Forms.MouseEventArgs e)
   {
       labelMousePosition.Text = string.Format("x={0:0000}; y={1:0000}", e.X, e.Y);
   }
   
   private void HookManager_MouseClick(object sender, System.Windows.Forms.MouseEventArgs e)
   {
       textBoxLog.Text += (string.Format("MouseClick - {0}\n", e.Button));
   
   }
   
   
   private void HookManager_MouseUp(object sender, System.Windows.Forms.MouseEventArgs e)
   {
       textBoxLog.Text += (string.Format("MouseUp - {0}\n", e.Button));
   
   }
   
   
   private void HookManager_MouseDown(object sender, System.Windows.Forms.MouseEventArgs e)
   {
       textBoxLog.Text += (string.Format("MouseDown - {0}\n", e.Button));
   
   }
   
   
   private void HookManager_MouseDoubleClick(object sender, System.Windows.Forms.MouseEventArgs e)
   {
       textBoxLog.Text += (string.Format("MouseDoubleClick - {0}\n", e.Button));
   
   }
   
   
   private void HookManager_MouseWheel(object sender, System.Windows.Forms.MouseEventArgs e)
   {
       labelWheel.Text = string.Format("Wheel={0:000}", e.Delta);
   }
   
   #endregion
   

9.Build and run your WPF App, you might need to add using Gma.UserActivityMonitor; directive in MainWindow.

10.All the credit goes to George Mamaladze who originally built it, you might want to send a thank you note to him, also check the distribution license for this code from original author if you want to make money out of it.

11.I just made couple of minor changes to make it work with WPF. Thanks for following this long post, I'm not sure how to share this code, that is why providing instructions like this.

12.I'm seriously sucking at formatting the code well on SO, can someone leave a comment on how to format code, XML well. Also someone help me format snippets in this post. Thank you.