Device misdetected as serial mouse

Question

I'm working on a device which communicates with a PC through a (virtual) serial port. The problem is that the data we are sending occasionally gets incorrectly identified by Windows as a bus mouse, after which the "Microsoft Serial Ballpoint" driver is loaded and the mouse pointer starts jumping around on the screen and randomly clicking on things.

A bit of Googling reveals that is an old and well-known problem with serial devices where the usual work-around is a bit of registry hacking to disable the offending driver. That it is a lot to demand from our users however and I'd rather not have our application messing around with the user's registry. Especially not when the fix is dependent on the Windows version and the user may well be using a bus mouse.

Instead I'd like to avoid the problem by changing our protocol to not send any data which may get us misidentified as a mouse. The only problem is that I'm not quite certain what patterns to avoid. Apparently Microsoft's Mouse protocol consists of packets of four bytes where the MSB of the first is set and that of the last three is clear.

Would sending only 7-bit ASCII suffice? Are there any other devices I need to worry about being detected as?

Answer 1

I just encountered this problem myself on Windows 7 Professional x64, and a solution that worked for me was to go into the registry and edit the following value:

Location: HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\sermouse   Key: Start   Value: 3 

Change Value to 4 and it will stop this problem occurring.

Here is a list of all valid Start values:

0 Boot (loaded by kernel loader). Components of the driver stack for the boot (startup) volume must be loaded by the kernel loader.  1 System (loaded by I/O subsystem). Specifies that the driver is loaded at kernel initialization.  2 Automatic (loaded by Service Control Manager). Specifies that the service is loaded or started automatically.  3 Manual. Specifies that the service does not start until the user starts it manually, such as by using Device Manager.  4 Disabled. Specifies that the service should not be started. 

---

A reg edit command would be as follows:

REG ADD "HKLM\SYSTEM\CurrentControlSet\Services\sermouse" /V Start /T REG_DWORD /F /D 4 

You then need to restart the computer, which should now start correctly and not attempt to discover a serial mouse.

good luck.

Answer 2

It turns out that mouse detection in Windows is normally handled by the serenum.sys filter driver. This driver implements support for legacy serial mice along with serial plug-and-play. Microsoft has even provided the sourcecode as a WDK sample.

During detection the ports switches to 1200-7-N-1 mode while asserting DTR+RTS to which a response is expected within 200 ms, with a couple of retries in case of failure. Unfortunately for a legacy mouse a single M or B character suffices as identification.

In our case the protocol was reworked to avoid these characters and now appears not to be misidentified anymore.

However we were using a virtual USB serial port and for a traditional serial port this approach may be somewhat difficult as anything sent at a different baud rate is liable to look like line noise. In this case I suppose the easiest workaround is probably, as has already been suggested, to avoid making any unsolicited transmissions.

Alternatively with the serial control signals actually hooked up, or intercepted by a USB CDC device, processing the DTR or RTS signals and holding off on output. Actually implementing the plug-and-play protocol would be an even niftier option. Supposedly there are cheap RS232 cables around without a full complement of control signals though so this approach might still fail.