Debugging release build on a client's machine

Question

We have a native C++ Win32 .exe built using Visual Studio 2005 that works flawlessly on all the machines we've tested in-house on (XP 32-bit, Vista 32-bit & Windows 7 64-bit). But of course, it crashes repeatedly on a client's 32-bit Vista machine.

Digging around on several websites I found tidbits that indicate if I ship the PDB files (vc80.pdb & projectName.pdb) along with the Release build of the executable to the customer, there is some way of generating minidumps when a crash occurs. I can then load the crash dump into Visual Studio and get a stack trace and some other useful information. Microsoft's Dr. Watson utility also seems to be involved in this process.

But I can't find any clear instructions on the steps to follow to make this happen

• what files to ship?
• how to get a crash dump to be generated?
• and how to load this into VS?

Can someone please describe this process?

Answer 1

We are able to get crash dumps from our Release builds out in the field and do not need to ship the pdb files with our product.

We build in calls to create the crash dump file ourselves in our top-level exception handler using MiniDumpWriteDump(). But even without that, you could have the user generate the crash file at the point of the crash using task manager as documented here: MSDN Instructions for creating dump file.

Once you have the dump file, the customer zips and mails it to you and you drop it onto Visual Studio. Within VS, you then pick Debug with Mixed or Debug with Native Only and it uses your local copy of the pdb files to show you the call stack, etc.

Answer 2

The process you should have looks like this:

1. Compile an executable and generate a PDB file. Make sure you don't make any changes to the code used for the executable, or keep a backup.
2. Ship the executable to the client. No need to ship the PDB file. The only reason to do ship it is if you'd like to debug on the client's machine, or use tools like Process Explorer to get a stack trace with function names at some point. Neither seem to be applicable in your case.
3. If it's an XP/2003 machine, use drwtsn32 to configure the creation of a crash dump. If it's Vista/7/2008, drwtsn32 is retired, and you should configure WER instead. Another option is using ADPlus to launch you app.
4. Once the crash occurs, have the dump delivered back to you, and load it in Visual Studio. You must have the exact same code, executable and PDB at hand in order to debug smoothly.

Note:

• WinDbg is useful for debugging on production environment. It is a very strong debugger and it's portable, but if you're used to VS, you'll be more comfortable using it.
• If you create a minidump, you'll get stack traces and some variables values. If you create a full dump, you'll get the complete heap, including all variables - and a much larger dump file... If the transfer is not a problem, use full dumps.
• If you register at Microsoft, you can have access to dumps created when your program crashes on clients sites. It's that annoying "Send the info to Microsoft?" window you get when a process crashes that will send the dump to MS, and you'll have access to it...

Answer 3

I feel your pain. Had to do that a while ago.

Anyway, have you tried google Breakpad ?

Breakpad is a library and tool suite that allows you to distribute an application to users with compiler-provided debugging information removed, record crashes in compact "minidump" files, send them back to your server, and produce C and C++ stack traces from these minidumps. Breakpad can also write minidumps on request for programs that have not crashed.

Breakpad is currently used by Google Chrome, Firefox, Google Picasa, Camino, Google Earth, and other projects

You can find it here : http://code.google.com/p/google-breakpad/

It does the same things as the other answers mentioned, but it does it automatically, saviong you a lot of time and efforts

Answer 4

I did this some time ago and I think I followed this guide. If it doesn't work, check out the Windbg utility, I recall you don't need to install it, just copy and run, even from a USB stick

Answer 5

I wrote 2 articles on DDj about post mortem debugging, that might help you:

Postmortem Debugging http://drdobbs.com/tools/185300443

and Post-Mortem Debugging Revisited http://drdobbs.com/architecture-and-design/227900186

the second article contains source code to fill stack dumps from a client machine with symbolic information retrieved either from map or pdb files.

Answer 6

If your client is using Vista SP1 or higher, you can configure the system to generate a local dump file each time an application crashes. You can find the full documentation on the MSDN site.

You can keep all your PDBs private. Those are only needed when you actually analyze the dump file. If you want to keep track of the PDBs corresponding to the versions of the product you ship, you should strongly consider using a symbol server.