How to use WinDbg to analyze the crash dump for VC++ application?

Question

How do I use WinDbg for analyzing a dump file?

Answer 1

Here are some general steps that will get you on your way:

First, you must change your compiler's settings so that it creates PDB files, even for release builds. Later versions of the Visual C++ compiler do this by default, but in many versions of Visual C++ you must do this yourself. Create program database files, and then keep an archive of those files along with each build of your application. It is critical that every build of your applications has its own set of PDBs. You can't just reuse the same ones you made with build 10 to examining the dumps generated by build 15, for example. Over the life of your project, you will end up with a ton of PDBs, so be prepared for that.

Next, you need to be able to identify the exact version of your application which generated the dump file. If you are creating your own MiniDumps (by calling MiniDumpWriteDump() for example), probably the easiest way to do this is to simply make part of the filename of the MiniDump the complete version number of your application. You'll need to have a reasonable version numbering scheme in place for this to work. In my shop, we increment the build number across all branches by one every time the autobuilder creates a build.

Now that you have received the dump file from the customer, you know the precise version of the application that created the dump, and you have found the PDB files for this build.

Now you need to go through your source control's history and find the source code for this exact version of the software. The best way to do this is to apply 'labels' to your branches every time you make a build. Set the value of the label to the exact version number, and it becomes easy to find in the history.

You're almost ready to fire up WinDbg/Visual C++:

1. Get the complete source tree for that version of your application. Put it in a separate place on your hard drive, say c:\app_build_1.0.100 for application version 1.0 build #100.
2. Get the binaries for that exact version of your application and put them somewhere on your hard drive. It might be easiest simply to install that version of your application to get the binaries.
3. Put the PDB files in the same location as the binaries in step 2.

Now you have two options for viewing the dump file. You can use Visual Studio or WinDbg. Using Visual Studio is easier, but WinDbg is much more powerful. Most of the time the functionality in Visual Studio will suffice.

To use Visual Studio, all you have to do is open the dump file like it is a project. Once opened, "run" the dump file (F5 by default) and if all the paths are set correctly it will take you right to the code that crashed, give you a call stack, etc.

To use WinDbg, you have to jump through a couple of hoops:

1. Start WinDbg
2. Open the dump file. (Ctrl + D by default)
3. Tell WinDbg to go get the correct MicroSoft symbol files. Type .symfix. This may take a few moments as it will pull a ton of stuff down from the Internet.
4. Tell WinDbg where the symbols (PDB files) are. Type .sympath+ c:\pdblocation, substituting wherever you put the PDB files for the pathname. Make sure you get the plus sign in there with no whitespace between .sympath and the + sign or else you'll screw up step 3.
5. Tell WinDbg where the source code is. Type .srcpath c:\app_build_1.0.100 substituting the path where you got code from source control for this version of the software.
6. Tell WinDbg to analyze the dump file. Type !analyze -v

After a few moments, if everything is configured correctly, WinDbg will take you right to the location of your crash. At this point you have a million options for digging deep into your application's memory space, the state of critical sections, windows, etc. But that is way beyond the scope of this post.

Good luck!

Answer 2

(see the "Dump" sections below)

Basic Tutorials and Demonstrations of Using WinDbg

• Installing and Configuring WinDbg (Windows Debug Tools)
• Mike Taulty - A word for WinDBG
• WinDbg Tutorials
• Windows Debuggers: Part 1: A WinDbg Tutorial

Different Ways to "Start"/Attach WinDBG

• Start Debugging with Windbg (includes how to debug an .msi)
• How to debug a Windows service
• Setting up Windows Debugging

Workspaces

Understanding how Workspaces work...

• Pimp up your debugger: Creating a custom workspace for windbg debugging
• Uncovering How Workspaces Work in WinDbg

Cmdtree

A "cmdtree" allows you to define a "menu" of debugger commands for easy access to frequently used commands without having to remember the terse command names.

You don't have to put all the command definitions into the same cmdtree text file....you can keep them separate and load multiple ones if you wish (they then get their own window).

• Amazing helper .cmdtree
• How do I make a cmdtree window dock at startup in WinDBG
• Making it easier to debug .net dumps in windbg using .cmdtree
• Microshaoft Cmdtree
• Special Command—Execute Commands from a Customized User Interface with .cmdtree

Startup Script

You can use the -c option on the command line to automatically run a WinDBG script when you start WinDBG.

Gives opportunity to turn on DML (Debugger markup language) mode, load particular extensions, set .NET exception breakpoints, set kernel flags (e.g. when kernel debugging you might need to change the DbgPrint mask so you see tracing information....ed nt!Kd_DEFAULT_Mask 0xffffffff), load cmdtrees, etc.

• http://yeilho.blogspot.co.uk/2012/10/windbg-init-script.html
• Take Control of WinDBG

An example script:

$$ Include a directory to search for extensions $$ (point to a source controlled or UNC common directory so that all developers get access) .extpath+"c:\svn\DevTools\WinDBG\Extensions" $$ When debugging a driver written with the Windows Driver Framework/KMDF $$ load this extension that comes from the WinDDK. !load C:\WinDDK\7600.16385.1\bin\x86\wdfkd.dll !wdftmffile C:\WinDDK\7600.16385.1\tools\tracing\i386\wdf01009.tmf $$ load some extensions .load msec.dll .load byakugan.dll .load odbgext.dll .load sosex .load psscor4 $$ Make commands that support DML (Debugger Markup Language) use it .prefer_dml 1 .dml_start $$ Show NTSTATUS codes in hex by default .enable_long_status 1 $$ Set default extension .setdll psscor4 $$ Show all loaded extensions .chain /D $$ Load some command trees .cmdtree c:\svn\DevTools\WinDBG\cmdtree\cmdtree1.txt .cmdtree c:\svn\DevTools\WinDBG\cmdtree\cmdtree2.txt $$ Show some help for the extensions !wdfkd.help !psscor4.help .help /D 

Command Cheat Sheets

• Crash Dump Analysis Poster v3.0
• SOS Cheat Sheet (.NET 2.0/3.0/3.5)
• WinDbg cheat sheet (Art of Dev)
• WinDbg Kernel-Mode Extension Commands Flashcards

Extensions

"Extensions" allow you to extend the range of commands/features supported inside WinDBG.

• bigLasagne (bldbgexts & blwdbgue)
  - assembly syntax highlighting and a driver mapping tool)
  
  
• BigLib Number Reader
  
  
• Byakugan
  - detect antidebugging methods, vista heap visualization/emulation, track buffers in memory
  
  
• Call Flow Analyzer + KnExt
  
  
• CmdHist
  - records every command you executed in your debug session so you can re-execute easily
  
  
• Core Analyzer
  - check heap structures for corruption, detect objects shared by threads, etc
  
  
• dom WinDBG Extension
  - (!stlpvector, !idt, !unhex, !grep, etc)
  
  
• dumppe
  - dumps PE file from memory
  
  
• Image Viewer Extension (Vladimir Vukićević)
  
  
• Intel UEFI Development Kit Debugger Tool
  - debug UEFI firmware
  
  
• leaktrap
  - GDI/USER handle tracker to aid in leak detection
  
  
• Mona (requires PyKD)
  - set of commands to aid in advanced analysis/find exploits
  
  
• MSEC
  - provides automated crash analysis and security risk assessment
  
  
• narly
  - lists info about loaded modules such as if using SafeSEH, ASLR, DEP, /GS (Buffer Security Checks)
  
  
• netext (Rodney Viana)
  - (!wservice - list WCF service objects, !wconfig - show .config lines, !whttp - list HttpContexts, !wselect/!wfrom - support SQL like queries on arrays)
  
  
• ODbgExt
  - open debugger extensions
  
  
• OllyMigrate
  - pass debuggee to another debugger without restarting
  
  
• Psscor2
  - a superset of SOS for assisting in debugging .NET 2.0 managed code
  
  
• Psscor4
  - a superset of SOS for assisting in debugging .NET 4 managed code
  
  
• PyDBGExt
  - allows python scripting to be used
  
  
• PyKD
  - allows Python to be used to script WinDBG
  
  
• sdbgext (Nynaeve)
  -(!valloc, !vallocrwx, !heapalloc, !heapfree, !remotecall, !remotecall64, !loaddll, !unloaddll, !close, !killthread, !adjpriv, !ret)
  
  
• SieExtPub
  -legacy extension...now built into WinDBG in ext.dll
  
  
• SOSEX
  - more commands for helping to debug managed NET 2.0 or 4.0 code
  
  
• SPT/SDBGExt2 (Steve Niemitz)
  - (!DumpHttpContext, !DumpASPNetRequests, !DumpSqlConnectionPools, !DumpThreadPool, etc)
  
  
• Uniqstack
  - source to a debugger extension (need an OSR Online account to access it)
  
  
• viscope
  - code coverage graph
  
  
• Wait Chain Traversal/wct.dll (Codeplex Debugging Extensions
  - display wait chains of application threads (helps find deadlocks)
  
  
• windbgshark
  - integrates Wireshark protocol analyser to enable VM traffic manipulation and analysis
  
  
• WinDBG Extensions (Sasha Goldstein)
  - Tracer, WCT, heap_stat, bkb, traverse_map, traverse_vector)
  
  
• WinDBG Highlight (ColorWindbg.dll) [Use Google Translate to translate link]
  - asm syntax highlighting

Write your own extension

• Tools of the Trade: Part IV - Developing WinDbg Extension DLLs
• The Basics of Debugger Extensions: Short Term Effort, Long Term Gain

Using WinDBG to Debug Managed Code

• Breaking on an Exception
• Breaking on specific CLR Exception
• Debugging .Net framework source code within Windbg
• Debugging exceptions in managed code using Windbg
• Debugging managed code using WinDbg and SOS.dll
• Debugging with WinDbg. Deadlocks in Applications.
• MANAGED DEBUGGING with WINDBG. Introduction and Index
• Setting .NET breakpoints in Windbg for applications that crash on startup

Scripting (C#, PS, Python, WinDBG)

• KDAR (Kernel Debugger Anti Rootkit)
  - a collection of WinDBG scripts
  
  
• Sysnative BSOD Scripts/Processing Apps
  
  
• WinDBG Script library
  - a collection of WinDBG scripts
  
  
• Scripting MDbg and DbgHostLib
  - allows managed code to script the Managed Debugger (MDBG) and the DbgEng
  
  
• ExtCS
  - allows control of WinDBG via C# scripts
  
  
• PowerDBG
  - allows control of WinDBG via Powershell scripts
  
  
• Pykd
  - allows control of WinDBG via Python scripts
  
  
• windbglib
  - python wrapper library around the pykd extension for WinDBG, mimicking immlib (so you can use scripts originally written for Immunity Debugger)

Debuggers/Tools that use the dbgeng.dll API/WinDBG Tools

• A Simple Dbgeng Based User Mode Debugger
• Acorns.Debugging NET Deadlock Detector (uses cdb.exe) (download)
• CLR Managed Debugger (MDBG)
• DbgHost - How to control a debugging engine
• Debug Diagnostic Tool v1.2 (DebugDiag), Ver 2.0 + DebugDiag Blog
• Dynamorio - dynamic binary instrumentation tool which can interact with WinDBG
• IDA + WinDBG plugin
• GUI WinDBG
• LeakShell (find managed leaks)
• mdbglib - Managed Debug API
• PyDbgEng
  - python wrapper for Windows Debugging Engine
• SOSNET - a WinDBG Fork/alternative shell that concentrates on using the SOS extension and supports C# scripting
• SOSNET O2 fork - fork of SOSNET that uses Rosyln for the C# REPL (read-eval-print-loop) scripting engine
• VDB/Vivisect (kenshoto) - provides a cross-platform debugging API layered on WinDBG
• WinAppDbg + Heappie-WinAppDbg
• Writing a basic Windows debugger

Different Ways to Generate Crash Dump Files for Post-Mortem Analysis

• DebugDiag 2.0
• Dump Cheat Sheet
  - includes how to generate dump from Hyper-V, VMWare ESX, and XenServer VMs.
• Citrix SystemDump
• Keyboard Keypress Combination
• MiniDumpWriteDump
  - (via WIN32 API call inside your application). (Example for C# applications)
• NMI Switch, or (here)
  (hardware based feature to generate an NMI...usually found on high-end servers e.g. HP or you can obtain an add-in PCI card "Universal PCI Dump Switch"). Microsoft NMI technology background.
• Procdump
• System|Advanced System Settings|Startup and Recovery
  (registry info),
  (how to configure a Complete (Full) Memory Dump),
  (how to enable Complete Memory Dump),
  (how to enable Complete Memory Dump on Windows 7 when PC has lots of memory...normally not available when more than 2GB of memory)
• Task Manager "Create Dump File"
• UserDump, instructions (very old tool)
• UserModeProcessDumper, instructions
• Visual Studio "Save Dump As…"
• WER (Windows Error Reporting....local dumps)
• WinDBG

Dump Analysis Tools

• BlueScreenView - finds the minidump .dmp files saved by Windows after a BSOD, and extracts information about what caused the crash
• Debug.Analyzer (can analyse dump files and plug-ins can be written in .NET)
• SAD - Simple After Dump (postmortem analyzer)
• Volatility - framework for analyzing "memory" recorded in dump files (cheat sheet)

Dump related Tools

• Citrix dumpcheck - checks consistency of dump file (looks like it's been abandoned link + link)
• dumpchk (part of Debugging Tools) - checks consistency of a Dump file
• MoonSols Windows Memory Toolkit (formerly windd) - converts various raw memory dump files into WinDBG compatible dmp files
• vm2dmp - Microsoft Hyper-V VM State to Memory Dump Converter
• vmss2core - converts VMWare snapshot file into a core dump file (download), (instructions)

Kernel Debugging Virtual Machines

• VMKD - Virtual Machine KD Extensions
• VirtualKD - (kernel debugger support for OS's hosted in VMWare/VirtualBox)

Videos

• .NET Cracking 101 #2 - WinDbg basics
• .NET Debugging for the Production Environment (Channel9)
• dotnetConf - Advanced Debugging with WinDbg and SOS
• David Truxall "Debugging with WinDBG"
• Mike Taulty Debugging Memory Leaks
• oredev 2009 Session: Debugging .NET Applications with WinDbg
• Pluralsight Advanced Windows Debugging
  (plus various other ones at Pluralsight)
• Tess Ferrandez WinDBG (Channel9)

Blogs

Some blogs (mixture of native and managed code debugging).

• Advanced .NET Debugging
• All Your Base Are Belong To Us (Sasha Goldshtein)
• Analyze-v
• ASP.NET Debugging
• Cyberiafreak (threading and advanced windows prog and debugging)
• Debug Analyzer.NET
• Debug and Beyond
• Debugging Experts Magazine Online
• Debugging Toolbox (Windbg scripts, debugging and troubleshooting tools and techniques to help you isolate software problems.)
• Decrypt my World
• greggm's WebLog
• Junfeng Zhang's Windows Programming Notes
• Kristoffer's tidbits
• Mark Russinovich's Blog
• Mike Stalls .NET Debugging Blog
• Naveen's Blog
• Never Doubt Thy Debugger (Carlo)
• Notes from a Dark Corner
• Ntdebugging Blog (Microsoft Global Escalation Services team)
• Nynaeve. Adventures in Windows debugging and reverse engineering
• PFE Developer Notes for the Field
• Visual Studio Debugger Team
• WinDbg by Volker von Einem

Advanced Articles and Tutorial Resources

• Advanced Debugging Techniques in WinDbg
• Debugging Applications for MS.Net and Windows (Powerpoint Slides)
• Debugging STL Containers with WinDbg
• Debug Tutorials 1-7 (CodeProject-Toby Opferman)
• Debugging.tv
• Developmentor WinDBG Tagged articles
• Dr Fu's Security Blog - Malware Analysis Tutorials - Reverse Engineering Approach
• Exploit writing tutorial part 5 : How debugger modules & plugins can speed up basic exploit development
• Hunting Rootkits
• Remote Microsoft Windows Server OS Kernel Debugging Using Dell Windows Debugger Utility (DWDU) (DELL(TM) Windows(R) Debugger Utility 1.1 README)

Alternative Debuggers

• Bokken - (Inguma) (GUI for radare)
• BugDbg
• Debug++ (not released yet)
• Debuggy
• Discoloured Ring 0 Debugger (download)
• edb (Linux)
• FDBG
• GoBug
• Hades (Ring 3 debugger with anti debugger detection strategy)
• Hopper (Linux, OSX and Windows) (Windows debugging not currently implemented)
• Hyperdbg
• IDA Debugger
• ImmunityDebugger
• Nanomite
• Obsidian (non-intrusive debugger)
• OllyDBG
• PEBrowse
• RaceVB6 (VB6 P-Code debugger)
• radare
• radare2ui (GUI for radare)
• Rasta Ring 0 Debugger (RR0D)
• Syser Kernel Debugger
• TRW 2000 (very old debugger circa W9x) + dions plugin archive
• VisualDux Debugger
• Wintruder (extendable debugger)
• WKTVDebugger (a debugger for Visual Basic P-Code) (download)
• x64_dbg
• Zeta Debugger

Other Links

• Collaborative RCE Tool Library
  - huge collection of debugger and system level tools
• cr4zyserb
  - huge collection of plugins and other debugging tools
• How to Write a Windows Debugger References (Devon Straw)
  - large collection of links giving you detailed information that you would need if you wanted to write your own debugger e.g. PDB file format, .DMP file formats, PE File structure, how to record stack traces, etc, etc.
• Tuts4You
  - unpackers, IDA, OllyDBG, Immunity Debugger plugins, etc.

Answer 3

This is a really broad question.

1. The first step is to load the dump file into a WinDbg instance.
2. Next, you need to make sure you have a symbols setup.
3. Finally, you can run the command !analyze -v to get a basic analysis performed on it. You need to have symbol information available for your code to make dump files worthwhile.

The website Memory Dump, Software Trace, Debugging, Malware, Victimware and Intelligence Analysis Portal has been very informative for me. I also really enjoyed the book, Advanced Windows Debugging by Mario Hewardt and Daniel Pravat.