WinDbg x64: Cannot debug a crash dump - failed to load data access DLL

Question

I attached WinDbg to a running process and had the process crashed (I have a separate question re. that case). Once the program crashed, WinDbg stopped and allowed me to debug the program. I took a crash dump for further investigation with a command ".dump /ma".

The program was compiled as "Any CPU" and I used WinDbg x64 to take the dump. Now I open WinDbg x64 on the same computer again and open the crash dump. Here is what it says:

Loading Dump File [C:\crashdump.dmp] User Mini Dump File with Full Memory: Only application data is available  Symbol search path is: SRV*c:\symbols*http://msdl.microsoft.com/download/symbols Executable search path is:  Windows 7 Version 7601 (Service Pack 1) MP (8 procs) Free x64 Product: WinNt, suite: SingleUserTS Machine Name: Debug session time: Mon Aug 15 10:24:57.000 2011 (UTC + 1:00) System Uptime: 17 days 0:54:39.021 Process Uptime: 12 days 14:01:31.000 ................................................................ ............................................................... This dump file has an exception of interest stored in it. The stored exception information can be accessed via .ecxr. (1be0.b78): Access violation - code c0000005 (first/second chance not available) *** WARNING: symbols timestamp is wrong 0x4dd2333e 0x4da4281c for clr.dll clr!WKS::gc_heap::find_first_object+0x92: 000007fe`ea129a1d f70100000080    test    dword ptr [rcx],80000000h ds:00000000`00003d80=???????? 

Then I try to load SOS by ".load sos clr" and it errors in:

The call to LoadLibrary(sos clr) failed, Win32 error 0n2     "The system cannot find the file specified." Please check your debugger configuration and/or network access. 

Trying with ".loadby sos clr" and it works. Now I want to see the stack with "!clrstack" and stick here:

Failed to load data access DLL, 0x80004005 Verify that 1) you have a recent build of the debugger (6.2.14 or newer)             2) the file mscordacwks.dll that matches your version of clr.dll is                  in the version directory             3) or, if you are debugging a dump file, verify that the file                  mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.             4) you are debugging on the same architecture as the dump file.                 For example, an IA64 dump file must be debugged on an IA64                 machine.  You can also run the debugger command .cordll to control the debugger's load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload. If that succeeds, the SOS command should work on retry.  If you are debugging a minidump, you need to make sure that your executable path is pointing to clr.dll as well. 

I tried ".symfix" and ".reload":

0:027> .symfix 0:027> .reload ..................*** WARNING: symbols timestamp is wrong 0x4dd2333e 0x4da4281c for clr.dll .............................................. ............................................................... 

Stuck. At the same time when the process is running under WinDgb I can pause the execution, load SOS and execute "!clrstack" command successfully.

Any ideas? Thank you.

UPDATE - Followed the steps provided in the second answer, still doesn't work.

1) My Symbol Path: SRV*c:\symbols*http://msdl.microsoft.com/download/symbols;srv*

2) CLR loaded: 4.0.30319.237:

0:027> lm v clr Unknown option 'r' start             end                 module name 00000000`77b60000 00000000`77d09000   ntdll      (pdb symbols)          c:\symbols\ntdll.pdb\6192BFDB9F04442995FFCB0BE95172E12\ntdll.pdb     Loaded symbol image file: ntdll.dll     Image path: C:\Windows\System32\ntdll.dll     Image name: ntdll.dll     Timestamp:        Sat Nov 20 13:11:21 2010 (4CE7C8F9)     CheckSum:         001B55EA     ImageSize:        001A9000     File version:     6.1.7601.17514     Product version:  6.1.7601.17514     File flags:       0 (Mask 3F)     File OS:          40004 NT Win32     File type:        2.0 Dll     File date:        00000000.00000000     Translations:     0409.04b0     CompanyName:      Microsoft Corporation     ProductName:      Microsoft® Windows® Operating System     InternalName:     ntdll.dll     OriginalFilename: ntdll.dll     ProductVersion:   6.1.7601.17514     FileVersion:      6.1.7601.17514 (win7sp1_rtm.101119-1850)     FileDescription:  NT Layer DLL     LegalCopyright:   © Microsoft Corporation. All rights reserved. 000007fe`e9fb0000 000007fe`ea915000   clr      # (pdb symbols)          c:\symbols\clr.pdb\1A7EA01DA29549DAB2B0BD012A6C5BA12\clr.pdb     Loaded symbol image file: clr.dll     Image path: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll     Image name: clr.dll     Timestamp:        Tue May 17 09:35:10 2011 (4DD2333E)     CheckSum:         00967144     ImageSize:        00965000     File version:     4.0.30319.237     Product version:  4.0.30319.237     File flags:       8 (Mask 3F) Private     File OS:          4 Unknown Win32     File type:        2.0 Dll     File date:        00000000.00000000     Translations:     0409.04b0     CompanyName:      Microsoft Corporation     ProductName:      Microsoft® .NET Framework     InternalName:     clr.dll     OriginalFilename: clr.dll     ProductVersion:   4.0.30319.235     FileVersion:      4.0.30319.235 (RTMGDR.030319-2300)     PrivateBuild:     DDBLD240     FileDescription:  Microsoft .NET Runtime Common Language Runtime - WorkStation     LegalCopyright:   © Microsoft Corporation.  All rights reserved.     Comments:         Flavor=Retail 

3) "C:\Windows\Microsoft.NET\Framework64\v4.0.30319\mscordacwks.dll" has version 4.0.30319.239

4) I found that when I load the dump into WinDbg it loads the correct "mscordacwks.dll" from the web, thus in the folder "C:\symbols\mscordacwks_AMD64_AMD64_4.0.30319.237.dll\4DD2333E965000" I have the file "mscordacwks_AMD64_AMD64_4.0.30319.237.dll".

5)

0:027> .cordll -ve -u -l CLR DLL status: No load attempts 

6)

0:027> !sym noisy noisy mode - symbol prompts on 0:027> .restart  Loading Dump File [C:\crashdump.dmp] User Mini Dump File with Full Memory: Only application data is available  DBGHELP: Symbol Search Path: srv*;srv*c:\symbols*http://msdl.microsoft.com/download/symbols DBGHELP: Symbol Search Path: cache*;SRV*http://msdl.microsoft.com/download/symbols;srv*c:\symbols*http://msdl.microsoft.com/download/symbols DBGHELP: Symbol Search Path: cache*;SRV*http://msdl.microsoft.com/download/symbols;srv*c:\symbols*http://msdl.microsoft.com/download/symbols Symbol search path is: srv*;SRV*c:\symbols*http://msdl.microsoft.com/download/symbols Executable search path is:  Windows 7 Version 7601 (Service Pack 1) MP (8 procs) Free x64 Product: WinNt, suite: SingleUserTS Machine Name: Debug session time: Mon Aug 15 10:24:57.000 2011 (UTC + 1:00) System Uptime: 17 days 0:54:39.021 Process Uptime: 12 days 14:01:31.000 ................................................................ ............................................................... DBGHELP: ntdll - public symbols            C:\Program Files\Debugging Tools for Windows (x64)\sym\ntdll.pdb\6192BFDB9F04442995FFCB0BE95172E12\ntdll.pdb DBGHELP: Symbol Search Path: cache*;SRV*http://msdl.microsoft.com/download/symbols;srv*c:\symbols*http://msdl.microsoft.com/download/symbols DBGHELP: Symbol Search Path: cache*;SRV*http://msdl.microsoft.com/download/symbols;srv*c:\symbols*http://msdl.microsoft.com/download/symbols This dump file has an exception of interest stored in it. The stored exception information can be accessed via .ecxr. (1be0.b78): Access violation - code c0000005 (first/second chance not available) *** WARNING: symbols timestamp is wrong 0x4dd2333e 0x4da4281c for clr.dll DBGHELP: clr - public symbols            C:\Program Files\Debugging Tools for Windows (x64)\sym\clr.pdb\1A7EA01DA29549DAB2B0BD012A6C5BA12\clr.pdb clr!WKS::gc_heap::find_first_object+0x92: 000007fe`ea129a1d f70100000080    test    dword ptr [rcx],80000000h ds:00000000`00003d80=???????? 

7)

0:027> !clrstack SYMSRV:  C:\Program Files\Debugging Tools for Windows (x64)\sym\mscordacwks_AMD64_AMD64_4.0.30319.237.dll\4DD2333E965000\mscordacwks_AMD64_AMD64_4.0.30319.237.dll not found SYMSRV:  mscordacwks_AMD64_AMD64_4.0.30319.237.dll from http://msdl.microsoft.com/download/symbols: 502892 bytes - copied          DBGHELP: C:\Program Files\Debugging Tools for Windows (x64)\sym\mscordacwks_AMD64_AMD64_4.0.30319.237.dll\4DD2333E965000\mscordacwks_AMD64_AMD64_4.0.30319.237.dll cached to C:\Program Files\Debugging Tools for Windows (x64)\sym\mscordacwks_AMD64_AMD64_4.0.30319.237.dll\4DD233F317b000\mscordacwks_AMD64_AMD64_4.0.30319.237.dll DBGHELP: C:\Program Files\Debugging Tools for Windows (x64)\sym\mscordacwks_AMD64_AMD64_4.0.30319.237.dll\4DD233F317b000\mscordacwks_AMD64_AMD64_4.0.30319.237.dll - OK Failed to load data access DLL, 0x80004005 Verify that 1) you have a recent build of the debugger (6.2.14 or newer)             2) the file mscordacwks.dll that matches your version of clr.dll is                  in the version directory             3) or, if you are debugging a dump file, verify that the file                  mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.             4) you are debugging on the same architecture as the dump file.                 For example, an IA64 dump file must be debugged on an IA64                 machine.  You can also run the debugger command .cordll to control the debugger's load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload. If that succeeds, the SOS command should work on retry.  If you are debugging a minidump, you need to make sure that your executable path is pointing to clr.dll as well. 

Answer 1

I hit this regularly when debugging with minidumps from the site. Quite how it's happened in your case I'm not sure. Usually, it happens when the version of CLR that was loaded when the dump was taken is not available on your debugging machine. In your case, they're the same machine, so it should all probably just work. I'm sure there will be others who can explain exactly why it isn't.

In the meantime, here's what I do with my site dumps. Windbg is looking for "the right version" of mscordacwks.dll. So we give it that version and tell it where to look for it.

First - if I spoof all of this, by deleting mscordacwks.dll, windbg goes off and loads it from the Microsoft symbol server, so do make sure your symbols are set up correctly to download symbols from the Microsoft symbol server and give it another go.

Now - assuming that didn't work, check exactly which version is the "right version". List the module info with "lm v clr" and check your CLR version that is ACTUALLY loaded. Mine is 4.0.30319.239. Ok - now find that version of mscordacwks.dll. Let's assume it can be found in the normal .NET framework installation on your machine (C:\Windows\Microsoft.NET\Framework64\v4.0.30319). Do check the version matches exactly (right-click, properties etc)! Take it and put it in a safe place (I use D:\Symbols\_Images). Follow the instructions that windbg gave you on renaming the file. mscordacwks_.dll would be mscordacwks_AMD64_AMD64_4.0.30319.239.dll.

Now set up your executable image path (".exepath D:\Symbols\_Images") so windbg knows where you've put it.

You've now got "the right version of mscordacwks", and renamed it so that Windbg knows what it's looking for, and told it where you've put it.

If that STILL isn't working, then try ".cordll -ve -u -l" and also "!sym noisy" to turn on verbose logging of both the cordll load and the symbol server, then try !CLRStack again. Maybe the output of those two commands will tell you exactly what it's trying to load and you can figure out why it won't do it...