How can I get complete stack traces for mixed-mode minidumps when (WPF) native images are involved?

Question

I have a mixed-mode C++/CLI application which uses WPF. Crashes from our customers are reported as minidumps to our own server.

When I try to investigate a minidump using the !pe or !clrstack commands from the windbg sos-extension, I often get incomplete information for stack frames from WPF assemblies, e.g.

SP       IP       Function
0013E370 564618E3 PresentationFramework_ni!Unknown+0x1bf
0013E3A4 56461258 PresentationFramework_ni!Unknown+0x58
0013E3CC 5634C6D8 PresentationFramework_ni!Unknown+0x18
0013E3D8 55C04AA2 PresentationFramework_ni!Unknown+0x502
...

The stack trace decoding also gets very slow in this case.

Using !sym noisy shows a lot of messages of the following from

SYMSRV:  C:\Symbols\PresentationFramework.ni.dll\488F142Edab000\PresentationFramework.ni.dll not found
SYMSRV:  http://msdl.microsoft.com/download/symbols/PresentationFramework.ni.dll/488F142Edab000/PresentationFramework.ni.dll not found
DBGHELP: C:\Program Files (x86)\Debugging Tools for Windows (x86)\PresentationFramework.ni.dll - file not found
DBGHELP: PresentationFramework.ni.dll not found in c:\Windows\System32
SYMSRV:  C:\Symbols\PresentationFramework.ni.dll\488F142Edab000\PresentationFramework.ni.dll not found
SYMSRV:  http://msdl.microsoft.com/download/symbols/PresentationFramework.ni.dll/488F142Edab000/PresentationFramework.ni.dll not found
DBGENG:  C:\WINDOWS\assembly\NativeImages_v2.0.50727_32\PresentationFramewo#\9519494798a88867406b5755e1dbded6\PresentationFramework.ni.dll - Couldn't map image from disk.
SYMSRV:  C:\Symbols\PresentationFramework.dll\488F142E50e000\PresentationFramework.dll not found
SYMSRV:  http://msdl.microsoft.com/download/symbols/PresentationFramework.dll/488F142E50e000/PresentationFramework.dll not found
DBGHELP: C:\Program Files (x86)\Debugging Tools for Windows (x86)\PresentationFramework.dll - file not found
DBGHELP: PresentationFramework.dll not found in c:\Windows\System32
SYMSRV:  C:\Symbols\PresentationFramework.dll\488F142E50e000\PresentationFramework.dll not found
SYMSRV:  http://msdl.microsoft.com/download/symbols/PresentationFramework.dll/488F142E50e000/PresentationFramework.dll not found
DBGENG:  C:\WINDOWS\assembly\GAC_MSIL\PresentationFramework\3.0.0.0__31bf3856ad364e35\PresentationFramework.dll image header does not match memory image header.
DBGENG:  C:\WINDOWS\assembly\GAC_MSIL\PresentationFramework\3.0.0.0__31bf3856ad364e35\PresentationFramework.dll - Couldn't map image from disk.

I used

c:\Windows\System32;SRV*C:\Symbols*http://msdl.microsoft.com/download/symbols

as the windbg symbol and image path.

As far as I have understood this only happens for .NET native images if the machine on which the crash occured and the machine with the debugger differ in terms of Windows version, .NET version of SPs. I have seen it mostly for WPF native images.

What can I do to avoid this problem?

Update to my initial question:

I forgot to mention that I struggled with a similar problem with different mscordacwks dll versions. In order to use SOS, the version of mscordacwks.dll used on the crashing computer is needed on the degbugging computer. So I started to collect various versions of this dll from different Windows and SP combinations and put them on our own symbol server. This is pretty awkward, of course, and even more so because they need to be named following a special convention (e.g. mscordacwks_x86_x86_2.0.50727.4952.dll).

If I understand Rick's answer from below correctly I have to do something similar for the native images of the .NET assemblies that we reference. I have tried this manually with one example (WindowsBase.ni.dll) but I couldn't store this dll on our symbol server easily. It seems that native images are not understood by symstore. The error message from symstore is:

SYMSTORE MESSAGE: Skipping file .\WindowsBase.ni.dll - not a known file type.

So I tried to put it into an extra dir and added it to my symbol or image path and then SOS decoded the WindowsBase_ni frames correctly.

But all this seems like a lot of annoying manual setup work: getting all the native images for various .NET versions (what about SPs and security updates), setting up the debugger manually because symstore cannot be used, ...

Is this really the only way?

It's probably not such an issue if you can control the environment of your customers. But it seems like a post-mortem debugging nightmare for orgnizations which build mixed-mode applications for large user bases.

Answer 1

The symbol server output shows that it is having trouble downloading the images, not the symbols for those images. While Microsoft is pretty good at ensuring symbols for all released files get put on the symbol server, in this case the DLLs themselves were not.

The reason that WinDbg wants the original DLLs in addition to the symbols is because in order to keep the minidump small, most of the memory image is left out. In this case the computer the minidump was created on was using a different version of the .NET framework than is installed on the machine that WinDbg is being run on. Let's say that the crashing computer has .NET3.5 running Windows XP and the analysis computer is .NET3.5 running on Windows 7. You would think they would be the same version but Windows 7 got it's own special version of .NET3.5 as can be seen here:

• .Net Framework Libraries

The solution is to put the DLLs that cannot be loaded from the symbol server somewhere on the symbol path. However, I don't see a straightforward way of downloading and installing just the reference assemblies for a particular .NET version you want. But since you implied that .loadby sos mscorwks worked for you, then it may be that the DLLs you want are already on the computer somewhere.

First you need to create a minidump of your program on a test computer you can control that produces these symptoms in WinDbg. I suggest trying Windows XP. Then use Process Explorer to find the full path to PresentationFramework.DLL on the test computer. Then compare the file size and date to DLLs on your computer in places like:

• C:\Windows\Microsoft.NET
• C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework
• C:\Program Files\Reference Assemblies\Microsoft\Framework

If you can find the file then put the folder it was found in on your symbol path. If you cannot find the file, then you can resort to copying the missing files from the test computer. This is not as bad as it sounds because there aren't very many published version of the .NET framework.