Is there a known issue relating to Windows 7 Kernel Symbols?

Question

I have a few Windows 7 machines that I am not able to read their memory dumps. I found something that I suspect may be related, but am not positive:

https://twitter.com/aionescu/status/634028737458114560

I also found this: http://support.microsoft.com/kb/2528507

However, the scenario message regarding wow64exts given in the doc is not seen in any of my dumps. I also cannot apply that hotfix at this time to test it. So I'm just looking for some more information or opinions.

I'm able to open any other OS dump as well as my own system's Windows 7 dump, but there are 2 other machines that run Win 7 and it's telling me I have the wrong kernel symbols.

I have tried clearing out my symbol cache, reinstalled the Windows SDK, and also tried to open the dumps on two other machines with the same result. If it matters, the crash is manually created using the scroll lock method.

Symbol path: SRV*c:\symbols*http://msdl.microsoft.com/download/symbols;

Seeing these errors: followed by "Type referenced: nt!_KPRCB"

Does anyone know about the issue mentioned by Alex in the twitter link and if it's possibly related to what I'm seeing?

Answer 1

Update 2015-10-22:

With the Microsoft patch day (2015-10-13) and KB3088195, symbols are available again.

However, symbols for the broken version have not been provided, so below may still be useful.

---

Microsoft has already published "good" symbols for ntdll in the past, containing type information like _TEB or _KPRCB. Starting from mid of July 2015, Microsoft has still published symbols for ntdll, but not containing that information.

So it depends on the version of ntdll whether you get type information or not. Old dumps referencing an old version of ntdll will download old PDBs containing type information while new dumps reference new versions of ntdll and WinDbg (or any other debugger) downloads PDBs without type information.

Could Microsoft remove type information of "good" symbols retroactively, thus making them "bad"?

Yes. As described in this answer, there is a tool to remove type information from existing PDBs. Doing that and replacing the PDB would result in such an effect.

Can Microsoft publish the "good" version of those PDBs which are currently "bad"?

That's hard to tell, since we don't know whether Microsoft has kept a copy of the "good" version so they could replace the "bad" version on the symbol server with the "good" one. Rebuilding ntdll from the same source code and thus creating new PDBs sounds possible, but the PDB gets a new time stamp and checksum. This can potentially be corrected manually, especially be Microsoft, since they should have the knowledge about the PDB internal format, but IMHO it's unlikely they'll do that. Things may go wrong and MS will hardly have tests to guarantee the correctness of such a thing.

So what can I do?

IMHO you can do nothing to really correct the situation.

You could assume that the types in ntdll have not changed so much. This would allow you to take an older version of wntdll.pdb and the new version of ntdll.dll and apply ChkMatch -m to it. This will copy the timestamp and checksum from the DLL to the PDB. After you did that (in an empty folder), replace the existing wntdll.pdb in your symbols directory with the hacked one.

WinDbg walkthrough (with output shortened to relevant things). I am using the latest version of wntdll.pdb I could find on my PC.

WARNING: doing the following may fix the type information but will likely destroy the correctness of the callstacks. Since any changes in the implementation (which are likely for security fixes) will change the method offsets.

0:005> dt nt!_PEB
*************************************************************************
***                                                                   ***
***    Either you specified an unqualified symbol, or your debugger   ***
...
***    Type referenced: nt!_PEB                                       ***
***                                                                   ***
*************************************************************************
Symbol nt!_PEB not found.

0:005> lm m ntdll
start    end        module name
773f0000 77570000   ntdll      (pdb symbols)          e:\debug\symbols\wntdll.pdb\FA9C48F9C11D4E0894B8970DECD92C972\wntdll.pdb

0:005> .shell cmd /c copy C:\Windows\SysWOW64\ntdll.dll e:\debug\temp\ntdllhack\ntdll.dll
        1 file(s) copied.

0:005> .shell cmd /c copy "E:\Windows SDk\8.0\Debuggers\x86\sym\wntdll.pdb\B081677DFC724CC4AC53992627BEEA242\wntdll.pdb" e:\debug\temp\ntdllhack\wntdll.pdb
        1 file(s) copied.

0:005> .shell cmd /c E:\debug\temp\ntdllhack\chkmatch.exe -m E:\debug\temp\ntdllhack\ntdll.dll E:\debug\temp\ntdllhack\wntdll.pdb
...
Executable: E:\debug\temp\ntdllhack\ntdll.dll 
Debug info file: E:\debug\temp\ntdllhack\wntdll.pdb 

Executable: 
TimeDateStamp: 55a69e20 
Debug info: 2 ( CodeView ) 
TimeStamp: 55a68c18  Characteristics: 0  MajorVer: 0  MinorVer: 0 
Size: 35  RVA: 000e63e0  FileOffset: 000d67e0  
CodeView format: RSDS 
Signature: {fa9c48f9-c11d-4e08-94b8-970decd92c97}  Age: 2  
PdbFile: wntdll.pdb 
Debug info: 10 ( Unknown ) 
TimeStamp: 55a68c18  Characteristics: 0  MajorVer: 565  MinorVer: 6526 
Size: 4  RVA: 000e63dc  FileOffset: 000d67dc  

Debug information file: 
Format: PDB 7.00 
Signature: {b081677d-fc72-4cc4-ac53-992627beea24}  Age: 4 

Writing to the debug information file... 
Result: Success.

0:005> .shell cmd /c copy E:\debug\temp\ntdllhack\wntdll.pdb E:\debug\symbols\wntdll.pdb\FA9C48F9C11D4E0894B8970DECD92C972\wntdll.pdb
        1 file(s) copied.

0:005> .reload
Reloading current modules
.............................

0:005> dt nt!_PEB
ntdll!_PEB
   +0x000 InheritedAddressSpace : UChar
   +0x001 ReadImageFileExecOptions : UChar
...

0:005> !heap -s
LFH Key                   : 0x219ab08b
Termination on corruption : DISABLED
  Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast 
                    (k)     (k)    (k)     (k) length      blocks cont. heap 
-----------------------------------------------------------------------------
Virtual block: 00920000 - 00920000 (size 00000000)
Virtual block: 02c60000 - 02c60000 (size 00000000)
Virtual block: 02e10000 - 02e10000 (size 00000000)
...

Note: using ChkMatch like this has the benefit that you do not need to turn on .symopt- 100, since that option would affect all PDB files, and you would not find potential other symbol issues. If you don't mind using .symopt, you could simply copy an old wntdll.PDB over the new one.

Answer 2

The issue is now fixed according to Microsoft and Microsoft told me that you should clear your symbol cache to get the new PDBs, otherwise Windbg would use the old Symbols which miss the information.