Symbols for ngen-ed assemblies are not loaded

Question

I am trying to decode the following call stack line from procmon:

29  System.Management.Automation.ni.dll System.Management.Automation.ni.dll + 0x897a0a  0x7fee2ae7a0a   C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Manaa57fc8cc#\a86698074f28597f1fc5ceabfed6fed6\System.Management.Automation.ni.dll

As you can see there is an NGEN-ed assembly in it: System.Management.Automation.ni.dll. I created a pdb file for it with ngen createpdb:

PS> ngen createpdb c:\Windows\assembly\NativeImages_v4.0.30319_64\System.Manaa57fc8cc#\a86698074f28597f1fc5ceabfed6fed6\System.Management.Automation.ni.dll c:\symbols\ngen

Successfully generated PDB for native assembly 'c:\Windows\assembly\NativeImages_v4.0.30319_64\System.Manaa57fc8cc#\a8698074f28597f1fc5ceabfed6fed6\System.Management.Automation.ni.dll'.
PDB generated in directory c:\symbols\ngen\System.Management.Automation.ni.pdb a86698074f28597f1fc5ceabfed6fed61\

My Symbol path in the _NT_SYMBOL_PATH variable is:

SRV*C:\symbols\ngen*;SRV*C:\symbols\dbg*http://referencesource.microsoft.com/symbols;SRV*C:\symbols\dbg*http://msdl.microsoft

.com/download/symbols

But I can still see that the newly generated symbol file is not loaded for the assembly:

PS a86698074f28597f1fc5ceabfed6fed6> dbh -n .\System.Management.Automation.ni.dll
verbose mode on.
DBGHELP: No header for .\System.Management.Automation.ni.dll.  Searching for image on disk
DBGHELP: C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Manaa57fc8cc#\a86698074f28597f1fc5ceabfed6fed6\System.Management.Automation.ni.dll - OK
SYMSRV:  C:\symbols\ngen\System.Management.Automation.pdb\6B8B8F14D0564CB893B6E84B43CAE67B1\System.Management.Automation.pdb - file not found
SYMSRV:  C:\tools\diag\Debugging Tools for Windows\x64\sym\System.Management.Automation.pdb\6B8B8F14D0564CB893B6E84B43CAE67B1\System.Management.Automation.pdb - file not found
SYMSRV:  C:\symbols\ngen\System.Management.Automation.pdb\6B8B8F14D0564CB893B6E84B43CAE67B1\System.Management.Automation.pdb not found
SYMSRV:  C:\tools\diag\Debugging Tools for Windows\x64\sym\System.Management.Automation.pdb\6B8B8F14D0564CB893B6E84B43CAE67B1\System.Management.Automation.pdb not found
DBGHELP: System.Management.Automation.ni - public symbols
        C:\symbols\dbg\System.Management.Automation.pdb\6B8B8F14D0564CB893B6E84B43CAE67B1\System.Management.Automation.pdb

I've checked the debug header in the .dll file and it has two entries:

PS> dumpbin /headers .\System.Management.Automation.ni.dll
...
  Debug Directories
        Time Type        Size      RVA  Pointer
    -------- ------- -------- -------- --------
    56BEFBC1 cv           11C 01F200A4  1F1E8A4    Format: RSDS, {A8669807-4F28-597F-1FC5-CEABFED6FED6}, 1, System.Management.Automation.ni.pdb
    56BEFBC1 cv            39 01F201C0  1F1E9C0    Format: RSDS, {6B8B8F14-D056-4CB8-93B6-E84B43CAE67B}, 1, System.Management.Automation.pdb
...

The A8669807-4F28-597F-1FC5-CEABFED6FED6 entry is first, but seems that is never used by dbh (or actually dbghelp), and it is only looking for the 6B8B8F14-D056-4CB8-93B6-E84B43CAE67B. I tried setting the symbols path only to the C:\symbols\ngen, but it didn't help - the symbol file was still not found.

My dbghelp version is: 10.0.10240.16399

Can someone point me what I'm doing wrong here?

EDIT 1:

Seems that the dbh verbose output is pretty consistent with what procmon is showing:

EDIT 2 (to Hans answer)

My application is actually a Powershell script. I listed .NET modules for powershell.exe in Process Hacker and found that it's using System.Management.Automation.dll version 3.0.0:

I suppose that the original assembly is located in GAC: c:\Windows\Microsoft.NET\assembly\GAC_MSIL\System.Management.Automation\v4.0_3.0.0.0__31bf3856ad364e35\System.Management.Automation.dll

which apparently was created for .NET 4.0:

PS temp> corflags c:\Windows\Microsoft.NET\assembly\GAC_MSIL\System.Management.Automation\v4.0_3.0.0.0__31bf3856ad364e35\System.Management.Automation.dll
Microsoft (R) .NET Framework CorFlags Conversion Tool.  Version  4.6.1055.0
Copyright (c) Microsoft Corporation.  All rights reserved.
Version   : v4.0.30319
CLR Header: 2.5
PE        : PE32
CorFlags  : 0x9
ILONLY    : 1
32BITREQ  : 0
32BITPREF : 0
Signed    : 1

Now, I've also looked for any other System.Management.Automation assemblies in the NativeImages folder, but there is only 1 for .NET 4.0 64-bit:

The .aux header also seems to mention only the 3.0.0 version. Notice also that the .ni file has TWO PDB files referenced in the Debug header. One of them is the one I want.

EDIT 3 (fuslogvw)

As advised by Hans I enabled the Fusion log for Native Images. Below is the snippet showing the path from which the Automation assembly is loaded:

... Pre-bind state information ...
LOG: DisplayName = System.Management.Automation, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35
 (Fully-specified)
LOG: Appbase = file:///C:/Windows/System32/WindowsPowershell/v1.0/
LOG: Initial PrivatePath = NULL
LOG: Dynamic Base = NULL
LOG: Cache Base = NULL
LOG: AppName = powershell.exe
Calling assembly : Microsoft.PowerShell.ConsoleHost, Version=3.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35.
...
LOG: Start validating all the dependencies.
LOG: [Level 1]Start validating native image dependency mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089.
LOG: [Level 1]Start validating native image dependency System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089.
LOG: [Level 1]Start validating IL dependency Microsoft.Management.Infrastructure, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35.
LOG: [Level 1]Start validating native image dependency System.Core, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089.
LOG: [Level 1]Start validating IL dependency System.Configuration.Install, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a.
LOG: [Level 1]Start validating IL dependency System.Transactions, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089.
LOG: [Level 1]Start validating IL dependency System.Xml, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089.
LOG: [Level 1]Start validating IL dependency Microsoft.Management.Infrastructure.Native, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35.
LOG: [Level 1]Start validating IL dependency System.Management, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a.
LOG: [Level 1]Start validating IL dependency System.Configuration, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a.
LOG: [Level 1]Start validating IL dependency System.DirectoryServices, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a.
LOG: [Level 1]Start validating IL dependency System.Runtime.Serialization, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089.
LOG: [Level 1]Start validating IL dependency Microsoft.CSharp, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a.
LOG: [Level 1]Start validating IL dependency System.ServiceModel.Internals, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35.
LOG: [Level 1]Start validating IL dependency System.Data, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089.
LOG: [Level 1]Start validating IL dependency System.Numerics, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089.
LOG: [Level 1]Start validating IL dependency System.Security, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a.
LOG: [Level 1]Start validating IL dependency SMDiagnostics, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089.
Native image has correct version information.
LOG: Validation of dependencies succeeded.
LOG: Bind to native image succeeded.
Attempting to use native image C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Manaa57fc8cc#\a86698074f28597f1fc5ceabfed6fed6\System.Management.Automation.ni.dll.
Native image successfully used.

Answer 1

Unfortunately, I think it is a bug in either dbghelp or ngen. I created a Test.dll assembly and ngen-ed it:

ngen install Test.dll

It landed in:

c:\Windows\assembly\NativeImages_v4.0.30319_64\Test\7dece650b5d91e7ac799a78b3d1b7c59\Test.ni.dll

as expected. I also created symbols for it:

ngen createpdb c:\Windows\assembly\NativeImages_v4.0.30319_64\Test\7dece650b5d91e7ac799a78b3d1b7c59\Test.ni.dll c:\symbols\ngen

When I checked the debug headers I got two again:

> dumpbin /headers c:\Windows\assembly\NativeImages_v4.0.30319_64\Test\7dece650b5d91e
7ac799a78b3d1b7c59\Test.ni.dll
  Debug Directories
        Time Type        Size      RVA  Pointer
    -------- ------- -------- -------- --------
    5824BFEB cv           11C 00003D40     1F40    Format: RSDS, {7DECE650-B5D9-1E7A-C799-A78B3D1B7C59}, 1, Test.ni.pdb
    5824BFEB cv           11C 00002E5C     205C    Format: RSDS, {F32EB2CE-973C-438F-BB78-A24D9971C194}, 1, C:\temp\Test.pdb

When I tried to load symbols for the Test.ni.dll, dbh tried to load .pdb file with the F32EB2CE-973C-438F-BB78-A24D9971C194 signatue (which is wrong). I then opened a HEX editor, and replaced the order in which the debug directories are listed in the PE file (I found them by timestamps :)):

Now, dumpbin showed them in a different order:

Time     Type        Size      RVA  Pointer
-------- ------- -------- -------- --------
5824BFEB cv           11C 00002E5C     205C    Format: RSDS, {F32EB2CE-973C-438F-BB78-A24D9971C194}, 1, C:\temp\Test.pdb
5824BFEB cv           11C 00003D40     1F40    Format: RSDS, {7DECE650-B5D9-1E7A-C799-A78B3D1B7C59}, 1, Test.ni.pdb

and dbh started working correctly:

> dbh -n -s:SRV*c:\symbols\ngen* c:\Windows\assembly\NativeImages_v4.0.30319_64\Test\
7dece650b5d91e7ac799a78b3d1b7c59\Test.ni.dll
verbose mode on.
DBGHELP: Symbol Search Path: SRV*c:\symbols\ngen*
Symbol Search Path: SRV*c:\symbols\ngen*
DBGHELP: No header for c:\Windows\assembly\NativeImages_v4.0.30319_64\Test\7dece650b5d91e7ac799a78b3d1b7c59\Test.ni.dll.  Searching for image on disk
DBGHELP: c:\Windows\assembly\NativeImages_v4.0.30319_64\Test\7dece650b5d91e7ac799a78b3d1b7c59\Test.ni.dll - OK
DBGHELP: Test.ni - public symbols & lines
        c:\symbols\ngen\Test.ni.pdb\7DECE650B5D91E7AC799A78B3D1B7C591\Test.ni.pdb
Test.ni [1000000]:

I created an issue on connect and kindly ask you to upvote it.