How to debug C# managed/unmanaged marshalling exceptions?

Question

I'm writing a DirectShow filter on C# and I don't want to use any third party library. This task is almost 100% based on the correct prototyping of the COM interfaces like IGraphBuilder, IBaseFilter, IPin, etc. And if the interface is prototyped incorrectly various exceptions at managed/native boundaries are thrown. The problem is to determine the location of the incorrectly prototyped interface method.

Currently I have: An exception of type 'System.NullReferenceException' occurred in DirectShow.dll (this is the name of my managed dll) and wasn't handled before a managed/native boundary

Call stack:

ntdll.dll!NtWaitForSingleObject() + 0xa bytes
KernelBase.dll!WaitForSingleObjectEx() + 0x9c bytes
clr.dll!CLREvent::WaitEx() + 0x20f bytes
clr.dll!CLREvent::WaitEx() + 0x1b8 bytes
clr.dll!CLREvent::WaitEx() + 0x73 bytes
clr.dll!Thread::WaitSuspendEventsHelper() + 0xcf bytes clr.dll!Thread::WaitSuspendEvents() + 0x10 bytes
clr.dll!string "d:\iso_whid\amd64fre\base\ntos\r"...() + 0x35688d bytes
clr.dll!Thread::RareDisablePreemptiveGC() + 0x118 bytes
clr.dll!GCHolderEEInterface<0,0,0>::~GCHolderEEInterface<0,0,0>() + 0x19 bytes clr.dll!Debugger::SendCatchHandlerFound() + 0x150 bytes
clr.dll!string "d:\iso_whid\amd64fre\base\ntos\r"...() + 0x3b9340 bytes
clr.dll!NotifyOfCHFFilterWrapper() + 0x77 bytes
clr.dll!string "d:\iso_whid\amd64fre\base\ntos\r"...() + 0x336941 bytes
msvcr100_clr0400.dll!__C_specific_handler() + 0x97 bytes
ntdll.dll!RtlpExecuteHandlerForException() + 0xd bytes ntdll.dll!RtlDispatchException() + 0x38f bytes ntdll.dll!KiUserExceptionDispatch() + 0x2e bytes
KernelBase.dll!RaiseException() + 0x3d bytes
clr.dll!NakedThrowHelper2() + 0xc bytes
clr.dll!NakedThrowHelper_RspAligned() + 0x3d bytes clr.dll!NakedThrowHelper_FixRsp() + 0x5 bytes
000007ff00179486()
clr.dll!COMToCLRDispatchHelper() + 0x4e bytes
clr.dll!SecurityDeclarative::CheckLinkDemandAgainstAppDomain() - 0x40e bytes
clr.dll!COMToCLRWorkerBody() + 0xd6 bytes
clr.dll!COMToCLRWorkerDebuggerWrapper() + 0x22 bytes
clr.dll!COMToCLRWorker() + 0x201 bytes clr.dll!GenericComCallStub() + 0x57 bytes
[Native to Managed Transition]
quartz.dll!CEnumConnectedPins::CEnumConnectedPins() + 0x4a bytes
quartz.dll!CFilterGraph::FindUpstreamInterface() + 0x150 bytes quartz.dll!CFilterGraph::FindUpstreamInterface() + 0xc1 bytes
quartz.dll!CFilterGraph::FindUpstreamInterface() + 0x171 bytes quartz.dll!CFilterGraph::FindUpstreamInterface() + 0xc1 bytes
quartz.dll!CFilterGraph::FindUpstreamInterface() + 0x171 bytes quartz.dll!CFilterGraph::FindUpstreamInterface() + 0xc1 bytes
quartz.dll!CWaveSlave::UpdateSlaveMode() + 0xa7 bytes
quartz.dll!CWaveOutInputPin::RemovePreroll() + 0x95 bytes
quartz.dll!CWaveOutInputPin::Receive() + 0x12f bytes
msmpeg2adec.dll!CBaseOutputPin::Deliver() + 0x22 bytes msmpeg2adec.dll!CIVIAudioFilter::DeliverOutSample() + 0x3da bytes
msmpeg2adec.dll!CIVIAudioCodec::DecodeDDPlus() + 0x556 bytes
msmpeg2adec.dll!CIVIAudioCodec::DecodeAll() + 0x121 bytes
msmpeg2adec.dll!CIVIAudioFilter::Process() + 0xda7 bytes
msmpeg2adec.dll!CIVIAudioFilter::Receive() + 0x16d bytes
msmpeg2adec.dll!CTransformInputPin::Receive() + 0x4c bytes msmpeg2adec.dll!CIVIAudioInPin::Receive() + 0x3f bytes quartz.dll!CBaseOutputPin::Deliver() + 0x22 bytes
quartz.dll!CBaseMSRWorker::TryDeliverSample() + 0x14f bytes
quartz.dll!CBaseMSRWorker::PushLoop() + 0x1da bytes
quartz.dll!CBaseMSRWorker::ThreadProc() + 0x90 bytes
quartz.dll!CAMThread::InitialThreadProc() + 0x1c bytes kernel32.dll!BaseThreadInitThunk() + 0xd bytes ntdll.dll!RtlUserThreadStart() + 0x21 bytes

In other words the pipeline is:

1. Native code function named CEnumConnectedPins()
2. Native to managed transition -> If exception is thrown here then the xxx from step 3 is only known to marshaller but we're in the middle of nowhere.
3. Managed code method named xxx.

So this doesn't get me anywhere and I don't know how to debug that.

Answer 1

This is very ugly to debug, the failure occurs in code you didn't write. Go slower to diagnose this. Write a native test program that obtains the interface pointer you want to test and verify the methods one by one, in v-table order. The bad one will pop out.

Beware that C# doesn't support multiple inheritance. Any COM interface that inherits from another interface that isn't IUnknown or IDispatch requires that you repeat the declarations of the methods in the base interface. Forgetting to do this causes the wrong method to be called. Or a non-existing one since the v-table is too short. NullReference or AccessViolation is then a common outcome.