Disassemble .NET IL to find error message

Question

My team is attempting to find where an error message is occuring within a .NET windows service, that encapsulates a web service, which was built by Microsoft and resides on our servers, to support a business product we use. Microsoft has personally told us that the error we are recieving is in fact one of their errors, however they cannot identify where or why it is occuring. It is a critical business process, causing much heartache to our team.

This error occurs, as far as we can tell, randomly, and has no comminalities that we can discover by looking at the stack traces. The last call on the stack trace is our method call attempting to read the response from the service, giving us no hint where or why the error is occuring within the method call to the service. The Microsoft windows service is not logging any errors to the event viewer.

We would like to trace where this error is coming from by looking for the literal error message string that is being thrown. If we can identify the method call generating this error, we can be one step closer to narrowing down how to tackle this error.

While we don't have access to the source code to this service, we can however view the IL by using the IL Disassembler (idlasm.exe). I have dumped the IL for a few of the assemblies, but their are no literal strings.

Is it possible to find a literal string within the IL? I assume it's encoded -- I just don't know enough about IL to determine where to look, or how to identify a string within the IL. If it doesn't exist as a literal string with a method in the IL, and is stored in a resource, where else could the error message be found?

The stack trace, with the app names replaced.

Raw Error:  There is insufficient memory to execute this function.
This can be caused by recursive function calls. Contact your system administrator.

at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)   
at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
at Product.DoSomething() in c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\da35f853\f48bba34\App_WebReferences.jygjwt-a.0.cs:line 701
at Product.DoSomething() in E:\Sites\Example.com\App_Code\File.vb:line 1105
at ASP.shop_checkout_aspx.ProcessCheckout() in E:\Sites\Example.com\processcheckout.aspx:line 546

A few side notes is that there is plenty of memory availible at the time of this error message, and when this error occurs, every subsequent call to any method returns this same error. It will only subside by restarting the Microsoft services.

We have narrowed down every single call to the services used by our site, but cannot reproduce this message. We can overload the service in development with about 1000x the traffic as seen in production, and cause an out of memory exception (by consuming all of the memory), but not the message which states, "Can be caused by a recursive function call." Agian, which Microsoft states is a unique error that their product specifically throws.

Update

We have managed to find the literal error message within an .etx file, which looks like the following:

//   ÚÄÄÄÄÄÄÄÄÄ¿
// ÚÄ´  STACK  ÃÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ¿
// ³ ÀÄÄÄÄÄÄÄÄÄÙ                                                              ³
// ³                                                                          ³
// ³  Stack Manager                                                           ³
// ³                                                                          ³
// ÀÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÙ
@1@15@0@
@2@1@0@
There is insufficient memory to execute this function.

This can be caused by recursive function calls that are used in the program. Contact your system administrator.
@2@END@0@

I assume that the @1@15@0@, @2@1@0@, and @2@END@0@ are some kind of reference that the assemblies use to reference these strings. Does anyone have any information on how to tie these back to one of the assemblies that use these strings? Thus far there doesn't seem to be any data in the IL dumps that match the format with the @ symbols.

Answer 1

You very likely need to look at the other end of the wire for the source of the exception. Don't forget, you're using Soap. The SoapMessage.Exception property gets thrown inside SoapHttpClientProtocol.ReadResponse() when it is not null.

The exception text hints at a so-called Soft Stack Overflow. Windows exception code 0xe053534f (googles well), thrown when interop code discovers there is not enough stack space left to run a function without causing a hard stack overflow.