Wrong line numbers in stack trace

Question

The problem

On our ASP .net website I keep getting wrong line numbers in stack traces of exceptions. I am talking about our live environment. There seems to be a pattern: The stack trace will always point to the line that contains the method's closing curly brackets.

For Example:

public class Foo {     public void Bar()     {         object someObject = null;         someObject.ToString();      /*           arbitrarily more lines of code      */      } // this line will be the one that the stack trace points to } 

More details

• To be clear: This does not only happen for some method(s), it happens for every exception that we log. So I would rule out (JIT) optimizations here, that might lead to line numbers being seemingly randomly off. What bothers me is, that the wrong line numbers seem to consistently point to the closing curly brackets of the containing method.

• Note that before .net 4.6 there actually was a bug in the framework, such that if you compiled targeting x64, exactly this would happen. However, Microsoft confirmed that this has been fixed. Also running a minimal example app for this, reaffirms their claim. But for some reason it still happens on the web servers.

• It puzzles me even more that it does not happen in our test and development environments. The test and live systems are pretty similarly set up. The only real difference is that live we are running Windows Server 2012, while on the test system we are still using Windows Server 2008.

What I have checked

• pdb files are valid (tested with chkmatch)
• Compile time code optimizations are not the issue
• The aforementioned bug in .net before 4.6 cannot be reproduced with a minimal example
• .NET versions are exactly the same
• Build comes from the same deploy script

Any clues towards solving this problem are highly appreciated. If you know anything that we could check, please let me know.

Answer 1

Compilation optimization as well as runtime optimization can change the actual execution, as well as the call stack information constructed. So you cannot treat the numbers that seriously.

More information can be found in posts such as Scott Hanselman's: Release IS NOT Debug: 64bit Optimizations and C# Method Inlining in Release Build Call Stacks.

It would be too difficult to troubleshoot a specific case without touching your bits. But if you know of WinDbg you might dive deeper, by live debugging the application when the exception occurs. Then you can dump the actual jitted machine code as well as other runtime information, so as to determine how the line number is constructed.

Answer 2

Thanks everyone for your help! We found out that the SCOM APM agent running on the machines in production caused our application to log wrong line numbers in the stack traces. As soon as we deactivated the agent, the line numbers were correct.