Are there any compiler options to make x64 release crash dumps more usable?

Question

Whenever I get a crash dump for an x64 release build of my application I find its rare I can view the locals, this makes it hard or impossible to fix some issues. In x86 I can usually view all locals without any issues.

Are there any compiler options in the release build that will allow me to view the locals in release build crash dumps? Obviously I don't want to turn off optimizations but perhaps there is some way to force it to save the locals with minor performance impact?

Answer 1

You've said a couple things that hint at why you can't see locals...

#1 - It's a release build.

With certain optimizations turned on, the compiler is free to do a few things that make looking at locals more difficult.

1. It can inline a function. When this happens, the locals of the function that aren't optimized away are mixed with the calling stack frame.
2. It can free up a register and save a couple clock cycles on the function call using a trick called Frame-Pointer Omission.
3. In order to save stack space, the compiler can reuse the location that held a variable earlier in the function body to hold a different variable later in the function. This means that where you are in the function determines which locals you are actually able to see.

#2 - It's an x64 build.

MSVC uses a new calling convention for 64-bit code aptly called the x64 Calling Convention. The first 4 function arguments are stored in registers instead of on the stack. This means that even though you are looking at a stack frame, you will not see some of the arguments and you may not even be able to recover them if the registers have been reused for something else by the time you look at them.

---

So, now what?

Now that we know why you are going to have such a difficult time, let's see what you can do to get around the issues above. None of those issues are really show stoppers, but they all work together to make things just that much more difficult for you.

1. Turn off some optimizations. You can try making with a release build with optimizations at a level that doesn't impede debugging quite so much. You would probably want to start with the optimizations mentioned above that play with stack frames (/Oy and /Ob). Then you would need to hope that you can still reproduce the issue with those optimizations turned off. Also, depending on your internal policy and the contract that you have with your customer, you may need to involve a lawyer before sending an unofficial build to the customer -- probably not the most fun thing in the world.

2. Build a better symbol file. VS2012 and higher has a new compiler switch, /d2Zi+ in VS2012 and /Zo in VS2013, that generates better debug info when optimizations are turned on. This puts debugging optimized code on par with GCC/Clang. Even though it is undocumented in VS2012, I'd still consider it pretty safe since I saw no difference in the generated code -- only in the symbol file. You may even be able to rebuild locally with this flag and force windbg to use your new symbol file via .symopt+ 0x40. This gives you a chance to get more out of the dumps you already have.

3. Use windbg extensions to do the heavy lifting. In other StackOverflow answers, I've mentioned a tool called CMKD that has saved my bacon a couple times. It, among other things, attempts to reconstruct the arguments in the x64 calling convention that were passed in registers. It's not a sure thing, but it is probably the best hope of getting them back.

Anyway, I hope my ramblings will prove helpful in your debugging.