Ollydbg 1.10 "Back to user mode" doesn't work

Question

I tried to learn "Lena's reversing for newbies", when some trouble arise. I start Pixtopian Book with ollyDbg, then try to have MessageBox with message about uregistered version. Then i switch to OllyDbg, stop program executing and press "Alt+F9" for "Back to user mode" which stop the program after it exit from DLL.

But after this program does not work, it's frozen and does not respond to my actions. If i turn off "Back to user mode" program normally work.

What's the problem? Can i try to use "Back to user mode" in IDA (uses WinDbg) or some other debugger and How i can do this? Can i repair it function in OllyDbg?

P.S. It's like the program stopped and didn't run after use "ALT+F9".

/Sorry for my English, i'm just learning ;-)/

Answer 1

First of all, Ollydbg is meant for 32 bit platform. It means that it will run only on a 32 bit OS and can only debug 32 bit apps.

In x64 Operating Systems (specifically Windows) there is a feature called compatibility mode that lets you run 32 bit apps. The 32 bit code is either run by emulation or natively (when the hardware itself implements the instruction set and then it is called x86-64).

So, when you try to run Ollydbg in a x64 environment it will run, but you will experience problems like the one you are facing. It occurs because Ollydbg is run in an emulation mode. Being a debugger it needs access to the registers and other system structures, which it is denied. What it can see is a virtual image of the system.

So the solution to the problem is using a Virtual Machine.

You would install a 32 bit OS in it and debug the app using Ollydbg. As far as Virtual Machines are concerned, I would recommend VMWare. You can use either the workstation or player version. The latter is free but does not support snapshots.

Other solutions are Virtual Box, Parallels Workstation and Microsoft Virtual PC. The disadvantages of them are that Virtual Box does not support hardware breakpoints, Parallels Workstation is no longer supported as of 2014 and moreover there you would get a BSOD if you try to single step through FPU instructions. I have not tested Virtual PC though.

Note : Ollydbg does not supports x64 but its author is working on a x64 version.