How to make my program DEP-compatible?

Question

I have a windows forms (.net 3.0) project that won't run on my customer's vista computer due to a DEP error. It runs on my vista machine, and in a clean version of vista sp1 in a virtual machine. I am having trouble tracking down ways to make my program DEP, Data Execution Prevention compatible. I really can't do anything to end user machines, it just has to run. Is there any way out of this latest vista development nightmare? My program uses devexpress controls, sql express, and the .net ie web browser control. I've already jumpered out the ie control, but to no avail. I have other program that use devexpress and sql express on that same machine and they run ok. I am at a loss to debug this on the user's computer.

Answer 1

DEP runs in one of two modes:

1) Hardware DEP is for CPUs that can mark memory pages as non-executable. This helps to prevent certain exploits such as buffer overflows.

2) Software DEP is for CPUs that do not have hardware DEP support. It doesn't prevent execution of code in data pages, but instead stops SEH overwrite (another type of attack).

On Windows XP with CPUs that support it, hardware DEP is enabled by default only for certain Windows system binaries, and also for programs that choose to "opt-in".

On Vista with CPUs that support it, hardware DEP is enabled by default for nearly all processes. This can occasionally be problematic, usually for older programs and drivers, and for ISVs that haven't done any Vista testing.

So I suspect that the first step is to discover whether you're dealing with software or hardware DEP. Also, are you using C#/VB or Managed C++? And are you using any native code or components? If your application uses a native component or an ActiveX control that was built using the old ATL framework, then it's quite possible that your application will fail with hardware DEP.

Since .NET Framework 2.0 SP1, I believe that the C# compiler emits managed code which is DEP-compatible. But if your application is generating DEP exceptions, then you can try clearing the IMAGE_DLLCHARACTERISTICS_NX_COMPAT flag for your executable. To do this you use EDITBIN.EXE from the VC toolset like so:

editbin.exe /NXCOMPAT:NO <your binary>

If you're using Visual Studio, you can add a post-build step to your executable's project. You'll need to setup the environment so that EDITBIN's dependencies can be resolved. When I'm using native code as part of my app, the post-build step looks like this:

call $(DevEnvDir)..\tools\vsvars32.bat
editbin.exe /NXCOMPAT:NO $(TargetPath)  

Answer 2

Older versions of ATL are not DEP aware, so if you use any ActiveX controls that are built using ATL, and were built on that version of ATL (version 7.1 and below, I think), you'll get DEP errors.

As a last resort, you can in fact disable DEP for the process by calling an API function: SetProcessDEPPolicy.

More information on SetProcessDEPPolicy

Answer 3

The compilers that shipped with .NET 2.0 SP1 turn on the NXCOMPAT flag in the executable file header. You can turn that flag off in a Post Build step by running EditBin.exe with the /NXCOMPAT:NO option.

Answer 4

FWIW, it's worth explicitly mentioning that in many cases, applications aren't "incompatible with DEP" but rather were about to crash anyway and DEP "dove in to save the day." Very often, once you disable DEP, you'll find that you're hitting an "ordinary" AV.

If your project is written solely in .NET 3.0, this is almost certainly the case, because .NET doesn't do any of the "crazy" things that trigger DEP (e.g. function thunking, etc).

To debug, install a debugger or enable Watson to generate a .DMP file, then take that .DMP file to the developer's machine and figure out what went wrong.