"xxx.exe is not a valid Win32 application" after VS just built it

Question

I have been successfully developing a WinAPI application in visual studio 2015 (using the IDE) on my Windows-7-64 PC. I usually test the program in Release mode.

I then made some edits to my source. The program compiled and linked without error but the program did not behave quite as I expected, so I switched to Debug mode and tried to build and run. Again VS compiled and linked without error but then complained

"Unable to start program 'f:\dropbox\blah\x64\Debug\xxx.exe'. 'f:\dropbox\blah\x64\Debug\xxx.exe' is not a valid Win32 application".

I thought it was strange, so I flipped back to release mode and tried again - the program started fine. I made some edits and re-built a few times, but then later VS declared

"Unable to start program 'f:\dropbox\blah\x64\Release\xxx.exe'. 'f:\dropbox\blah\x64\Release\xxx.exe' is not a valid Win32 application".

I tried clean all, restarted VS, even restarted my PC.. but all to no avail, I still get exactly the same errors.

EDIT: After reading about similar reports, I tried pausing dropbox syncing. It then seemed to work but only once or twice and then the problem returned. I then tried switching off multi-processor compilation and this seems to have allowed the release version of my program to run again. I have since edited-rebuilt-run many (50+?) times without problem - but it still refuses to run the debug version.

EDIT: FYI my antivirus software is Microsoft Security Essentials

EDIT: calling dumpbin and passing my (non-running debug exe) produces the following output:

File Type: EXECUTABLE IMAGE

  Summary

        1000 .00cfg
    77BB8000 .data
        1000 .gfids
        4000 .idata
        4000 .pdata
       31000 .rdata
        4000 .reloc
        1000 .rsrc
       DD000 .text

EDIT: Just tried compile-build-run on a different machine altogether (windows-10-64) that was linked via dropbox and have exactly the same symptoms, i.e. runs in release mode but not in debug mode.

EDIT: On the advice of Michael Burr I ran dependancy walker on my (non-working) debug exe and it reported these errors: then out of curiosity I thought I'd have a look at what dep-walker said about my (working) release exe and found that I got exactly the same list of errors!... upon more searching I found this SO question in which it was concluded: "The gist of it: As someone elsewhere stated, the tool is a bit dated by now and does not always work properly with newer OS. Thus keep an eye open and don't get mislead by missing 'API-MS-WIN-CORE-COM-L1-1-0.DLL', ... the problem probably lies entirely elsewhere."

EDIT: I switch between debug and release mode from the selection box on the left in the image below and I run the program by clicking the green triangle.

EDIT: I generated the map file for the debug exe. It's too big to show here, but it starts off with the following lines...

 Timestamp is 5811bed3 (Thu Oct 27 09:46:11 2016)

 Preferred load address is 0000000140000000

 Start         Length     Name                   Class
 0001:00000000 00002840H .text$di                CODE
 0001:00002840 000da860H .text$mn                CODE
 0001:000dd0a0 00001020H .text$mn$00             CODE
 0001:000de0c0 00001eb0H .text$x                 CODE
 0001:000dff70 0000104bH .text$yd                CODE
 0002:00000000 00000110H .CRT$XCA                DATA
 0002:00000110 00000110H .CRT$XCAA               DATA
 0002:00000220 00000110H .CRT$XCL                DATA
 0002:00000330 00000128H .CRT$XCU                DATA
 0002:00000458 00000110H .CRT$XCZ                DATA
 0002:00000568 00000110H .CRT$XIA                DATA
 0002:00000678 00000110H .CRT$XIAA               DATA
 0002:00000788 00000110H .CRT$XIAC               DATA
 0002:00000898 00000110H .CRT$XIZ                DATA
 0002:000009a8 00000110H .CRT$XPA                DATA
 0002:00000ab8 00000110H .CRT$XPZ                DATA
 0002:00000bc8 00000110H .CRT$XTA                DATA
 0002:00000cd8 00000118H .CRT$XTZ                DATA
 0002:00000df0 0002c960H .rdata                  DATA
 0002:0002d750 00000998H .rdata$r                DATA
 0002:0002e0e8 00000178H .rdata$zzzdbg           DATA
 0002:0002e260 00000110H .rtc$IAA                DATA
 0002:0002e370 00000188H .rtc$IMZ                DATA
 0002:0002e4f8 00000110H .rtc$IZZ                DATA
 0002:0002e608 00000110H .rtc$TAA                DATA
 0002:0002e718 00000188H .rtc$TMZ                DATA
 0002:0002e8a0 00000110H .rtc$TZZ                DATA
 0002:0002e9b0 00003b68H .xdata                  DATA
 0002:00032518 00000275H .xdata$x                DATA
 0002:0003278d 00000000H .edata                  DATA
 0003:00000000 000023e0H .data                   DATA
 0003:000023e0 00000580H .data$r                 DATA
 0003:00002960 77376001H .bss                    DATA
 0004:00000000 0000369cH .pdata                  DATA
 0005:00000000 00000ed0H .idata$5                DATA
 0005:00000ed0 000000c8H .idata$2                DATA
 0005:00000f98 00000018H .idata$3                DATA
 0005:00000fb0 00000ed0H .idata$4                DATA
 0005:00001e80 00001fc6H .idata$6                DATA
 0006:00000000 0000015eH .gfids$y                DATA
 0007:00000000 0000011bH .00cfg                  DATA
 0008:00000000 00000170H .rsrc$01                DATA
 0008:00000170 000002ccH .rsrc$02                DATA

  Address         Publics by Value              Rva+Base               Lib:Object

 0000:00000000       __guard_iat_table          0000000000000000     <absolute>
 0000:00000000       __guard_longjmp_count      0000000000000000     <absolute>
 0000:00000000       __guard_longjmp_table      0000000000000000     <absolute>
 0000:00000000       __guard_fids_count         0000000000000000     <absolute>
 0000:00000000       ___safe_se_handler_table   0000000000000000     <absolute>
 0000:00000000       ___safe_se_handler_count   0000000000000000     <absolute>
 0000:00000000       __guard_iat_count          0000000000000000     <absolute>
 0000:00000000       __guard_fids_table         0000000000000000     <absolute>
 0000:00000000       __dynamic_value_reloc_table 0000000000000000     <absolute>
 0000:00000100       __guard_flags              0000000000000100     <absolute>
 0000:00000000       __ImageBase                0000000140000000     <linker-defined>
 0001:00002aa0       ?readstring@@YAXPEAD0@Z    0000000140003aa0 f   COMMAND.obj
 0001:00002b70       ?make_phere@@YAXH@Z        0000000140003b70 f   COMMAND.obj
 0001:00002c50       ?load_snap@@YAXXZ          0000000140003c50 f   COMMAND.obj
 0001:00002d30       ?i_rand_0_n_inclusive@@YAHH@Z 0000000140003d30 f   COMMAND.obj

Answer 1

  77BB8000 .data

That's almost certainly the problem, you have a very large data section. Its size is suspiciously close to what it is possible for a single executable module on Windows. You can get a more consistent repro from this sample C program:

unsigned char kaboom[0x7d000000];

int main()
{
    return 0;
}

Not a very good error message btw, Microsoft did not reserve an error code for this corner-case. And sure, it is not going to repeat that well when you are close to the edge with 0x77BB8000. The executable image must fit a single view of the memory-mapped file that the loader creates to map the code and data into memory. The view has a hard upper-limit of 2 gigabytes, fundamental for a 32-bit process and an MMF view size restriction even on the 64-bit version of Windows.

The amount of space that is available for that data section varies from one run to the next. Subtracted from the view size are the non-mappable regions at the start and the end of the address space and the space required for the operating system DLLs (at least ntdll.dll and kernel32.dll) in a 32-bit EXE process. And the space you lose due to ASLR (Address Space Layout Randomization), a number that changes. And DLLs that are injected, like those used by anti-malware and Dropbox.

It can't be guessed why your data section needs to be so large. Ask the linker to generate a .map file so you get a breakdown of the section, the large global variable ought to jump out. Be sure to target x64 so you have lots of address space available and use the free store (malloc etc) to allocate large arrays.