Self-modifying code on Darwin 10.15 resulting in "malformed mach-o image"?

Question

I have a program that generates self-modifying code (see https://tigress.wtf/selfModify.html in case you're interested). It runs on x86 Darwin and Linux. On Darwin, I compile with

gcc -g -segprot __TEXT rwx rwx self_modifying.c -o self_modifying.exe

Recently, this seems not to work, I get

dyld: malformed mach-o image: __TEXT segment maps start of file but is writable

when I run the program.

I'm running clang version 6.0.1 on MacOS 10.15.3. Any help would be appreciated.

Answer 1

@AlexDenisov is pretty close, but the restriction does not apply only to executables running on Catalina with min macOS 10.15.0 and higher.

There are 2 formats of Mach-O load command that indicate the min MacOS the executable itself can use:
- LC_BUILD_VERSION(the new one introduced around 10.14 if I recall correctly)
- LC_VERSION_MIN_MACOSX (the legacy one)

Even with fallbacking to older MacOS version using LC_VERSION_MIN_MACOSX:

gcc -segprot __TEXT rwx rwx -mmacosx-version-min=10.6 self_modifying.c 

we run into the same problem.

To bypass the check one solution I found to work is to get rid of min macos version altogether. I'm not aware though of any gcc or ld flag that can achieve this. Fortunately we can do the processing after linking the executable with the help of Jonathan Levin's jtool2

So the chain of commands become:

gcc -segprot __TEXT rwx rwx self_modifying.c 
jtool2 -l a.out                             
    LC 00: LC_SEGMENT_64             Mem: 0x000000000-0x100000000   __PAGEZERO
    LC 01: LC_SEGMENT_64             Mem: 0x100000000-0x100001000   __TEXT
        Mem: 0x100000f60-0x100000f83        __TEXT.__text   (Normal)
        Mem: 0x100000f84-0x100000f8a        __TEXT.__stubs  (Symbol Stubs)
        Mem: 0x100000f8c-0x100000fa6        __TEXT.__stub_helper    (Normal)
        Mem: 0x100000fa6-0x100000fb2        __TEXT.__cstring    (C-String Literals)
        Mem: 0x100000fb4-0x100000ffc        __TEXT.__unwind_info    
    LC 02: LC_SEGMENT_64             Mem: 0x100001000-0x100002000   __DATA_CONST
        Mem: 0x100001000-0x100001008        __DATA_CONST.__got  (Non-Lazy Symbol Ptrs)
    LC 03: LC_SEGMENT_64             Mem: 0x100002000-0x100003000   __DATA
        Mem: 0x100002000-0x100002008        __DATA.__la_symbol_ptr  (Lazy Symbol Ptrs)
        Mem: 0x100002008-0x100002010        __DATA.__data   
    LC 04: LC_SEGMENT_64             Mem: 0x100003000-0x100004000   __LINKEDIT
    LC 05: LC_DYLD_INFO             
           Rebase info: 8     bytes at offset 12288 (0x3000-0x3008)
           Bind info:   24    bytes at offset 12296 (0x3008-0x3020)
        No Weak info
           Lazy info:   16    bytes at offset 12320 (0x3020-0x3030)
           Export info: 48    bytes at offset 12336 (0x3030-0x3060)
    LC 06: LC_SYMTAB                
    LC 07: LC_DYSYMTAB              
            1 local symbols at index     0
            2 external symbols at index  1
            2 undefined symbols at index 3
           No TOC
           No modtab
            3 Indirect symbols at offset 0x30b8
    LC 08: LC_LOAD_DYLINKER         /usr/lib/dyld
    LC 09: LC_UUID                  UUID: 6AE91487-DB61-3FA8-8DBE-686FEC1DA8FC
    LC 10: LC_BUILD_VERSION         Build Version:           Platform: MacOS 10.15.0 SDK: 10
    LC 11: LC_SOURCE_VERSION        Source Version:          0.0.0.0.0
    LC 12: LC_MAIN                  Entry Point:             0xf60 (Mem: 0x100000f60)
    LC 13: LC_LOAD_DYLIB            /usr/lib/libSystem.B.dylib
    LC 14: LC_FUNCTION_STARTS       Offset:     12384, Size:      8 (0x3060-0x3068)
    LC 15: LC_DATA_IN_CODE          Offset:     12392, Size:      0 (0x3068-0x3068)
jtool2 -rc 10 --inplace a.out

Now your a.out should launch correctly :-)

Answer 2

The problem you are observing is the restriction of macOS Catalina and not related to your compiler.

Looking at the dyld source code (can be found here https://opensource.apple.com/release/macos-1015.html) the error message is coming from this code:

if ( (segCmd->initprot & VM_PROT_WRITE) == VM_PROT_WRITE ) {
  if ( context.strictMachORequired )
    dyld::throwf("malformed mach-o image: %s segment maps start of file but is writable", segCmd->segname);
}

The exception is thrown only when strictMachORequired, which is always true on macOS 10.15 or higher, based on the other snippet from the dyld sources:

#if __MAC_OS_X_VERSION_MIN_REQUIRED
  gLinkContext.strictMachORequired = false;
  // <rdar://problem/22805519> be less strict about old macOS mach-o binaries
  ((dyld3::MachOFile*)mainExecutableMH)->forEachSupportedPlatform(^(dyld3::Platform platform, uint32_t minOS, uint32_t sdk) {
    if ( (platform == dyld3::Platform::macOS) && (sdk >= DYLD_PACKED_VERSION(10,15,0)) ) {
      gLinkContext.strictMachORequired = true;
    }
  });
  if ( gLinkContext.iOSonMac )
    gLinkContext.strictMachORequired = true;
#else
  // simulators, iOS, tvOS, watchOS, are always strict
  gLinkContext.strictMachORequired = true;
#endif