Mapping process and mapped file have different Team IDs

Question

macOS 10.14.5 --> Sorry my Catalina is in quarantine at work! Xcode 11.2.1

com.apple.security.cs.disable-library-validation does not behave as documented.

My test application is X11 (XQuartz) based and thus needs libraries from /opt/X11/ and /usr/lib/ which have a different team ID from mine.

The app contains a C binary which and works calls "winteracter" a small FORTRAN binary which displays an X11 window and dialogue. "winteracter" is signed/hardened/sandboxed (inherit) before importing to Xcode. It is placed in the Resources folder but placing it in MacOS makes no difference.

The entitlements are as follows: Main application:

<key>com.apple.security.cs.disable-library-validation</key>
<true/>
<key>com.apple.security.temporary-exception.files.absolute-path.read-only</key>
<array>
    <string>/opt/X11/lib/</string>
    <string>/usr/lib/</string>
</array>

Auxiliary binaries:

<key>com.apple.security.inherit</key>
<true/>

The application successfully notarizes (thus signing, hardening and sandboxing are Okay) High Sierra: It works. Mojave: it gives the runtime error (Console:error+fault+winteracter):

Library Validation failed: Rejecting '/opt/X11/lib/libXt.6.dylib' (Team ID: NA574AWV7E, platform: no) for process 'winteracter(11592)' (Team ID: P65398CN49, platform: no), reason: mapping process and mapped file (non-platform) **have different Team IDs**

This is odd for two reasons: 1- the rejection of different team IDs contradicts with Apple's description of "disable-library-validation" 2- december 2019 I got a functional notarized application by simply using the command line codesign with "-o runtime". This no longer works.

Any suggestion ?

Answer 1

Here a more detailed description of my solution.

I finally found a solution to obtain a notarised application which is functional. The bundle is organised as follows:

  TestApp.app
     Contents
        MacOS
           testAppp  <--  a launcher (C binary)
           myAlert     <-- an alert dialogue (Cocoa binary)
        Resources
           winteracter <-- an X11 (Xquartz) window and dialogue (Fortran binary)

The binary "winteracter" is based on the Winteracter library <http://www.winteracter.com> which uses OpenMotif and Xquartz

winteracter binary

This binary is hardened manually (i.e. from outside Xcode) prior to the others:

codesign -d --force --options runtime --verbose=4 -s "$DEV_CERTIFICATE" --entitlements "winteracter.entitlements" "winteracter"

winteracter.entitlements should only contain the following key:

<key>com.apple.security.cs.disable-library-validation</key>
<true>

testApp and myAlert binaries

These are signed and hardened either manually:

codesign -d --force --options runtime --verbose=4 -s "$DEV_CERTIFICATE" "$MYBINARY"

or using Xcode:

In that case be sure to manually erase the field "*Code Signing Entitlements*" in Build Settings.

It may remain filled in even if you delete "Sandboxing" from the Capabilities interface.  

Once notarized the application is functional. I tried sandboxing it but the infamous error "mapping process and mapped file (non-platform) have different Team IDs" went back !

But what were my mistakes?

1- Apple says that sandboxing keys must appear only once in the entitlement file of the main binary, with others having only the "inherit" key. I wrongly assumed that com.apple.security.cs.disable-library-validation was inherited too.

2- I thought the key com.apple.security.app-sandbox was compulsory in entitlement files. Not at all!

3- I wrongly assumed that setting com.apple.security.app-sandbox to false is equivalent to no entitlement file. Wrong again!

4- I wrongly assumed that the Xcode interface "Signing & Capabilities" correctly manages the Build Settings.

5- I performed so many trial an errors that I do not remember all by all dead ends where I got lost.