Debugging sporadic app crashes with dylib in iOS13/iPadOS 13

Question

After updating to iOS 13.x / iPadOS 13.x we see sporadic crashes with our recent Testflight builds. We've not seen reports from users using our pre-13 released version yet, but it seems not many have updated to 13 yet, so we really don't know.

The app is a cordova-app with cordova-ios 5.0.2 using WkWebView. Data protection entitlement is set to full.

I have a hard time making sense of the crash, and it seems to be somewhere deep in the iOS-stack? The crash is sporadic and seems only to happen occasionally when the app has been put into the background.

I'm looking for some guidance/pointers on how to get to the bottom of this.

Crash report from TestFlight:

Incident Identifier: 900F9C19-EE4A-4A9D-B1AB-E834F6387565
Beta Identifier:     7194E7C0-152C-43E4-9716-BE2AF29A0BD7
Hardware Model:      iPad7,5
Process:             SomeApp [677]
Path:                /private/var/containers/Bundle/Application/745F3054-AB4B-4A1A-A7AB-2AFD0516706C/SomeApp.app/SomeApp
Identifier:          SomeApp
Version:             380 (2.0.0)
AppStoreTools:       11A1002b
Beta:                YES
Code Type:           ARM-64 (Native)
Role:                Non UI
Parent Process:      launchd [1]
Coalition:           SomeApp [620]


Date/Time:           2019-10-03 07:10:10.2716 +0200
Launch Time:         2019-10-02 15:50:25.8963 +0200
OS Version:          iPhone OS 13.1.2 (17A860)
Release Type:        User
Baseband Version:    n/a
Report Version:      104

Exception Type:  EXC_BAD_ACCESS (SIGBUS)
Exception Subtype: KERN_MEMORY_ERROR at 0x00000001048a5c8c
VM Region Info: 0x1048a5c8c is in 0x104884000-0x1048a8000;  bytes after start: 138380  bytes before end: 9075
      REGION TYPE                      START - END             [ VSIZE] PRT/MAX SHRMOD  REGION DETAIL
      MALLOC_LARGE           0000000104880000-0000000104884000 [   16K] rw-/rwx SM=PRV  
--->  mapped file            0000000104884000-00000001048a8000 [  144K] r--/rw- SM=COW  ...t_id=18017271
      shared memory          00000001048a8000-00000001048ac000 [   16K] r--/r-- SM=SHM  

Termination Signal: Bus error: 10
Termination Reason: Namespace SIGNAL, Code 0xa
Terminating Process: exc handler [677]
Triggered by Thread:  12

Thread 12 name:
Thread 12 Crashed:
0   libdyld.dylib                   0x000000018a2f76c0 dyld3::closure::ObjCStringTable::hash(char const*, unsigned long) const + 16 (Closure.cpp:1339)
1   libdyld.dylib                   0x000000018a2f7cd4 dyld3::closure::ObjCStringTable::getIndex(char const*) const + 52 (Closure.h:840)
2   libdyld.dylib                   0x000000018a2f7a6c dyld3::closure::ObjCStringTable::getPotentialTarget(char const*) const + 20 (Closure.h:824)
3   libdyld.dylib                   0x000000018a2f7d40 dyld3::closure::ObjCClassDuplicatesOpt::getClassLocation(char const*, objc_opt::objc_opt_t const*... + 44 (Closure.cpp:1483)
4   libdyld.dylib                   0x000000018a3044bc dyld3::AllImages::forEachObjCClass(char const*, void (void*, bool, bool*) block_pointer) const + 72 (AllImages.cpp:1915)
5   libobjc.A.dylib                 0x000000018a236c38 getPreoptimizedClass + 148 (objc-opt.mm:279)
6   libobjc.A.dylib                 0x000000018a2218d8 getClassExceptSomeSwift(char const*) + 20 (objc-runtime-new.mm:1607)
7   libobjc.A.dylib                 0x000000018a222784 look_up_class + 100 (objc-runtime-new.mm:6843)
8   Foundation                      0x000000018a8b4b00 NSClassFromString + 200 (NSObjCRuntime.m:0)
9   BoardServices                   0x000000018f2ba1b0 +[BSXPCServiceConnectionProxy invokeMethod:onTarget:withMessage:forConnection:] + 104 (BSXPCServiceConnectionProxy.m:329)
10  BoardServices                   0x000000018f2b9374 -[BSXPCServiceConnectionProxy invokeMessage:onTarget:] + 144 (BSXPCServiceConnectionProxy.m:177)
11  BoardServices                   0x000000018f2c080c __63-[BSXPCServiceConnectionEventHandler connection:handleMessage:]_block_invoke + 428 (BSXPCServiceConnectionEventHandler.m:184)
12  BoardServices                   0x000000018f2d6bf0 BSXPCServiceConnectionExecuteCallOut + 344 (BSXPCServiceConnection.m:1049)
13  BoardServices                   0x000000018f2c062c -[BSXPCServiceConnectionEventHandler connection:handleMessage:] + 172 (BSXPCServiceConnectionEventHandler.m:173)
14  BoardServices                   0x000000018f2d53e4 -[BSXPCServiceConnection _connection_handleMessage:fromPeer:withHandoff:] + 644 (BSXPCServiceConnection.m:808)
15  libdispatch.dylib               0x000000018a1bf610 _dispatch_call_block_and_release + 24 (init.c:1408)
16  libdispatch.dylib               0x000000018a1c0184 _dispatch_client_callout + 16 (object.m:495)
17  libdispatch.dylib               0x000000018a16c464 _dispatch_lane_serial_drain$VARIANT$mp + 608 (inline_internal.h:2487)
18  libdispatch.dylib               0x000000018a16ce88 _dispatch_lane_invoke$VARIANT$mp + 468 (queue.c:3820)
19  libdispatch.dylib               0x000000018a16c330 _dispatch_lane_serial_drain$VARIANT$mp + 300 (inline_internal.h:2528)
20  libdispatch.dylib               0x000000018a16ce88 _dispatch_lane_invoke$VARIANT$mp + 468 (queue.c:3820)
21  libdispatch.dylib               0x000000018a176340 _dispatch_workloop_worker_thread + 588 (queue.c:6386)
22  libsystem_pthread.dylib         0x000000018a20ffa4 _pthread_wqthread + 276 (pthread.c:2323)
23  libsystem_pthread.dylib         0x000000018a212ae0 start_wqthread + 8

Stack after update to iPadOS 13.2, of note: PluginKit:

Thread 5 name:
Thread 5 Crashed:
0   libdyld.dylib                   0x00000001bc5505d0 dyld3::closure::ObjCStringTable::hash(char const*, unsigned long) const + 16 (Closure.cpp:1339)
1   libdyld.dylib                   0x00000001bc550be4 dyld3::closure::ObjCStringTable::getIndex(char const*) const + 52 (Closure.h:841)
2   libdyld.dylib                   0x00000001bc55097c dyld3::closure::ObjCStringTable::getPotentialTarget(char const*) const + 20 (Closure.h:825)
3   libdyld.dylib                   0x00000001bc550c50 dyld3::closure::ObjCClassDuplicatesOpt::getClassLocation(char const*, objc_opt::objc_opt_t const*... + 44 (Closure.cpp:1483)
4   libdyld.dylib                   0x00000001bc55d3cc dyld3::AllImages::forEachObjCClass(char const*, void (void*, bool, bool*) block_pointer) const + 72 (AllImages.cpp:1915)
5   libobjc.A.dylib                 0x00000001bc48fc28 getPreoptimizedClass + 148 (objc-opt.mm:279)
6   libobjc.A.dylib                 0x00000001bc47a7d8 getClassExceptSomeSwift(char const*) + 20 (objc-runtime-new.mm:1620)
7   libobjc.A.dylib                 0x00000001bc47b684 look_up_class + 100 (objc-runtime-new.mm:6880)
8   BaseBoard                       0x00000001bf30447c _BSXPCEncodeObjectForKey + 124 (BSXPCCoder.m:377)
9   BaseBoard                       0x00000001bf30428c -[BSXPCCoder encodeObject:forKey:] + 96 (BSXPCCoder.m:181)
10  RunningBoardServices            0x00000001bf29f604 __44+[RBSXPCMessage messageForMethod:arguments:]_block_invoke + 288 (RBSXPCUtilities.m:152)
11  RunningBoardServices            0x00000001bf29f35c +[RBSXPCMessage messageWithEncoder:] + 72 (RBSXPCUtilities.m:132)
12  RunningBoardServices            0x00000001bf29f408 +[RBSXPCMessage messageForMethod:arguments:] + 148 (RBSXPCUtilities.m:140)
13  RunningBoardServices            0x00000001bf29f788 +[RBSXPCMessage messageForMethod:varguments:] + 192 (RBSXPCUtilities.m:170)
14  RunningBoardServices            0x00000001bf28f234 -[RBSConnection _invalidateAssertionIdentifier:error:] + 144 (RBSConnection.m:1361)
15  RunningBoardServices            0x00000001bf28808c -[RBSConnection invalidateAssertion:error:] + 80 (RBSConnection.m:275)
16  RunningBoardServices            0x00000001bf28605c -[RBSAssertion _clientInvalidateWithError:] + 124 (RBSAssertion.m:317)
17  AssertionServices               0x00000001c0e6278c -[BKSAssertion _invalidateSynchronously:] + 104 (BKSAssertion.m:164)
18  AssertionServices               0x00000001c0e6718c -[BKSProcessAssertion invalidate] + 92 (BKSProcessAssertion.m:291)
19  Foundation                      0x00000001bcb2e624 ___NSExtensionTearDownRequestWithIdentifier_block_invoke_2 + 84 (NSExtension.m:1098)
20  PlugInKit                       0x00000001c9e6db4c -[PKHostPlugIn endUsing:] + 152 (PKHostPlugIn.m:814)
21  Foundation                      0x00000001bcb2d090 __64-[NSExtension _safelyEndUsingWithProcessAssertion:continuation:]_block_invoke + 116 (NSExtension.m:1670)
22  libdispatch.dylib               0x00000001bc418610 _dispatch_call_block_and_release + 24 (init.c:1408)
23  libdispatch.dylib               0x00000001bc419184 _dispatch_client_callout + 16 (object.m:495)
24  libdispatch.dylib               0x00000001bc3c5404 _dispatch_lane_serial_drain$VARIANT$mp + 608 (inline_internal.h:2484)
25  libdispatch.dylib               0x00000001bc3c5df8 _dispatch_lane_invoke$VARIANT$mp + 420 (queue.c:3863)
26  libdispatch.dylib               0x00000001bc3cf314 _dispatch_workloop_worker_thread + 588 (queue.c:6445)
27  libsystem_pthread.dylib         0x00000001bc468f98 _pthread_wqthread + 276 (pthread.c:2323)
28  libsystem_pthread.dylib         0x00000001bc46bad4 start_wqthread + 8

For others experiencing these problems here are some other relevant links:

https://forums.developer.apple.com/thread/123728 https://forums.developer.apple.com/message/384064#384064

Answer 1

It seems that the dyld3 saves the generated clojure files in the app tmp directory and if you use NSFileProtectionComplete the closure files have this property too.

To generate a crahs close the app, lock the phone, send a push that will wake up the app and then the crash is generated.

The solution seems to be simple, just change the permissions for the app tmp direcptry to NSFileProtectionCompleteUntilFirstUserAuthentication and the app will not crash. I really don't know why apple did this.