Mysterious crash inside CoreAnalytics::Client / FrameworkConfiguration::init

Question

I'm getting a strange crash that is proving difficult to debug. I can trigger it at will, immediately after launching my app, simply by pressing and releasing the Command key a couple of times; no other interaction with my app is needed. Needless to say, my app does not do anything when the command key changes state. Other keypresses also trigger the crash, actually, including letter keys; there's nothing special about Command except that it seems unlikely to have other side effects.

The crash is always in a background thread, always with the same backtrace that contains no code of mine whatsoever:

#0  0x00007fff603da3ec in __cxa_throw ()
#1  0x00007fff42e90492 in nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer>::parser::unexpect(nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer>::lexer::token_type) const ()
#2  0x00007fff42e8ccad in nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer>::parser::parse_internal(bool) ()
#3  0x00007fff42e8fe63 in nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer>::parser::parse() ()
#4  0x00007fff42e8fcd4 in nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer> nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer>::parse<char const*, 0>(char const*, char const*, std::__1::function<bool (int, nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer>::parse_event_t, nlohmann::basic_json<std::__1::map, std::__1::vector, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, bool, long long, unsigned long long, double, std::__1::allocator, nlohmann::adl_serializer>&)>) ()
#5  0x00007fff42e826be in FrameworkConfiguration::init() ()
#6  0x00007fff42e82478 in FrameworkConfiguration::create(applesauce::xpc::object) ()
#7  0x00007fff42e82100 in CoreAnalytics::Client::handleConfigurationChange_sync(applesauce::xpc::dict const&) ()
#8  0x00007fff42e81f52 in CoreAnalytics::Client::handleServerMessage_sync(applesauce::xpc::dict) ()
#9  0x00007fff42e94039 in ___ZZN13CoreAnalytics6Client4initEvENK3$_1clEv_block_invoke ()
#10 0x00007fff634b8a54 in _xpc_connection_call_event_handler ()
#11 0x00007fff634b795c in _xpc_connection_mach_event ()
#12 0x000000010040482f in _dispatch_client_callout4 ()
#13 0x0000000100421b3a in _dispatch_mach_msg_invoke ()
#14 0x000000010040bb40 in _dispatch_lane_serial_drain ()
#15 0x0000000100422eda in _dispatch_mach_invoke ()
#16 0x000000010040bb40 in _dispatch_lane_serial_drain ()
#17 0x000000010040cae8 in _dispatch_lane_invoke ()
#18 0x0000000100419f2e in _dispatch_workloop_worker_thread ()
#19 0x000000010049002c in _pthread_wqthread ()
#20 0x000000010048f157 in start_wqthread ()

The nlohmann:: stuff is a JSON library that I recognize because I've used it in other projects, but I do not use it in this project. FrameworkConfiguration:: and CoreAnalytics:: are, according to Google, related to internal Apple analytics-gathering cruft that I have no direct control over. I have turned off all analytics in System Preferences on this machine, in the Security & Privacy panel, and then rebooted the machine; the bug is still 100% reproducible.

The main thread in the app is doing nothing when this occurs; typically it's sitting in mach_msg_trap(). The backtrace on the main thread also has no code of mine, and is the typical macOS backtrace for a CFRunLoop that is idling.

I have tried turning on the various debugging bells and whistles that Xcode provides – address sanitizer, undefined behavior sanitizer, malloc scribble / guard edges / guard malloc / zombie objects / malloc stack. None of those results in any output related to the problem. (I.e., no output except the initial output from the malloc stuff saying that it has been enabled, blah blah blah.)

I'm on Catalina 10.15.3, Xcode 11.3.1. My next step will be to upgrade both of those; I had forgotten that I was out of date, but only slightly, so it'll be surprising if that makes any difference.

Obviously other apps on my system don't crash when I press keys, so there must be something my app is doing that triggers this problem, but I can't imagine what, and I'm at a loss as to how to debug this. I guess if upgrading doesn't help I'll make a copy of my project and start ripping out big chunks of code until the bug no longer reproduces, and try to narrow down what part of the app is responsible. I thought I'd check here first to see if anybody has seen this, though. Thanks.

---

ADDITION: I've just updated to macOS 10.15.4 and Xcode 11.4.1. No change in this crash. Time to start carving. I will post here if I figure out the cause, for others to find if they hit this.

Answer 1

Well, the somewhat unsatisfying answer is that it appears to be the result of some kind of corrupted Xcode project file. I started with a new Cocoa app project created in 11.4.1, and added in source files and resources one by one, testing as I went. I ended up with the full project recreated, and the bug is no longer present. The old Xcode project had been carried forward from older versions, by following all of Xcode's project upgrade recommendations. I've gone through all the build settings for the two projects with a fine-toothed comb, and can't see anything that would explain the difference. It's a mystery. But for posterity: if you see this problem, rebuild your project from scratch.

Answer 2

For me, it wasn't a deepseated issue either.

The fix, was to delete the version of the app on the device (which was a new device I had just registered, which already had the App Store version of the app loaded).

I did also "Clean Build Folder", but the next run, no exceptions were thrown by Apple's CoreAnalytics which clearly has a bug.