Android Firebase Crashlytics NDK backtraces are missing frames

Question

The problem was there when I was using Fabric Crashlytics and now with Firebase Crashlytics latest version it still exists. Everything is implemented based on official NDK docs and the symbol (cSYM) files are uploaded properly after every build. I managed to produce an intentional crash to compare the tombstone and reported native crash backtrace, here is the result:

Reported crash on Firebase console:

The crash saved tombstone file:

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'samsung/klteduosxx/klte:5.0/LRX21T/G900FDXXU1BOJ1:user/release-keys'
Revision: '14'
ABI: 'arm'
pid: 652, tid: 1160, name: AsyncTask #5  >>> com.mnhaami.pasaj <<<
signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
Abort message: 'stack corruption detected'
    r0 00000000  r1 00000488  r2 00000006  r3 00000000
    r4 656f3db8  r5 00000006  r6 00000016  r7 0000010c
    r8 656f3878  r9 00000000  sl 134eaf00  fp 13f14800
    ip 00000488  sp 656f37f8  lr b6eeefd5  pc b6f11b74  cpsr 600d0010
    d0  ffffffffffffffff  d1  ffffffffffffffff
    d2  ff816306ff836406  d3  ff795f04ff7e6206
    d4  ffff000000000000  d5  0000000000000003
    d6  ffff000000000000  d7  ffffffffffffffff
    d8  0000000000000000  d9  0000000000000000
    d10 0000000000000000  d11 0000000000000000
    d12 0000000000000000  d13 0000000000000000
    d14 0000000000000000  d15 0000000000000000
    d16 0000000000000000  d17 6e6f697470757272
    d18 ff916d0aff997208  d19 ff825f0aff896708
    d20 ff715613ff785a11  d21 ff6b520fff6b520f
    d22 ff6f5309ff6e530e  d23 ff775b07ff735609
    d24 0000000000000000  d25 0000000000000000
    d26 0002000100010001  d27 0002000200020002
    d28 0080008000800080  d29 0080008000800080
    d30 0800080008000800  d31 0800080008000800
    scr 2800001b

backtrace:
    #00 pc 00036b74  /system/lib/libc.so (tgkill+12)
    #01 pc 00013fd1  /system/lib/libc.so (pthread_kill+52)
    #02 pc 00014bef  /system/lib/libc.so (raise+10)
    #03 pc 00011531  /system/lib/libc.so (__libc_android_abort+36)
    #04 pc 0000fcbc  /system/lib/libc.so (abort+4)
    #05 pc 00012701  /system/lib/libc.so (__libc_fatal+16)
    #06 pc 000368fb  /system/lib/libc.so (__stack_chk_fail+6)

stack:
         656f37b8  00000001  
         656f37bc  00430000  
         656f37c0  00550000  
         656f37c4  00000000  
         656f37c8  00430000  
         656f37cc  b4e18218  
         656f37d0  00000000  
         656f37d4  2f94ff4a  /dev/ashmem/dalvik-main space (deleted)
         656f37d8  9866fb50  
         656f37dc  2f94ff4a  /dev/ashmem/dalvik-main space (deleted)
         656f37e0  656f3db8  [stack:1160]
         656f37e4  00000006  
         656f37e8  00000016  
         656f37ec  00000488  
         656f37f0  00000000  
         656f37f4  b6eeefcd  /system/lib/libc.so (pthread_kill+48)
    #00  656f37f8  656f3db0  [stack:1160]
         ........  ........
    #01  656f37f8  656f3db0  [stack:1160]
         656f37fc  00000006  
         656f3800  00000000  
         656f3804  656f3820  [stack:1160]
         656f3808  656f3898  [stack:1160]
         656f380c  b6eefbf3  /system/lib/libc.so (raise+14)
    #02  656f3810  656f381c  [stack:1160]
         656f3814  b6eec535  /system/lib/libc.so (__libc_android_abort+40)
    #03  656f3818  000000a8  
         656f381c  ffffffdf  
         656f3820  00000000  
         656f3824  8c93ac00  
         656f3828  ae907e40  
         656f382c  2f94ff4a  /dev/ashmem/dalvik-main space (deleted)
         656f3830  93bcd2fc  
         656f3834  b4bc0435  /system/lib/libart.so (art::CheckJNI::GetIntArrayRegion(_JNIEnv*, _jintArray*, int, int, int*))
         656f3838  b6f44e04  
         656f383c  b6eeacc0  /system/lib/libc.so (__bionic_clone)
    #04  656f3840  00000000  
         656f3844  b6eed705  /system/lib/libc.so (__fortify_chk_fail)
    #05  656f3848  b6f359a5  /system/lib/libc.so
         656f384c  656f385c  [stack:1160]
         656f3850  8f38e000  
         656f3854  b6f118ff  /system/lib/libc.so (__stack_chk_fail+10)
         656f3858  b6f359a5  /system/lib/libc.so
         656f385c  2f94ff4a  /dev/ashmem/dalvik-main space (deleted)
         656f3860  8f38e000  
         656f3864  00000000  

And my gradle config is as follows:

Project build.gradle:

buildscript {
    repositories {
        google()
        jcenter()
        mavenCentral()
        maven { url "https://jitpack.io" }
    }
    dependencies {
        classpath "com.android.tools.build:gradle:3.6.3"
        classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:1.3.61"
        classpath "com.google.gms:google-services:4.3.3"  // Google Services plugin
        classpath "com.google.firebase:perf-plugin:1.3.1"  // Performance Monitoring plugin
        classpath "com.google.firebase:firebase-crashlytics-gradle:2.1.1"
    }
}

allprojects {
    repositories {
        google()
        jcenter()
        mavenCentral()
        maven {
            url 'https://jitpack.io'
        }
        flatDir {
            dirs 'libs'
        }
    }
}

task clean(type: Delete) {
    delete rootProject.buildDir
}

App module build.gradle:

repositories {
    maven { url 'http://dl.bintray.com/piasy/maven' }
    mavenCentral()
}

apply plugin: 'com.android.application'
apply plugin: 'kotlin-android'
apply plugin: 'kotlin-android-extensions'
apply plugin: 'com.google.firebase.crashlytics'
apply plugin: 'com.google.firebase.firebase-perf'
android {
    externalNativeBuild {
        ndkBuild {
            path 'src/main/jni/Android.mk'
        }
    }

    ....

    buildTypes {
        debug {
            ndk {
                abiFilters "armeabi-v7a", "arm64-v8a", "x86", "x86_64"
            }
            FirebasePerformance {
                instrumentationEnabled false
            }
        }
        release {
            minifyEnabled true
            useProguard = true
            proguardFile getDefaultProguardFile('proguard-android.txt')
            proguardFile 'proguard-rules.pro'

            ndk {
                abiFilters "armeabi-v7a", "arm64-v8a", "x86_64"
            }

            /* Add the firebaseCrashlytics extension (by default,
            * it's disabled to improve build speeds) and set
            * nativeSymbolUploadEnabled to true. */
            firebaseCrashlytics {
                nativeSymbolUploadEnabled true
            }
        }
    }

    compileOptions {
        sourceCompatibility JavaVersion.VERSION_1_8
        targetCompatibility JavaVersion.VERSION_1_8
    }
}

// Upload Crashlytics NDK symbols
tasks.whenTaskAdded { task -> if (task.name.startsWith('assemble') && task.name.endsWith("Release")) task.finalizedBy "uploadCrashlyticsSymbolFile" + task.name.substring('assemble'.length()) }

dependencies {
    implementation fileTree(include: ['*.jar', '*.aar'], dir: 'libs')

    implementation platform('com.google.firebase:firebase-bom:25.2.1')
    implementation "com.google.firebase:firebase-crashlytics-ndk:17.0.0"
    implementation 'com.google.firebase:firebase-iid'
    implementation 'com.google.firebase:firebase-messaging'
    implementation 'com.google.firebase:firebase-appindexing'
    implementation 'com.google.firebase:firebase-analytics'
    implementation 'com.google.firebase:firebase-perf'

    ...
}
apply plugin: 'com.google.gms.google-services'

There are lots and lots of crashes reported through the Firebase console like the one I captured. Not a single one of them has a proper usable backtrace.

Any hint is appreciated, the native reports are currently useless without the frames.

Edit:

I have addressed both stipped and unstripped native libs directory based on Kevin's answer as follows:

android {
    ...
    buildTypes {
        ...
        release {
            firebaseCrashlytics {
                nativeSymbolUploadEnabled true
                applicationVariants.all { variant ->
                    firebaseCrashlytics {
                        strippedNativeLibsDir "build/intermediates/stripped_native_libs/${variant.name}/out/lib"
                        unstrippedNativeLibsDir "build/intermediates/ndkBuild/${variant.name}/obj/local"
                    }
                }
            }
        }
    }
}

And managed to produce a crash that has a callstack that goes through my own native libraries, but the problem still persist:

Edit 2:

I used this command to debug the symbol files uploading issue as Kevin suggested (I used find /I instead of grep since I'm on Windows):

gradlew app:assembleRelease --console=plain --debug | find /I "[com.google.firebase.crashlytics]"

And turns out using implementation 'com.google.firebase:firebase-iid' along with implementation platform('com.google.firebase:firebase-bom:25.2.1') was what holding me from redemption :) since apparently Firebase was already using instance id dependency inside com.google.firebase:firebase-messaging itself. So long story short using only this config helped me to upload my symbols properly:

android {
    ...
    buildTypes {
        ...
        release {
            firebaseCrashlytics {
                nativeSymbolUploadEnabled true
            }
        }
    }
}

// Upload Crashlytics NDK symbols
tasks.whenTaskAdded { task -> if (task.name.startsWith('assemble') && task.name.endsWith("Release")) task.finalizedBy "uploadCrashlyticsSymbolFile" + task.name.substring('assemble'.length()) }

dependencies {
    ...
    implementation platform('com.google.firebase:firebase-bom:25.2.1')
    implementation 'com.google.firebase:firebase-iid:20.1.5'
    implementation 'com.google.firebase:firebase-messaging'
    ...
}

And at least get the callstack on my own native libraries:

Answer 1

Firebaser here-

It looks like this crash report you've pasted a screenshot for is occurring in system libraries only. We can see that in the Crashlytics crash report (and the tombstone) that all the stackframes are associated with "libc.so" - this is an Android system runtime library, and any libraries like those Crashlytics is unable to symbolicate.

This is because Crashlytics can only symbolicate crashes or stackframes that come from libraries that you've uploaded symbols for. And since libc.so is a system library for which both you and Crashlytics both don't have the symbols for, this is expected behavior. Crashlytics still shows the frames even though it can't symbolicate just so that it's still reporting as much info as possible.

If you're able to submit a test crash report that has a callstack that goes through your own native libraries for which you're attempting to upload symbols, and you're still seeing "(Missing)" everywhere, that would be unexpected. If that's the case, I recommend making sure you've followed these instructions in the doc you linked, and reaching out to Firebase support if you're still having issues.