Getting KeyStoreException and GeneralSecurityException by using EncryptedSharedPreferences, how can I solve those?

Question

Background

On one of the apps I work on, I store important stuff (tokens) into EncryptedSharedPreferences (taken from here and here):

/** a hardware-encrypted based shared preference (for the values).
 * Note that it is a bit slow, so it's better to always use it in a background thread.
 * Also, avoid having it being backed-up in the manifest, as it's hardware based and will become useless: https://stackoverflow.com/a/63795282/878126*/
object SecuredSharedPreferences {
    private var cachedDefaultSharedPreferences: SharedPreferences? = null

    /**warning: using this function can take some time (249 ms on Pixel 4, for example). Very recommended to avoid calling it on UI thread */
    @WorkerThread
    fun getDefaultSecuredSharedPreferences(context: Context): SharedPreferences {
        if (cachedDefaultSharedPreferences != null)
            return cachedDefaultSharedPreferences!!
        synchronized(this) {
            if (cachedDefaultSharedPreferences != null)
                return cachedDefaultSharedPreferences!!
            cachedDefaultSharedPreferences = getSecuredSharedPreferences(context, context.packageName + "_secured_preferences")
        }
        return cachedDefaultSharedPreferences!!
    }

    @WorkerThread
    private fun getSecuredSharedPreferences(context: Context, fileName: String): SharedPreferences {
        val masterKey = MasterKey.Builder(context, MasterKey.DEFAULT_MASTER_KEY_ALIAS).setKeyScheme(MasterKey.KeyScheme.AES256_GCM).build()
        return EncryptedSharedPreferences.create(context, fileName, masterKey,
                EncryptedSharedPreferences.PrefKeyEncryptionScheme.AES256_SIV,
                EncryptedSharedPreferences.PrefValueEncryptionScheme.AES256_GCM
        )
    }
}

gradle:

implementation 'androidx.security:security-crypto:1.1.0-alpha03'

The problem

I've noticed 2 bugs being reported via Crashlytics when using this code (reported here):

1. First one is of the MasterKey.Builder line of GeneralSecurityException :

Fatal Exception: java.security.GeneralSecurityException: Keystore operation failed
       at androidx.security.crypto.MasterKeys.generateKey(MasterKeys.java:146)
       at androidx.security.crypto.MasterKeys.getOrCreate(MasterKeys.java:97)
       at androidx.security.crypto.MasterKey$Builder.buildOnM(MasterKey.java:357)
       at androidx.security.crypto.MasterKey$Builder.build(MasterKey.java:314)
...
Caused by java.security.ProviderException: Keystore operation failed
       at android.security.keystore.AndroidKeyStoreKeyGeneratorSpi.engineGenerateKey(AndroidKeyStoreKeyGeneratorSpi.java:372)
       at javax.crypto.KeyGenerator.generateKey(KeyGenerator.java:612)
       at androidx.security.crypto.MasterKeys.generateKey(MasterKeys.java:142)
       at androidx.security.crypto.MasterKeys.getOrCreate(MasterKeys.java:97)
       at androidx.security.crypto.MasterKey$Builder.buildOnM(MasterKey.java:357)
       at androidx.security.crypto.MasterKey$Builder.build(MasterKey.java:314)

2. Second one is on EncryptedSharedPreferences.create line of KeyStoreException, and occurs more often and for more users :

Fatal Exception: java.security.KeyStoreException: the master key android-keystore://_androidx_security_master_key_ exists but is unusable
       at com.google.crypto.tink.integration.android.AndroidKeysetManager$Builder.readOrGenerateNewMasterKey(AndroidKeysetManager.java:275)
       at com.google.crypto.tink.integration.android.AndroidKeysetManager$Builder.build(AndroidKeysetManager.java:236)
       at androidx.security.crypto.EncryptedSharedPreferences.create(EncryptedSharedPreferences.java:155)
       at androidx.security.crypto.EncryptedSharedPreferences.create(EncryptedSharedPreferences.java:120)
...
Caused by java.security.UnrecoverableKeyException: Failed to obtain information about key
       at android.security.keystore.AndroidKeyStoreProvider.loadAndroidKeyStoreSecretKeyFromKeystore(AndroidKeyStoreProvider.java:282)
       at android.security.keystore.AndroidKeyStoreSpi.engineGetKey(AndroidKeyStoreSpi.java:98)
       at java.security.KeyStore.getKey(KeyStore.java:825)
       at com.google.crypto.tink.integration.android.AndroidKeystoreAesGcm.<init>(AndroidKeystoreAesGcm.java:58)
       at com.google.crypto.tink.integration.android.AndroidKeystoreKmsClient.getAead(AndroidKeystoreKmsClient.java:164)
       at com.google.crypto.tink.integration.android.AndroidKeysetManager$Builder.readOrGenerateNewMasterKey(AndroidKeysetManager.java:267)
       at com.google.crypto.tink.integration.android.AndroidKeysetManager$Builder.build(AndroidKeysetManager.java:236)
       at androidx.security.crypto.EncryptedSharedPreferences.create(EncryptedSharedPreferences.java:155)
       at androidx.security.crypto.EncryptedSharedPreferences.create(EncryptedSharedPreferences.java:120)

EDIT: seems there are even more types of exceptions:

3. InvalidProtocolBufferException:

Fatal Exception: com.google.crypto.tink.shaded.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
       at com.google.crypto.tink.shaded.protobuf.GeneratedMessageLite.parsePartialFrom(GeneratedMessageLite.java:1566)
       at com.google.crypto.tink.shaded.protobuf.GeneratedMessageLite.parseFrom(GeneratedMessageLite.java:1663)
       at com.google.crypto.tink.proto.Keyset.parseFrom(Keyset.java:957)
       at com.google.crypto.tink.integration.android.SharedPrefKeysetReader.read(SharedPrefKeysetReader.java:84)
       at com.google.crypto.tink.CleartextKeysetHandle.read(CleartextKeysetHandle.java:58)
       at com.google.crypto.tink.integration.android.AndroidKeysetManager$Builder.read(AndroidKeysetManager.java:328)
       at com.google.crypto.tink.integration.android.AndroidKeysetManager$Builder.readOrGenerateNewKeyset(AndroidKeysetManager.java:287)
       at com.google.crypto.tink.integration.android.AndroidKeysetManager$Builder.build(AndroidKeysetManager.java:238)
       at androidx.security.crypto.EncryptedSharedPreferences.create(EncryptedSharedPreferences.java:160)
       at androidx.security.crypto.EncryptedSharedPreferences.create(EncryptedSharedPreferences.java:120)

4. NullPointerException:

Fatal Exception: java.lang.NullPointerException: Attempt to invoke interface method 'android.security.keymaster.OperationResult android.security.IKeystoreService.begin(android.os.IBinder, java.lang.String, int, boolean, android.security.keymaster.KeymasterArguments, byte[], int)' on a null object reference
       at android.security.KeyStore.begin(KeyStore.java:501)
       at android.security.keystore.AndroidKeyStoreCipherSpiBase.ensureKeystoreOperationInitialized(AndroidKeyStoreCipherSpiBase.java:248)
       at android.security.keystore.AndroidKeyStoreCipherSpiBase.engineInit(AndroidKeyStoreCipherSpiBase.java:109)
       at javax.crypto.Cipher.tryTransformWithProvider(Cipher.java:2977)
       at javax.crypto.Cipher.tryCombinations(Cipher.java:2884)
       at javax.crypto.Cipher$SpiAndProviderUpdater.updateAndGetSpiAndProvider(Cipher.java:2789)
       at javax.crypto.Cipher.chooseProvider(Cipher.java:956)
       at javax.crypto.Cipher.init(Cipher.java:1199)
       at javax.crypto.Cipher.init(Cipher.java:1143)
       at com.google.crypto.tink.integration.android.AndroidKeystoreAesGcm.encryptInternal(AndroidKeystoreAesGcm.java:84)
       at com.google.crypto.tink.integration.android.AndroidKeystoreAesGcm.encrypt(AndroidKeystoreAesGcm.java:72)
       at com.google.crypto.tink.integration.android.AndroidKeystoreKmsClient.validateAead(AndroidKeystoreKmsClient.java:248)
       at com.google.crypto.tink.integration.android.AndroidKeystoreKmsClient.getAead(AndroidKeystoreKmsClient.java:165)
       at com.google.crypto.tink.integration.android.AndroidKeysetManager$Builder.readOrGenerateNewMasterKey(AndroidKeysetManager.java:267)
       at com.google.crypto.tink.integration.android.AndroidKeysetManager$Builder.build(AndroidKeysetManager.java:236)
       at androidx.security.crypto.EncryptedSharedPreferences.create(EncryptedSharedPreferences.java:155)
       at androidx.security.crypto.EncryptedSharedPreferences.create(EncryptedSharedPreferences.java:120)

What I've tried

Searching the Internet, I've found clues only for the first exception (GeneralSecurityException), that it might be that it occurs for custom ROMs, as they might not implement well the hardware keys for encryption.

And indeed, looking at the devices on Crashlytics, and looking at the Android version of each, I've found that they are ahead of what I see about the latest version that was supported for them.

For the second exception, sadly, I couldn't find any explanation and no solution either. I think it might be related to recovery of apps, but it's weird as it occurs quite often. On reddit (here), someone wrote that in case of such an exception, he chose to wrap the initialization of EncryptedSharedPreferences with "clear all data if fails" and bite the bullet. Also suggested it might be related to having android:allowBackup being disabled (and indeed it is).

No idea about the rest.

The question

Why do these exceptions occur? What can I do against them?

Is clear-data the only thing that can be done? I'm not even sure it really helps, because if I choose to have it, it means the crash report will be gone each time it's about to happen...

Is it related to android:allowBackup being disabled?

Answer 1

I think your instinct about android:allowBackup=false being the culprit is correct. I experienced a similar issue (encrypted shared preferences throwing errors on app update when allowBackup was set to false). In my case I managed to reproduce by installing, logging in (to set the encrypted preference), and then updating to a more recent version of the app and attempting to read the preference. I ended up working around the issue by removing android:allowBackup=false

Answer 2

Beware about this active glitch prior to using Jetpack Security EncryptedSharedPreferences ( even stable ), It's hitting on our crash-free sessions hard mostly from exotic devices - https://issuetracker.google.com/issues/176215143?pli=1

The only dirty workaround is - https://github.com/google/tink/issues/535#issuecomment-912661574

I'll update the answer once I find an amicable solution. Here is the workaround solution -

/**
 * A builder for creating an encrypted shared preference class.
 */
private const val KEYSTORE_PROVIDER = "AndroidKeyStore"
private const val SHARED_PREFS_FILENAME = "TVPrefs"

@KoinApiExtension
class EncryptedSharedPreferenceBuilder(var context: Context) : KoinComponent {
    private val reporter: Reporter by inject()

    private val masterKeyAlias =
        MasterKey.Builder(context).setKeyScheme(MasterKey.KeyScheme.AES256_GCM).build()

    fun build(): SharedPreferences {
        return try {
            createSharedPreferences()
        } catch (gsException: GeneralSecurityException) {
            reporter.logException(gsException)
            Timber.d("EncryptedSharedPref: Error occurred while create shared pref=$gsException")
            // There's not much point in keeping data you can't decrypt anymore,
            // delete & re-create; user has to start from scratch
            deleteSharedPreferences()
            createSharedPreferences()
        }
    }

    private fun createSharedPreferences() = EncryptedSharedPreferences.create(
        context,
        SHARED_PREFS_FILENAME,
        masterKeyAlias,
        EncryptedSharedPreferences.PrefKeyEncryptionScheme.AES256_SIV,
        EncryptedSharedPreferences.PrefValueEncryptionScheme.AES256_GCM
    )

    // Clearing getSharedPreferences using default Preference wrapper.
    // This is to work around any key-mismatches that may happen.
    fun clearSharedPreference() {
        context.getSharedPreferences(SHARED_PREFS_FILENAME, Context.MODE_PRIVATE).edit().clear()
            .apply()
    }

    // Workaround [https://github.com/google/tink/issues/535#issuecomment-912170221]
    // Issue Tracker - https://issuetracker.google.com/issues/176215143?pli=1
    private fun deleteSharedPreferences() {
        try {
            val sharedPrefsFile =
                File("${context.filesDir.parent}/shared_prefs/$SHARED_PREFS_FILENAME.xml")

            // Clear the encrypted prefs
            clearSharedPreference()

            // Delete the encrypted prefs file
            if (sharedPrefsFile.exists()) {
                val deleted = sharedPrefsFile.delete()
                Timber.d("EncryptedSharedPref: Shared pref file deleted=$deleted; path=${sharedPrefsFile.absolutePath}")
            } else {
                Timber.d("EncryptedSharedPref: Shared pref file non-existent; path=${sharedPrefsFile.absolutePath}")
            }

            // Delete the master key
            val keyStore = KeyStore.getInstance(KEYSTORE_PROVIDER)
            keyStore.load(null)
            keyStore.deleteEntry(MasterKey.DEFAULT_MASTER_KEY_ALIAS)
        } catch (e: Exception) {
            Timber.d("EncryptedSharedPref:  Error occurred while trying to reset shared pref=$e")
        }
    }
}