SharedPreferences in Android not persisted to disk when key contains newline

Question

In Android, I'd like to write SharedPreferences key-value pairs where the keys are Base64 strings.

// get a SharedPreferences instance
SharedPreferences prefs = getSharedPreferences("some-name", Context.MODE_PRIVATE);
// generate the base64 key
String someKey = new String(Base64.encode("some-key".getBytes("UTF-8"), Base64.URL_SAFE), "UTF-8");
// write the value for the generated key
prefs.edit().putBoolean(someKey, true).commit();

In the last line, the call to commit returns true. So this key-value pair should have been saved successfully.

When I close and destroy the Activity where this piece of code was used and then re-create the Activity (running this code again), the specified value is returned for the key that we used.

But it turns out that, when I destroy the whole application/process (e.g. using "Force stop" in app settings), the value for our key is lost on the next launch of the Activity.

When I don't use Base64.URL_SAFE but Base64.URL_SAFE | Base64.NO_WRAP as the flags for the Base64 encoding, it works fine.

So this problem has been caused by the newlines at the end of the Base64 keys. Keys like abc can be written without any problems. But when the key is abc\n, it fails.

The problem is that it appears to work without problems first, returning true on commit() and returning the correct preference value on subsequent calls. But when the whole application is destroyed and re-started, the value has not been persisted.

Is this intended behaviour? A bug? Does the documentation say anything about valid key names?

Answer 1

I took a look at GrepCode and see that the operations will be the following (I do not mention useless ones) :

1. android.app.SharedPreferencesImpl.commit()
2. android.app.SharedPreferencesImpl.commitToMemory()

3. android.app.SharedPreferencesImpl.queueDiskWrite(MemoryCommitResult,Runnable)

   3.1. XmlUtils.writeMapXml(Map, OutputStream)

   3.2. XmlUtils.writeMapXml(Map, String, XmlSerializer)

   3.3. XmlUtils.writeValueXml(Object v, String name, XmlSerializer ser)

---

First : how your data are converted ?

The method XmlUtils.writeValueXml writes the Object value in a XML tag with the attribute name set to the String value. This String value contains exactly the value you specified at the SharedPreference's name.

(And I confirmed this by doing a step-by-step debug with your piece of code).

The XML will be with an unescaped line break character. Actually, the XmlSerializer instance is a FastXmlSerializer instance and it does not escape the \n character (see the link for this class at the end if you want to read the source code)

Interesting piece of code :

writeValueXml(Object v, String name, XmlSerializer out) {
    // -- "useless" code skipped
    out.startTag(null, typeStr);
    if (name != null) {
        out.attribute(null, "name", name);
    }
    out.attribute(null, "value", v.toString());
    out.endTag(null, typeStr);
    // -- "useless" code skipped
}

Second : why the result is true ?

The commit method has the following code :

public boolean commit() {
    MemoryCommitResult mcr = commitToMemory();
    SharedPreferencesImpl.this.enqueueDiskWrite(
        mcr, null /* sync write on this thread okay */);
    try {
        mcr.writtenToDiskLatch.await();
    } catch (InterruptedException e) {
        return false;
    }
    notifyListeners(mcr);
    return mcr.writeToDiskResult;
}

So it returns the mcr.writeToDiskResult which is set in the SharedPreferencesImpl.writeToFile(MemoryCommitResult) method. Interesting piece of code :

writeToFile(MemoryCommitResult mcr) {
    // -- "useless" code skipped
    try {
        FileOutputStream str = createFileOutputStream(mFile);
        if (str == null) {
            mcr.setDiskWriteResult(false);
            return;
        }
        XmlUtils.writeMapXml(mcr.mapToWriteToDisk, str);
        FileUtils.sync(str);
        str.close();
        ContextImpl.setFilePermissionsFromMode(mFile.getPath(), mMode, 0);
        try {
            final StructStat stat = Libcore.os.stat(mFile.getPath());
            synchronized (this) {
                mStatTimestamp = stat.st_mtime;
                mStatSize = stat.st_size;
            }
        } catch (ErrnoException e) {
            // Do nothing
        }
        // Writing was successful, delete the backup file if there is one.
        mBackupFile.delete();
        mcr.setDiskWriteResult(true);
        return;
    } catch (XmlPullParserException e) {
        Log.w(TAG, "writeToFile: Got exception:", e);
    } catch (IOException e) {
        Log.w(TAG, "writeToFile: Got exception:", e);
    }
    // -- "useless" code skipped
}

As we see at the previous point : the XML writing is "ok" (do not throw anything, do not fails), so the sync in the file will be too (just a copy of a Stream in another one, nothing checks the XML content here !).

Currently : your key was converted to (badly formatted) XML and correctly wrote in a File. The result of the whole operation is true as everything went OK. Your changes are comitted to the disk and in the memory.

Third and last : why do I get back the correct value the first time and a bad one the second time

Take a quick look to what happen when we commit the changes to memory in SharedPreferences.Editor.commitToMemory(...) method (interesting part only... :)):

for (Map.Entry<String, Object> e : mModified.entrySet()) {
    String k = e.getKey();
    Object v = e.getValue();
    if (v == this) {  // magic value for a removal mutation
        if (!mMap.containsKey(k)) {
            continue;
        }
        mMap.remove(k);
    } else {
        boolean isSame = false;
        if (mMap.containsKey(k)) {
            Object existingValue = mMap.get(k);
            if (existingValue != null && existingValue.equals(v)) {
                continue;
            }
        }
        mMap.put(k, v);
    }

    mcr.changesMade = true;
    if (hasListeners) {
        mcr.keysModified.add(k);
    }
}

Important point : the changes are commited to the mMap attribute.

Then, take a quick look to how we get back a value :

public boolean getBoolean(String key, boolean defValue) {
    synchronized (this) {
        awaitLoadedLocked();
        Boolean v = (Boolean)mMap.get(key);
        return v != null ? v : defValue;
    }
}

We are taking back the key from mMap (no reading of the value in the file for now). So we have the correct value for this time :)

When you reload your application, you will load the data back from the disk, and so the SharedPreferencesImpl constructor will be called, and it will call the SharedPreferencesImpl.loadFromDiskLocked() method. This method will read the file content and load it in the mMap attribute (I let you see the code by yourself, link provided at the end).

A step-by-step debug shown me that the abc\n was written as abc (with a whitespace character). So, when you will try to get it back, you will never succeed.

---

To finish, thank you to @CommonsWare to give me a hint about the file content in the comment :)

Links

XmlUtils

FastXmlSerializer

SharedPreferencesImpl

SharedPreferencesImpl.EditorImpl.commit()

SharedPreferencesImpl.EditorImpl.commitToMemory()

SharedPreferencesImpl.enqueueDiskWrite(MemoryCommitResult, Runnable)

SharedPreferencesImpl.writeToFile(MemoryCommitResult)

SharedPreferencesImpl.loadFromDiskLocked()