Add new field or change the structure on all Firestore documents

Question

Consider a collection of users. Each document in the collection has name and email as fields.

{
  "users": {
    "uid1": {
      "name": "Alex Saveau",
      "email": "[email protected]"
    },
    "uid2": { ... },
    "uid3": { ... }
  }
}

Consider now that with this working Cloud Firestore database structure I launch my first version of a mobile application. Then, at some point I realize I want to include another field such as last_login.

In the code, reading all the users documents from the Firestore DB using Java would be done as

FirebaseFirestore.getInstance().collection("users").get()
        .addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
            @Override
            public void onComplete(@NonNull Task<QuerySnapshot> task) {
                if (task.isSuccessful()) {
                    for (DocumentSnapshot document : task.getResult()) {
                        mUsers.add(document.toObject(User.class));
                    }
                }
            }
        });

where the class User contains now name, email and last_login.

Since the new User field (last_login) is not included in the old users stored in the DB, the application is crashing because the new User class is expecting a last_login field which is returned as null by the get() method.

What would be the best practice to include last_login in all the existing User documents of the DB without losing their data on a new version of the app? Should I run an snippet just once to do this task or are there any better approaches to the problem?

Answer 1

You fell into a gap of NOSQL databases: Document oriented databases do not guarantee structural integrity of the data (as RDBMS do)

The deal is:

• in an RDBMS all stored data have the same structure at any given time (within the same instance or cluster). When changing the structure (ER-diagram) you have to migrate the data for all existing records which costs time and effort.

  As a result, your application can be optimized for the current version of the data structure.

• in a Document oriented database each record is an independent "Page" with its own independent structure. If you change the structure it only applies to new documents. So you don't need to migrate the existing data.

  As a result, your application must be able to deal with all versions of the data structure you've ever used in your current database.

I don't know about firebase in detail but in general you never update a document in a NOSQL database. You only create a new version of the document. So even if you update all documents your application must be prepared to deal with the "old" data structure...

Answer 2

I wrote some routines to help automate this process back when I posted the question. I did not post them since these are a bit rudimentary and I was hoping for an elegant Firestore-based solution. Because such solution is not still available, here are the functions I wrote.

In short, we have functions for renaming a field, adding a field, or deleting a field. To rename a field, different functions are used depending on the data type. Maybe someone could generalise this better? The functions below are:

• add_field: Adds a field in all documents of a collection.
• delete_field: Deletes a field in all documents of a collection.
• rename_*_field: Renames a field containing a certain data type (*) in all documents of a collection. Here I include examples for String, Integer, and Date.

Add field:

public void add_field (final String key, final Object value, final String collection_ref) {
    FirebaseFirestore.getInstance().collection(collection_ref).get()
            .addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
                @Override
                public void onComplete(@NonNull Task<QuerySnapshot> task) {
                    if (task.isSuccessful()) {
                        WriteBatch batch = db.batch();

                        for (DocumentSnapshot document : task.getResult()) {
                            DocumentReference docRef = document.getReference();
                            Map<String, Object> new_map = new HashMap<>();
                            new_map.put(key, value);
                            batch.update(docRef, new_map);
                        }
                        batch.commit();
                    } else {
                        // ... "Error adding field -> " + task.getException()
                    }
                }
            })
            .addOnFailureListener(new OnFailureListener() {
                @Override
                public void onFailure(@NonNull Exception e) {
                    // ... "Failure getting documents -> " + e
                }
            });
}

Delete field:

public void delete_field (final String key, final String collection_ref) {
    FirebaseFirestore.getInstance().collection(collection_ref).get()
            .addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
                @Override
                public void onComplete(@NonNull Task<QuerySnapshot> task) {
                    if (task.isSuccessful()) {

                        WriteBatch batch = db.batch();

                        for (DocumentSnapshot document : task.getResult()) {
                            DocumentReference docRef = document.getReference();
                            Map<String, Object> delete_field = new HashMap<>();
                            delete_field.put(key, FieldValue.delete());
                            batch.update(docRef, delete_field);
                        }
                        // Commit the batch
                        batch.commit().addOnCompleteListener(new OnCompleteListener<Void>() {
                            @Override
                            public void onComplete(@NonNull Task<Void> task) {
                                // ...
                            }
                        });

                    } else {
                        // ... "Error updating field -> " + task.getException()
                    }
                }
            })
            .addOnFailureListener(new OnFailureListener() {
                @Override
                public void onFailure(@NonNull Exception e) {
                    // ... "Failure getting notices -> " + e
                }
            });
}

Rename field:

public void rename_string_field (final String old_key, final String new_key, final String collection_ref) {
    FirebaseFirestore.getInstance().collection(collection_ref).get()
            .addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
                @Override
                public void onComplete(@NonNull Task<QuerySnapshot> task) {
                    if (task.isSuccessful()) {

                        WriteBatch batch = db.batch();

                        for (DocumentSnapshot document : task.getResult()) {
                            DocumentReference docRef = document.getReference();
                            String old_value = document.getString(old_key);

                            if (old_value != null) {
                                Map<String, Object> new_map = new HashMap<>();
                                new_map.put(new_key, old_value);

                                Map<String, Object> delete_old = new HashMap<>();
                                delete_old.put(old_key, FieldValue.delete());

                                batch.update(docRef, new_map);
                                batch.update(docRef, delete_old);
                            }
                        }
                        // Commit the batch
                        batch.commit().addOnCompleteListener(new OnCompleteListener<Void>() {
                            @Override
                            public void onComplete(@NonNull Task<Void> task) {
                                // ...
                            }
                        });

                    } else {
                        // ... "Error updating field -> " + task.getException()
                    }
                }
            })
            .addOnFailureListener(new OnFailureListener() {
                @Override
                public void onFailure(@NonNull Exception e) {
                    // ... "Failure getting notices ->" + e
                }
            });
}

public void rename_integer_field (final String old_key, final String new_key, final String collection_ref) {
    FirebaseFirestore.getInstance().collection(collection_ref).get()
            .addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
                @Override
                public void onComplete(@NonNull Task<QuerySnapshot> task) {
                    if (task.isSuccessful()) {

                        WriteBatch batch = db.batch();

                        for (DocumentSnapshot document : task.getResult()) {
                            DocumentReference docRef = document.getReference();
                            int old_value = document.getDouble(old_key).intValue();
                            Integer ov = old_value;
                            if (ov != null) {
                                Map<String, Object> new_map = new HashMap<>();
                                new_map.put(new_key, old_value);

                                Map<String, Object> delete_old = new HashMap<>();
                                delete_old.put(old_key, FieldValue.delete());

                                batch.update(docRef, new_map);
                                batch.update(docRef, delete_old);
                            }
                        }
                        // Commit the batch
                        batch.commit().addOnCompleteListener(new OnCompleteListener<Void>() {
                            @Override
                            public void onComplete(@NonNull Task<Void> task) {
                                // ...
                            }
                        });

                    } else {
                        // ... "Error updating field -> " + task.getException()
                    }
                }
            })
            .addOnFailureListener(new OnFailureListener() {
                @Override
                public void onFailure(@NonNull Exception e) {
                    // ... "Failure getting notices -> " + e
                }
            });
}

public void rename_date_field (final String old_key, final String new_key, final String collection_ref) {
    FirebaseFirestore.getInstance().collection(collection_ref).get()
            .addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
                @Override
                public void onComplete(@NonNull Task<QuerySnapshot> task) {
                    if (task.isSuccessful()) {

                        WriteBatch batch = db.batch();

                        for (DocumentSnapshot document : task.getResult()) {
                            DocumentReference docRef = document.getReference();
                            Date old_value = document.getDate(old_key);
                            if (old_value != null) {
                                Map<String, Object> new_map = new HashMap<>();
                                new_map.put(new_key, old_value);

                                Map<String, Object> delete_old = new HashMap<>();
                                delete_old.put(old_key, FieldValue.delete());

                                batch.update(docRef, new_map);
                                batch.update(docRef, delete_old);
                            }
                        }
                        // Commit the batch
                        batch.commit().addOnCompleteListener(new OnCompleteListener<Void>() {
                            @Override
                            public void onComplete(@NonNull Task<Void> task) {
                                // ...
                            }
                        });

                    } else {
                        // ... "Error updating field -> " + task.getException()
                    }
                }
            })
            .addOnFailureListener(new OnFailureListener() {
                @Override
                public void onFailure(@NonNull Exception e) {
                    // ... "Failure getting notices -> " + e
                }
            });
}