How to import test data into Firebase Firestore emulator from production for easy testing?

Question

I am building a firebase function that listens to a trigger and sends a push notification to users. The trigger is based on firestore data that's too complex for me to manually re-recreate every time on firestore emulator.

I tried emulating functions, but hooking up to firestore production, but seems like in this case, trigger functions don't work 😢

What I want to do is export my data from my production Firestore then import it into my emulator firestore, so that at least, the copy I am working with is closely mimicking what I have on prod; to be clear, the data is small, so I am not worried about downloading terabytes of data.

I found a way of importing data into an emulator, but not sure if this would work with production data or how I would dump the data from production.

firebase emulators:start --import=./some-directory

Answer 1

You can manually export from Production and then import into the Firestore emulator.

Start by doing an export of your production data to a file. In the example, I export just the "users" collection within an onRequest function so I can kick-off by calling the URL. Note, this this function will not handle subcollections.

exports.exportFirestore = functions.https.onRequest(async (req, res) => {
  const fs = require("fs");

  const collection = "users";
  const fileName = "fs-export.json";
  const exportedData = {};
  exportedData[collection] = {};

  await admin
    .firestore()
    .collection(collection)
    .get()
    .then((snapshot) => {
      return snapshot.forEach((doc) => {
        exportedData[collection][doc.id] = doc.data();
      });
    })
    .catch(console.error);

  fs.writeFile(fileName, JSON.stringify(exportedData), (err) => {
    if (err) {
      console.log(err);
    } else {
      console.log("Firestore Export Complete.");
      res.send("Firestore Export Complete.");
    }
  });
});

Next, import the file into the local Firestore emulator. Very important: be sure to have the Firestore emulator running otherwise you'll be importing your data back to prod. Be sure to check that Firestore is running by checking the web UI: http://localhost:4000. Start the emulator with: firebase emulators:start

exports.importFirestore = functions.https.onRequest(async (req, res) => {
  const fs = require("fs");

  let collection;
  const fileName = "fs-export.json";
  const exportedData = {};
  exportedData[collection] = {};

  fs.readFile(fileName, "utf8", async (err, data) => {
    if (err) {
      return console.log(err);
    }

    const arr = JSON.parse(data);

    const batch = admin.firestore().batch();

    for (let i in arr) {
      collection = i;
      for (let doc in arr[i]) {
        if (arr[i].hasOwnProperty(doc)) {
          const ref = admin.firestore().collection(collection).doc(doc);
          batch.set(ref, arr[i][doc]);
        } else {
          console.log("Missing:", JSON.stringify(doc, null, 2));
        }
      }
    }

    await batch
      .commit()
      .then(() => {
        return console.log("Import to Firestore Complete");
      })
      .catch(console.error);

    return res.send("Import to Firestore Complete");
  });
});