How to import data from cloud firestore to the local emulator?

Question

I want to be able to run cloud functions locally and debug against a copy from the production data. Is there a way to copy the data that is online to the local firestore emulator?

Answer 1

This can be accomplished through a set of commands in terminal on the existing project:

1. Login to firebase and Gcloud:

firebase login gcloud auth login 

2. See a list of your projects and connect to one:

firebase projects:list firebase use your-project-name  gcloud projects list gcloud config set project your-project-name 

3. Export your production data to gcloud bucket with chosen name:

gcloud firestore export gs://your-project-name.appspot.com/your-choosen-folder-name 

4. Now copy this folder to your local machine, I do that in functions folder directly:

cd functions gsutil -m cp -r gs://your-project-name.appspot.com/your-choosen-folder-name . 

5. Now we just want to import this folder. This should work with the basic command, thanks to latest update from Firebase team https://github.com/firebase/firebase-tools/pull/2519.

firebase emulators:start --import ./your-choosen-folder-name 

Check out my article on Medium about it and a shorthanded script to do the job for you https://medium.com/firebase-developers/how-to-import-production-data-from-cloud-firestore-to-the-local-emulator-e82ae1c6ed8

Note: Its better to use a different bucket for it, as copying into your project bucket will result in the folder created in your firebase storage.
If you are interested in gsutil arguments like -m, you can see them described by executing gsutil --help.

Answer 2

My method is somewhat manual but it does the trick. I've shared it in this useful Github thread but I'll list the steps I did here if you find them useful:

1. Go to my local Firebase project path.
2. Start the emulators using: firebase emulators:start
3. Create manually some mockup data using the GUI at http://localhost:4000/firestore using the buttons provided: + Start Collection and + Add Document.
4. Export this data locally using: emulators:export ./mydirectory
5. About the project data located at Firebase Database / Cloud Firestore, I exported a single collection like this: gcloud firestore export gs://my-project-bucket-id.appspot.com --collection-ids=myCollection The export is now located under Firebase Storage in a folder with a timestamp as name (I didn't use a prefix for my test)
6. Download this folder to local drive with: gsutil cp -r gs://my-project-bucket-id.appspot.com/myCollection ./production_data_export NOTE: I did this in a Windows environment... gsutil will throw this error: "OSError: The filename, directory name, or volume label syntax is incorrect" if the folder has invalid characters for a folder name in Windows (i.e. colons) or this error: "OSError: Invalid argument.9.0 B]" if an inner file in the folder has invalid characters too. To be able to download the export locally, rename these with a valid Windows name (i.e. removing the colons) like this: gsutil mv gs://my-project-bucket-id.appspot.com/2020-05-22T02:01:06_86152 gs://my-project-bucket-id.appspot.com/myCollection
7. Once downloaded, imitate the local export structure renaming the folder to firestore_export and copying the firebase-export-metadata.json file from the local export folder. Just to be visual, here's the structure I got:

$ tree . . ├── local_data_export │   ├── firebase-export-metadata.json │   └── firestore_export │       ├── all_namespaces │       │   └── all_kinds │       │       ├── all_namespaces_all_kinds.export_metadata │       │       └── output-0 │       └── firestore_export.overall_export_metadata └── production_data_export     ├── firebase-export-metadata.json     └── firestore_export         ├── all_namespaces         │   └── kind_myCollection         │       ├── all_namespaces_kind_myCollection.export_metadata         │       ├── output-0         │       └── output-1         └── firestore_export.overall_export_metadata  8 directories, 9 files 

8. Finally, start the local emulator pointing to this production data to be imported: firebase emulators:start --import=./mock_up_data/production_data_export/
9. You should see the imported data at: http://localhost:4000/firestore/

This should assist readers for now, while we await a more robust solution from the Firebase folks.