importing data from firestore into bigQuery

Question

I have upgraded my account to blaze which is one of the prerequisite. I have tried to followed the FAQ Link BigQuery to Firebase. still not able to see any of the data from firestore or firebase-realtimeDatabase into bigQuery.

I see option in bigQuery to create Dataset. however after creating dataset it allows me to upload data from [file], [cloud storage], [bigQuery],[google Drive] . but not firestore database.

Plesae help.

Answer 1

Firestore now allows to import / export data. The data is exported to a Cloud Storage bucket and from there it can be imported into Big Query. Here are some links that might be helpful:

• https://firebase.google.com/docs/firestore/manage-data/export-import
• https://cloud.google.com/bigquery/docs/loading-data-cloud-datastore (Firestore uses the same format as Datastore for imports / exports)

** Edit: Docs for BigQuery imports from Firestore are now available https://cloud.google.com/bigquery/docs/loading-data-cloud-firestore

Answer 2

In case anyone need it. I ended up automating scripts because current export data option only allows stream data and preserve for 30 days.

export data

gcloud beta firestore export --collection-ids=users gs://mybucket/users

load backup into bq

bq load --source_format=DATASTORE_BACKUP mydataset.users gs://gs://mybucket/users/all_namespaces/kind_users/all_namespaces_kind_users.export_metadata