How do I create a lot of sample data for firestore?

Question

Let's say I need to create a lot of different documents/collections in firestore. I need to add it quickly, like copy and paste json. I can't do that with standard firebase console, because adding 100 documents will take me forever. Is there any solutions for to bulk create mock data with a given structure in firestore db?

Answer 1

If you switch to the Cloud Console (rather than Firebase Console) for your project, you can use Cloud Shell as a starting point.

From the Cloud Shell environment you'll find tools like node and python installed and available. Using whatever one you prefer, you can write a script using the Server Client libraries.

For example in Python:

from google.cloud import firestore
import random

MAX_DOCUMENTS = 100
SAMPLE_COLLECTION_ID = u'users'
SAMPLE_COLORS = [u'Blue', u'Red', u'Green', u'Yellow', u'White', u'Black']

# Project ID is determined by the GCLOUD_PROJECT environment variable
db = firestore.Client()

collection_ref = db.collection(SAMPLE_COLLECTION_ID)

for x in range(0, MAX_DOCUMENTS - 1):
collection_ref.add({
    u'primary': random.choice(SAMPLE_COLORS),
    u'secondary': random.choice(SAMPLE_COLORS),
    u'trim': random.choice(SAMPLE_COLORS),
    u'accent': random.choice(SAMPLE_COLORS)
})

While this is the easiest way to get up and running with a static dataset, it lives a little to be desired. Namely with Firestore, live dynamic data is needed to exercises it's functionally, such as real-time queries. For this task, using Cloud Scheduler & Cloud Functions is a relatively easy way to regularly updating sample data.

In addition to the sample generation code, you'll specific the update frequency in the Cloud Scheduler. For instance in the image below, */10 * * * * defines a frequency of every 10 minutes using the standard unix-cron format:

For non-static data, often a timestamp is useful. Firestore provides a way to have a timestamp from the database server added at write-time as one of the fields:

u'timestamp': firestore.SERVER_TIMESTAMP

It is worth noting that timestamps like this will hotspot in production systems if not sharded correctly. Typically 500 writes/second to the same collection is the maximum you will want so that the index doesn't hotspot. Sharding can be as simple something like as each user having their own collection (500 writes/second per user). However for this example, writing 100 documents every minute via a scheduled Cloud Function is definitely not an issue.