Firebase: Reduce number of reads for a large collection

Question

After a lot of searching, I've found a lot of "what not to do" when it comes to reading firebase collections (main example being the incident of the $30k bill), however I'm struggling to find a suitable solution for our problem.

Our website contains a product catalogue. The catalogue is populated from a firestore collection called 'products', with the JSON structure as:

"products"{
   "productId"{
      "name":
      "price":
      "brand":
       ...
  }
}

In order to load the collection, I use the following:

this.products = this.db.collection('products').valueChanges({idField: 'productID'});

At its peak, we expect the collection to contain anywhere up to 2000 products. My concern is that as the website scales, this is going to become extremely costly. I've noticed that every time the 'catalogue' page is navigated to, it loads the collection again. Therefore, taking a somewhat not unimaginable example, if the page was loaded 5000 times in a day and loaded the full collection, as it currently does, it'll cause 10,000,000 document reads (at the very least).

I'm sure there must be a better way to do this, however, I can't seem to crack it.

Any advice is greatly appreciated on a better way to approach this and keep the costs down.

Answer 1

How many products will the average visitor to your homepage see? Say that you show 5-10 products above the fold, so before they have to scroll. If 50% of the visitors leave the page straight away (which is a fairly optimistic value for this so-called bounce) you only need to load 5,000 visits/day * 10 products = 50,000 reads to satisfy half of your visitors.

And the 50% visitors that stay on your site to see more content, actually are the ones for whom it is worth to load more data. But I'd still consider optimizing

Loading all 2,000 products is likely wasteful for the vast majority of your visitors, which is why Doug recommends paginating the data (or nowadays more common: load them with an infinite-scroll view).

An even more efficient alternative is to store the initial 10 products into a single document, essentially storing pre-aggregated content. If you do that, you only need to read one document per visitor to get the initial products. And then you can load the rest of the documents on demand as before.

You can take this approach further and pre-aggregate multiple types of content, e.g. the most popular products for each category, the related products for a specific product, etc. While each of these steps makes generating the content more tricky, you easily earn it back if you get more visitors.

To learn all about this and many other considerations when using Cloud Firestore, I'd recommend the excellent video series Getting to know Cloud Firestore.