Difference between Firestore Query stream & get?

Question

I'm planning to query large set of data from Firestore database using NodeJS api. Is there any benefit on using stream api (https://cloud.google.com/nodejs/docs/reference/firestore/0.13.x/Query?#stream) instead of reqular query get (https://cloud.google.com/nodejs/docs/reference/firestore/0.13.x/Query?#get)?

My impression is that streaming is more efficient in terms of memory consumption.

My environment for querying is Firebase functions.

Answer 1

Yes, using stream over get will reduce the memory consumption of your Cloud Function! I have experienced it first-hand: my function's memory went from 1GB to 200MB when I replaced get with stream.

To give more details: with the Node.js Admin SDK, in order to access your data:

• First you define a query. You can specify a collection and add where, offset, limit, startAfter to describe more precisely the data that you need. If you don't add any specifier, you are querying the entire collection.
• Then you fetch the corresponding results, through one of these 3 functions:
  • get, which returns a snapshot containing all documents matching your query (or the whole collection if you did not limit your query). You absolutely need to add a limit clause to your query if you do not want to explode your memory.
  • onSnapshot, which creates a listener which is going to provide you with documents matching your query. I noticed that it sends an initial big batch of all documents, then sends documents as they are created. So it can explode your memory, and you don't know when to stop listening, so I don't use it in Cloud Functions (but I really like it on client-side, another story)
  • stream, which provides you with a Node.js readable stream. You can consume this stream which provides documents one by one, so it is nice to your memory.