Spurious MaxListenersExceededWarning EventEmitter memory leak when running consecutive firestore queries

Question

I have a firebase HTTP function which in turns calls some firestore operations. If I call the HTTP function several times, letting each call finish before calling the next, I get the following error in the firebase functions log:

(node:2) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added. Use emitter.setMaxListeners() to increase limit

The firebase function is an import task which takes the data to import, check for duplicates by calling a firestore query, and if there is none, it adds the data to the firestore DB by another DB operation.

Here is the firebase function, with parts removed for brevity:

module.exports = functions.https.onCall(async (obj, context) => {
  // To isolate where the problem is
  const wait = (ms: number) => new Promise(resolve => setTimeout(resolve, ms))
  try {
    const photo = getPhoto(obj)
    // Query to look for duplicates
    const query = db
      .collection(`/Users/${context.auth.uid}/Photos`)
      .where('date', '==', photo.date)
      .where('order', '==', photo.order)
      .limit(1)

    await wait(300)
    log.info('Before query')
    const querySnap = await query.get()
    log.info('After Query')
    await wait(300)

    // And then the rest of the code, removed for brevity
  } catch (error) {
    throw new functions.https.HttpsError('internal', error.message)
  }
})

I inserted a pause before and after the const querySnap = await query.get() to show that it really is this invocation that causes the error message.

I also set the firestore logger to output its internal logging to help debug the issue, by doing this:

import * as admin from 'firebase-admin'
admin.initializeApp()
admin.firestore.setLogFunction(log => {
    console.log(log)
})

So the more complete log output I get is this: (read it bottom to top)

12:50:10.087 pm: After Query
12:50:10.087 pm: Firestore (2.3.0) 2019-09-13T19:50:10.087Z RTQ7I [Firestore._initializeStream]: Received stream end
12:50:10.084 pm: Firestore (2.3.0) 2019-09-13T19:50:10.084Z RTQ7I [Firestore._initializeStream]: Releasing stream
12:50:10.084 pm: Firestore (2.3.0) 2019-09-13T19:50:10.084Z RTQ7I [Firestore.readStream]: Received response: {"document":null,"transaction":{"type":"Buffer","data":[]},"readTime":{"seconds":"1568404210","nanos":76771000},"skippedResults":0}
12:50:10.026 pm: (node:2) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added. Use emitter.setMaxListeners() to increase limit
12:50:10.020 pm: Firestore (2.3.0) 2019-09-13T19:50:10.020Z RTQ7I [Firestore.readStream]: Sending request: {"parent":"[redacted]/documents/Users/SpQ3wTsFzofj6wcsF7efRrSMrtV2","structuredQuery":{"from":[{"collectionId":"Photos"}],"where":{"compositeFilter":{"op":"AND","filters":[{"fieldFilter":{"field":{"fieldPath":"date"},"op":"EQUAL","value":{"stringValue":"2019-06-26"}}},{"fieldFilter":{"field":{"fieldPath":"order"},"op":"EQUAL","value":{"integerValue":0}}}]}},"limit":{"value":1}}}
12:50:10.019 pm: Firestore (2.3.0) 2019-09-13T19:50:10.019Z RTQ7I [ClientPool.acquire]: Re-using existing client with 100 remaining operations
12:50:10.012 pm: Before query

The interesting thing is that I usually run these imports in batches of 10. I seem to only get the error during the first batch of 10. If I then quickly run more batches, I don't seem to get the error again. But if I wait some time, it returns. Also, it is not consistent in which invocation within a batch the error occurs. It may be the 9th or 2nd or invocation, or any other.

Finally, the error doesn't stop execution. In fact, the imports seem to never fail. But, I don't like have unaccounted for errors in my logs! I won't be able to sleep at night with them there. :-)

I'm grateful for any help you can offer.

Answer 1

I got a useful response from the Firebase support team. They told me to try to install the latest version of firebase-admin (which upgraded it from 8.5.0 to 8.6.0) and that resolved the issue, even without the workaround of installing grpc. So, I think this should be the correct answer now.

Answer 2

Looks like this bug MaxListenersExceededWarning: Possible EventEmitter memory leak detected #694 might the problem here.

Workaround is to use npm install @grpc/[email protected] --save-exact until bug is fixed and the Firestore library starts using it.