Firestore suddenly has a huge trigger delay

Question

We are running an application on Firestore and got a simple trigger that when order's details are created or updated some of it's information should be rewritten in the parent order collection.

The function for this got following code

export const updateOrderDetails = functions
  .region(FUNCTION_REGION)
  .firestore.document("orders/{orderId}/details/pickupAndDropoff")
  .onWrite(async (change, context) => {
    return await admin
      .firestore()
      .collection("orders")
      .doc(context.params.orderId)
      .set({ pickupAndDropoff: change.after.data() }, { merge: true });
  });

It was work fine before, but now at random about every third of its executions is delayed. Sometimes by few minutes. In Cloud Function logs we see normal execution times <200ms, so it seems the trigger runs after a huge pause.

What's worse from time to time our change.after.data() is undefined, but we never delete anything - it's just updates and creates.

It was working fine, we did not changed nothing since last week, but now it started to have this unexpected delays. We've also checked the firebase status, but there are no malfunctions in firebase functions service. What can be the cause of this?

Answer 1

The problem can be due to the monotonically increasing orderId as the parameter passed here:

...
.collection("orders")
  .doc(context.params.orderId)
...

If you can check once if the orderId passed here is monotonically increasing with each request? It can lead to hotspots which impacts latency.

To explain, I think the write rate must be changing at different day's and time's - as the user traffic using the application or load testing requests changes - which is creating the unexpected kind of behaviour. At low write rate, the requests are working as expected most of the time. At high write rate, the requests are facing hotspot situation in the firestore as mentioned in the firestore documentation resulting in delays (latency issue).

Here is the relevant link to firestore best practices documentation.