Menu
I have a script that periodically runs aggregation on a mongodb collection. As the dataset has grown, the amount of time it takes to aggregate has also grown. My aggregation script has recently stopped working consistently, and the error logs show:
error: { [MongoError: server <x> timed out]
name: 'MongoError',
message: 'server <x> timed out' }
I've tried debugging this, and the only pattern I can find is that this timeout seems to only occur when the aggregation takes longer than 2 minutes (it times out right around 2m). Does anyone have additional debugging tips for this? The 2-minute thing is giving me the impression that I just need to configure some timeout somewhere but I can't figure out where or if i'm just falling into a red-herring trap.
About the system configuration: This aggregation script is a node.js (v5.9.1) application running in an alpine-based docker (v1.9.1) container. It uses the mongodb node driver (v2.1.19). Single mongodb server (though this is also happening in a separate environment with a replSet) running mongod (v3.2.6)
337
When the client takes too much time to connect to MongodB server than the preset timeout value, it can result in error. And the fix involves in raising the timeout limits on the MongoDB client. For this, we first check with the customer on the settings that they use on their client. We then compare the values set in the MongoDB server.
With the introduction of $merge in version 4.2, MongoDB provides two stages, $merge and $out, for writing the results of the aggregation pipeline to a collection. The following summarizes the capabilities of the two stages:
The $out operator lets the aggregation framework return result sets of any size. Starting in MongoDB 4.4, $out can take a document to specify the output database as well as the output collection: The output database name. For a replica set or a standalone, if the output database does not exist, $out also creates the database.
When the client takes too much time to connect to MongodB server than the preset timeout value, it can result in error. And the fix involves in raising the timeout limits on the MongoDB client. For this, we first check with the customer on the settings that they use on their client.
I got the same problem for logs time aggregation. I think I have the solution for you.
I found that the option socketTimeoutMS is responsible for that.
Check your mongo_client.js default socketTimeoutMS value. For me it was 2min. Mongodb module version 2.1.18.
So just add this option into your url :
mongodb://localhost:27017/test?maxPoolSize=2&socketTimeoutMS=60000
It will set timeout to 10 mins. That does the trick for me.
143
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
