"$push used too much memory and cannot spill to disk. Memory limit: 104857600 bytes"

Question

I am sending a get request from an application and getting the below mentioned error:

{"$push used too much memory and cannot spill to disk. Memory limit: 104857600 bytes"}

I am using atlas managed mongo-db with 4GB Ram and 20 GB storage. I think, I need to update some configuration setting, but I am missing on it. Can anyone help me resolve this error.

Adding the query

    db.collection1.aggregate([ { '$match': {} }, { '$lookup': { from: 'collection2', localField: 'pfId', foreignField: 'pfId', as: 'pt' } }, { '$unwind': '$pt' }, { '$match': {} }, { '$lookup': { from: 'collection3', localField: 'pt.ptId', foreignField: 'ptId', as: 'pd' } }, { '$unwind': '$pd' }, { '$match': {} }, { '$lookup': { from: 'collection4', localField: 'pd.pdId', foreignField: 'pdId', as: 'dl' } }, { '$unwind': '$dl' }, { '$match': { 'dl.currentdl': { '$ne': 'OBSOLETE' }, '$or': [ { 'pd.pdName': { '$regex': /^.*world.*/i } }, { 'dl.r.dF': { '$regex': /^.*world.*/i } }, { 'dl.dlType': { '$regex': /^.*world.*/i } }, { 'dl.dlId': 'WORLD' }, { 'dl.logUserId': 'WORLD' } ] } }, { '$project': {//some_fields} }, { '$group': { _id: null, count: { '$sum': 1 }, results: { '$push': '$$ROOT' } } }, { '$project': { count: 1, results: { '$slice': [ '$results',
0, 10 ] } } }], { allowDiskUse: true })

I have to paginate the data, along with the retrieving the count of data fetched.

Answer 1

So, after so much of struggle, here is what I did to resolve my problem. First thing was understood that I cann't push the data into an array without limiting it. So I used $limit and $skip, before grouping the data in the pipeline.(This also maintains the pagination for my query). But the problem was to maintain the total count of record as it was lost due to limit before grouping. So, the solution i figured out is to use $facet, which helps me in implementing two pipelines within the same aggregation pipeline.

db.prdfam.aggregate([{
$facet: {
"counts":[
{ '$match': {} },
{ '$lookup': { from: 'pt', localField: 'pfId', foreignField: 'pfId', as: 'pt' } },
{ '$unwind': '$pt' }, { '$match': {} }, 
{ '$lookup': { from: 'prds', localField: 'pt.ptId', foreignField: 'ptId', as: 'prd' } },
{ '$unwind': '$prd' }, { '$match': {} }, 
{ '$lookup': { from: 'del', localField: 'prd.prdId', foreignField: 'prdId', as: 'delivery' } }, 
{ '$unwind': '$delivery' }, { '$match': { 'delivery.currentDelivery': { '$ne': 'OBSOLETE' }, 
'$or': [ { 'prd.prdName': { '$regex': /^.*world.*/i } },
{ 'delivery.rInfo.dataFormat': { '$regex': /^.*world.*/i } },
{ 'delivery.dType': { '$regex': /^.*world.*/i } }, 
{ 'delivery.dId': 'WORLD' }, { 'delivery.UserId': 'WORLD' } ] } }, 
{ '$group': { _id: null, count: { '$sum': 1 } } }
],
"results":[
//same lookup & match conditions as in above element 
{ '$project': { //fields to project } }, 
{ '$sort': { updatedAt: -1 } }, {$skip: 0},{ $limit : 10 },
{ '$group': { _id: null, results: { '$push': '$$ROOT' } } },
{ '$project': { results: 1 } }
]
}
}], 
{ allowDiskUse: true })

Hope this will help others. Cheers :)