Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Map-Reduce count number of documents in each minute MongoDB

I have a MongoDB collection which has a created_at stored in each document. These are stored as a MongoDB date object e.g.

{ "_id" : "4cacda7eed607e095201df00", "created_at" : "Wed Oct 06 2010 21:22:23 GMT+0100 (BST)", text: "something" }
{ "_id" : "4cacdf31ed607e0952031b70", "created_at" : "Wed Oct 06 2010 21:23:42 GMT+0100     (BST)", text: "something" }
....

I would like to count the number of items created between each minute, so I can pass the data into Google Charts to generate something like this:

alt text

How do I do this with a map reduce function, or is there a fancy MongoDB aggregate function which I could use instead?

like image 872
Tom Avatar asked Oct 07 '10 07:10

Tom


People also ask

Why mapReduce is discouraged in MongoDB?

Pitfalls of MongoDB can be summarised as - So again we had to learn the hard way that MongoDb's map-reduce functionality just isn't meant for real time computing; it is extremely slow, especially when you have a large amount of data in a shared environment.

What is MongoDB map reduce?

In MongoDB, map-reduce is a data processing programming model that helps to perform operations on large data sets and produce aggregated results. MongoDB provides the mapReduce() function to perform the map-reduce operations. This function has two main functions, i.e., map function and reduce function.

What is emit function in MongoDB?

The map function may optionally call emit(key,value) any number of times to create an output document associating key with value . In MongoDB 4.2 and earlier, a single emit can only hold half of MongoDB's maximum BSON document size. MongoDB removes this restriction starting in version 4.4.

Is MongoDB count fast?

MongoDB 'count()' is very slow.


1 Answers

Map function should emit a timestamp object, adjusted up to the minute, and a count of 1. The reduce function should sum all the counts:

map = function() {
    var created_at_minute = new Date(this.created_at.getFullYear(),
                                     this.created_at.getMonth(), 
                                     this.created_at.getDate(), 
                                     this.created_at.getHours(), 
                                     this.created_at.getMinutes());
    emit(created_at_minute, {count: 1});
}

reduce = function(key, values) { 
    var total = 0;
    for(var i = 0; i < values.length; i++) { 
        total += values[i].count; 
    }
    return {count: total};
}
like image 93
rubayeet Avatar answered Nov 15 '22 06:11

rubayeet