MongoDB database schema design

Question

I have a website with 500k users (running on sql server 2008). I want to now include activity streams of users and their friends. After testing a few things on SQL Server it becomes apparent that RDMS is not a good choice for this kind of feature. it's slow (even when I heavily de-normalized my data). So after looking at other NoSQL solutions, I've figured that I can use MongoDB for this. I'll be following data structure based on activitystrea.ms json specifications for activity stream So my question is: what would be the best schema design for activity stream in MongoDB (with this many users you can pretty much predict that it will be very heavy on writes, hence my choice of MongoDB - it has great "writes" performance. I've thought about 3 types of structures, please tell me if this makes sense or I should use other schema patterns.

1 - Store each activity with all friends/followers in this pattern:

 

    {
     _id:'activ123',
     actor:{
            id:person1
            },
    verb:'follow',
    object:{
            objecttype:'person',
            id:'person2'
            },
    updatedon:Date(),
    consumers:[
            person3, person4, person5, person6, ... so on
            ]

    }

2 - Second design: Collection name- activity_stream_fanout

    {
    _id:'activ_fanout_123',
    personId:person3,
    activities:[
    {
     _id:'activ123',
     actor:{
            id:person1
            },
    verb:'follow',
    object:{
            objecttype:'person',
            id:'person2'
            },
    updatedon:Date(),
    }

    ],[
    //activity feed 2
    ]

    }

3 - This approach would be to store the activity items in one collection, and the consumers in another. In activities, you might have a document like:

    { _id: "123",
      actor: { person: "UserABC" },
      verb: "follow",
      object: { person: "someone_else" },
      updatedOn: Date(...)

    } 

And then, for followers, I would have the following "notifications" documents:

    { activityId: "123", consumer: "someguy", updatedOn: Date(...) }
    { activityId: "123", consumer: "otherguy", updatedOn: Date(...) }
    { activityId: "123", consumer: "thirdguy", updatedOn: Date(...) } 

Your answers are greatly appreciated.

Answer 1

I'd go with the following structure:

1. Use one collection for all actions that happend, Actions

2. Use another collection for who follows whom, Subscribers

3. Use a third collection, Newsfeed for a certain user's news feed, items are fanned-out from the Actions collection.

The Newsfeed collection will be populated by a worker process that asynchronously processes new Actions. Therefore, news feeds won't populate in real-time. I disagree with Geert-Jan in that real-time is important; I believe most users don't care for even a minute of delay in most (not all) applications (for real time, I'd choose a completely different architecture).

If you have a very large number of consumers, the fan-out can take a while, true. On the other hand, putting the consumers right into the object won't work with very large follower counts either, and it will create overly large objects that take up a lot of index space.

Most importantly, however, the fan-out design is much more flexible and allows relevancy scoring, filtering, etc. I have just recently written a blog post about news feed schema design with MongoDB where I explain some of that flexibility in greater detail.

Speaking of flexibility, I'd be careful about that activitystrea.ms spec. It seems to make sense as a specification for interop between different providers, but I wouldn't store all that verbose information in my database as long as you don't intend to aggregate activities from various applications.

Answer 2

I believe you should look at your access patterns: what queries are you likely to perform most on this data, etc.

To me The use-case that needs to be fastest is to be able to push a certain activity to the 'wall' (in fb terms) of each of the 'activity consumers' and do it immediately when the activity comes in.

From this standpoint (I haven't given it much thought) I'd go with 1, since 2. seems to batch activities for a certain user before processing them? Thereby if fails the 'immediate' need of updates. Moreover, I don't see the advantage of 3. over 1 for this use-case.

Some enhancements on 1? Ask yourself if you really need the flexibility of defining an array of consumers for every activity. Is there really a need to specify this on this fine-grained scale? instead wouldn't a reference to the 'friends' of the 'actor' suffice? (This would a lot of space in the long run, since I see the consumers-array being the bulk of the entire message for each activity when consumers typically range in the hundreds (?).

on a somewhat related note: depending on how you might want to implement realtime notifications for these activity streams, it might be worth looking at Pusher - http://pusher.com/ and similar solutions.

hth