How to model a social news feed on Google App Engine

Question

We want to implement a "News feed" where a user can see messages broadcasted by her friends, sorted with newest message first. But the feed should reflect changes in her friends list. (If she adds new friends, messages from those should be included in the feed, and if she removes friends their messages should not be included.) If we use the pubsub-test example and attach a recipient list to each message this means a lot of manipulation of the message recipients lists when users connect and disconnect friends.

We first modeled publish-subscribe "fan out" using conventional RDBMS thinking. It seemed to work at first, but then, since the IN operator works the way it does, we quickly realized we couldn't continue on that path. We found Brett Slatkin's presentation from last years Google I/O and we have now watched it a few times but it isn't clear to us how to do it with "dynamic" recipient lists.

What we need are some hints on how to "think" when modeling this.

Answer 1

Pasting the answer I got for this question in the Google Group for Google App Engine http://groups.google.com/group/google-appengine/browse_thread/thread/09a05c5f41163b4d# By Ikai L (Google)

A couple of thoughts here:

• is removing of friends a common event? similarly, is adding of friends a common event? (All relative, relative to "reads" of the news feed)

• From what I remember, the only way to make heavy reads scale is to write the data multiple times in peoples' streams. Twitter does this, from what I remember, using a "eventually consistent" model. This is why your feed will not update for several minutes when they are under heavy load. The general consensus, though, is that a relational, normalized model simply will not work.

• the Jaiku engine is open source for your study: http://code.google.com/p/jaikuengine. This runs on App Engine Hope these help when you're considering a design.