Building a simple news feed in node + Mongodb + Redis

Question

My goal is to build a simple news feed in node.js with the help of mongodb and redis. It similar like twitter

So the scenario is pretty straight forward, once User A follow User B. Later on User's A News feed (Home page) will be shown User B's Activity like what he posted.

Schema for User

const UserSchema = new Schema({
  email: { type: String, unique: true, lowercase: true},
});



const followSchema = new Schema(
    {
        user: { type: Schema.Types.ObjectId, required: true, ref: 'User' },
        target: { type: Schema.Types.ObjectId, required: true, ref: 'User' },
    });

Currently the design of my user's schema is pretty simple, when I follow another user, I will just create the Follow Schema Object

and there is another schema, which is post schema

/* This is similar like the Tweet */
var PostSchema = new Schema({
  // Own by the user
  creator: { type: Schema.Types.ObjectId, ref: 'User' }
  body: String,
});

This schema is for user to post anything, similar like twitter posting.

Let say I have followed bunch of users

{
   user: 'me',
   target: 'draco'
},

{
  user: 'me',
  target: 'donald'
},

{
  user: 'me',
  target: 'joker'
}

and let say one of my followers, post something. How do i present it to my current news feed?

/* I'm following Joker */
app.post('/follow', (req, res, next) => {
   let follow = new Follow();
   follow.user = "me";
   follow.target = "joker";
   // Do i need to use redis to subscribe to him?
   follow.save();
})


/* Joker posted something */
app.post('/tweet',(req, res, next) => {
   let post = new Post();
   post.creator = "joker";
   post.body = "Hello my name is joker"
   post.save();
   // Do i need to publish it using redis so that I will get his activity?

});

Here's my attempt

app.get('/feed', function(req, res, next) {

     // Where is the redis part?
     User.findOne({ _id: req.user._id }, function(err, foundUser) {
        // this is pretty much my attempt :(
     })
})

When should I use redis to actually do the pub and sub? so that I could take the content of one of my followers and show it on my timeline?

Answer 1

I have built a social network which has a news feed, too. Here is how I did it.

---

Basically, you have 2 methods to built a newsfeed:

1. Fanout on write (push) method
2. Fanout on read (pull) method

---

Fanout on write

First, you will need another collection:

const Newsfeed = new mongoose.model('newsfeed', {
  owner: {type: mongoose.Types.ObjectId, required: true},
  post: {type: mongoose.Types.ObjectId, required: true}
});

When a user post something:

1. Get n follower
2. Push (fanout) this post to n follower

When a user get a feed:

1. Get from Newsfeed collection

Example:

router.post('/tweet', async (req, res, next) => {
  let post = await Post.create({});

  let follows = await Follow.find({target: req.user.id}).exec();

  let newFeeds = follows.map(follow => {
    return {
      user: follow.user,
      post: post.id
    }
  });
  await Newsfeed.insertMany(newFeeds);
});

router.get('/feed', async (req, res, next) => {
  let feeds = await Newsfeed.find({user: req.user.id}).exec();
});

---

Fanout on read

When a user post something:

1. Save

When a user get feed

1. Get n following
2. Get posts from n following

Example

router.post('/tweet', async (req, res, next) {
  await Post.save({});
});

router.get('/feeds', async (req, res, next) {
  let follows = await Follow.find({user: req.user.id}.exec();

  let followings = follows.map(follow => follow.target);

  let feeds = await Post.find({user: followings}).exec();
});

---

You don't need Redis or pub/sub to implement a newsfeed. However, in order to improve the performance, you may need Redis to implement some kind of cache for this.

For more information or advance technique, you may want to take a look at this.