Designing a Firebase based scalable feed model

Question

Question :

How to design a social network "feed" with Firebase as backend, that scales ?

Possible answers :

"MVP" solution is to design a feeds root child, one for each user, and append any new post from the followed user in every follower's feeds.

users
  user1
    name: bob
  user2
    name: alice
    follows: 
      user1: true

posts
  post1
     author: user1
     text: 'Hi there'

feeds
  user2
    post1: true

This works well, and is demoed in the Firefeed project. But it does not scale well : if Katy Perry wants to post something, her mobile phone will have to write to millions of feed.

Hence the solution reported in this SO question to delegate this operation to a server based process.

My problem is, Firebase is a "no-backend" solution, and this is the main reason why I use it, so I'd like to make sure there is absolutely no chance of implementing this feature without a server.

What if the feeds child is removed in the above schema ?

Then do this :

baseRef.child('posts')
       .orderBy('author')
       .whereIn(baseRef.child('users/user2/follows').keys())

Unfortunately, whereIn does not exists in Firebase API, nor subqueries :(

Any other model structure possible without the need of a server ?

Thanks

Answer 1

Firebase guys kinda replied on their blog : https://www.firebase.com/blog/2015-10-07-how-to-keep-your-data-consistent.html

The post is about "Data fanning" (spreading items across many nodes in one atomic write operation).

The technique greatly addresses the feed model of the original question

The post actually contains example code for implementing it :

• Function for creating the fannout object (actually a simple object with keys being API endpoints to be written)

  function fanoutPost({ uid, followersSnaphot, post }) {
          // Turn the hash of followers to an array of each id as the string
          var followers = Object.keys(followersSnaphot.val());
          var fanoutObj = {};
          // write to each follower's timeline
          followers.forEach((key) => fanoutObj['/timeline/' + key] = post);
          return fanoutObj;  
  }
  

• And the logic using this function :

  var followersRef = new Firebase('https://<YOUR-FIREBASE-APP>.firebaseio.com/followers');
  var followers = {};
  followersRef.on('value', (snap) => followers = snap.val());
  var btnAddPost = document.getElementById('btnAddPost');
  var txtPostTitle = document.getElementById('txtPostTitle');
  btnAddPost.addEventListener(() => {
        // make post
        var post = { title: txtPostTitle.value };
        // make fanout-object
        var fanoutObj = fanoutPost({ 
            uid: followersRef.getAuth().uid, 
            followers: followers, 
            post: post 
        });
        // Send the object to the Firebase db for fan-out
        rootRef.update(fanoutObj);
  });
  

---

Note: this is way more scalable than a loop writing each time in one follower feed. However, it could nevertheless be insufficient for millions of followers. In that case, it would be safer to trust a server operation making several writes. I think client-side can be used for up to a few hundreds followers, which is the average number of followers on social media. (This needs to be verified by testing though)