How safe is it to use only redis to implement activity streams?

Question

I am building a website that will allow users to 'follow' different members and whenever that member posts, it adds it to all the followers activity streams in reverse chronological order. Pretty common stuff these days.

But I am having trouble determining the most scalable/future-proof way of implementing this (without joins).

I am using flask/heroku/sqlalchemy/postgres. I am thinking using the Open-redis addon and doing something like instagram mentioned: each user has a redis list and whenever a 'followed' posts, it adds the post to the beginning of each followers list and trims its total size.

I just don't know enough about redis to know if it is safe to rely soley on it for every users streams. With open-redis, they don't mention anything about persistence so I don't know if that is just a common enough thing these days that it is just assumed.

If redis is not 100% reliable/safe, any thoughts on how I could do this with postgres/Hstore (no joins please).

Answer 1

We built a solution for the 2nd largest online fashion community and open sourced our approach: https://github.com/tschellenbach/Feedly It's currently the largest open source library aimed at solving this problem. You can use both Redis and Cassandra as storage backends.

We've ran on Redis for over a year without any troubles. If properly setup it runs extremely stable and is unlikely to lose data.

The same team which built Feedly also offers a hosted API, which handles the complexity for you. Have a look at getstream.io There are client libraries for Python, PHP, Node and Ruby.

In addition have a look at this high scalability post were we explain some of the design decisions involved: http://highscalability.com/blog/2013/10/28/design-decisions-for-scaling-your-high-traffic-feeds.html

This tutorial will help you setup a system like Pinterest's feed using Redis. It's quite easy to get started with.

To learn more about feed design I highly recommend reading some of the articles which we based Feedly on:

• Yahoo Research Paper
• Twitter 2013 Redis based, with fallback
• Cassandra at Instagram
• Etsy feed scaling
• Facebook history
• Django project, with good naming conventions. (But database only)
• http://activitystrea.ms/specs/atom/1.0/ (actor, verb, object, target)
• Quora post on best practises
• Quora scaling a social network feed
• Redis ruby example
• FriendFeed approach
• Thoonk setup
• Twitter's Approach

Answer 2

I think it is safe enough. Redis supports two forms of persistence. You can configure AOF persistence so you will likely not lose any data in the event of a crash.

Redis is relatively simple software and does not crash easily. My anecdotal evidence is an app where everything is stored in Redis. It currently stores about 340k keys in Redis for about 220k users and related info. Redis has never crashed or had any problems. The app has been running for almost 6 months and I've only restarted Redis when restarting the server a couple of times.