Persist all users notifications in a list with redis?

Question

I have aspects of a social network in my app and I have implemented an activity stream like this answer:

How to implement the activity stream in a social network

Like said there, every notification in the system, I push in redis for each user (key) that is notified, a list (value) of the IDs from the Activities relational table:

key                         value
user:1:notifications        [25, 24, 23]
user:2:notifications        [24, 22, 17, 13, 5, 4]
...

So, my table only has the activities and the user that causes this. What happens is that I only have the users that received the notifications in redis and nothing in mysql...

My question is that if is correct to persist this ids infinite in redis or just for a memcached of updates and periodic I trim this list?

Answer 1

This is more of an application architecture / design question than a programming one, so there is no one right answer in theory.

However, in practice - Redis / Memcache and many other such implementations are not meant to persist very large (or rapidly growing) data sets.

As a nosql data store, Redis uses memory, coupled with a mirror on the hard disk. So while there is no limit on the size of data you can store, ideally it should always be less than the free memory you plan to allocate to Redis.

The easiest solution to cover all bases is to store user activity data in Redis as it is generated. Use the Redis to display notifications, etc. Keep a cron running that truncates all activity logs older than a pre-defined number of days (or a pre-defined number of activities per user) and saves them to a regular database.

When a user wishes to retrieve all notifications, some speed loss is acceptable (since it is not a frequent, required or promoted action) and you can pull them from the database, by-passing Redis.

Alternative: Again, the solution to use is best chosen based on the actual numbers of your application. But you could do this:

• Store all activities in the database
• For any logged in user, fetch and store all activities in Redis
• On log-out remove that user's activities / notifications from Redis
• When adding an activity, some additional logic required to check if the users affected are online or not. In both cases you need to add the activity to the database, but if the affected user is online, then push it to his hash in Redis as well.