Menu
I'd like to create a reusable Django app that handles status updates of the Users. Much like facebook's "news feed".
Use cases includes, for example:
One concern is how to handle different kinds of Updates and display the correct "html snippet" for it. The other, which is more important, is how to design the Models of this "Django way".
About the former, I could think of two ways of doing it:
I searched before posting here, but I didn't find anything. I checked Pinax to see if they had it implemented, but they don't. So, I'm here looking for more suggestions on how to handle this in a nice and non-hacky way.
Thanks in advance,
580
Python is actually a great language for building Activity Streams and Newsfeeds. Tommaso and I have written the Stream Framework package. https://github.com/tschellenbach/stream-framework It is currently the most used Python solution for building newsfeeds. We're also offering a hosted solution at https://getstream.io. The Django client is by far the easiest to get started with: https://github.com/GetStream/stream-django and python can be found here (https://github.com/getstream/stream-python)
The templating part works like this
{% load stream_django %}
{% for activity in activities %}
{% render_activity activity %}
{% endfor %}
This will render a template located in activity/tweet.html with the activity as context. For instance
{{ activity.actor.username }} said "{{ activity.object.body }} {{ activity.created_at|timesince }} ago"
The full docs are here: https://github.com/GetStream/stream-django#templating
The Stream Framework allows you to build any type of newsfeed using either Redis or Cassandra. It's build to scale and creates the individual newsfeeds using a fanout process.
Besides the Stream Framework (which I obviously prefer) there are many other solutions out there. A full list is available on django packages: https://www.djangopackages.com/grids/g/activities/
Note that with newsfeeds there are a few scaling issues to keep in mind. In general there are 3 common approaches:
Denormalization strategies
Pull Most users start out this way. When you open the feed page you simply query the feeds from all the users you follow. If the users feeds are stored in memory this will keep on working for quite some time. Eventually it is quite hard to keep on using such as strategy though as you often have to query most of the nodes storing your user's feeds.
Push The push approach writes your activity to all of your followers feeds. Of course this means you're wasting a ton of resources, but the end result is a pre computed feed per user. This approach (though initially not very efficient) scales nicely.
Combination Some optimized systems use a combination of these two approaches. Also see the Yahoo paper on this topic.
Storage options
In terms of storing all this data the most common options are Redis, Cassandra and MongoDB. Let's quickly compare these:
Redis Redis is Extremely easy to setup and maintain. It however only stores data in memory. This means you'll have to optimize how you serialize data and maybe fallback to the database for less frequently queried data. Another problem is that it's not trivial to add machines to your Redis cluster.
MongoDB Mongo DB is used primarily by a few ruby projects and it's also available as a backend for pump.io by e14n. I've personally never ran it in production so I can't properly evaluate this option. There are however a lot of blogposts covering issues with the performance, scalability and maintainability of mongo.
Cassandra Fashiolista, Instagram and Spotify are all using Cassandra. Our hosted solution also uses Cassandra as a backend. It's extremely cost effective to operate and you can add more nodes with ease. The only problem is that it's hard to setup and maintain.
Articles
In addition have a look at this high scalability post were we explain some of the design decisions involved: http://highscalability.com/blog/2013/10/28/design-decisions-for-scaling-your-high-traffic-feeds.html
To learn more about feed design I highly recommend reading some of the articles which we based Feedly on:
54
I can think in two ways:
First, maybe you could make feeds for your models Assigments, PdfFiles, and Youtube link, and use the library feedparser to embed it in your news views, this is the easy way because you can define in the templates, the code for each kind of new activity.
The second thing I can think of is to make a class Activity:
class Activity(models.Model):
date = models.DateTimeField(auto_now_add = True)
content_type = models.ForeignKey(ContentType)
object_id = models.PositiveIntegerField()
content_object = generic.GenericForeignKey('content_type', 'object_id')
And through the signals make a new instance of Activity every time you have a new assigment or pdf upload or youtube link, and for each class make a method like render_to_html, in this way in your view, you can make a for over Activities and call the method render_to_html
34
After more googling and one helpful keyword("Activity") that diegueus9 mentioned and that I haven't thought before, I was able to find more relevant material.
First, two blog posts on how to build a tumbleblog using django using the ContentType framework:
After that, another post that gives suggestions on how to reduce the (1 + n) queries problem (which was one of my concerns initially, but I didn't mention to avoid cluttering the question).
And finally an reusable Django app that has some of the feature that I needed and can be useful for further reference:
23
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
