Lightweight notification technique

Question

I need to develop a realtime recent activity feed in django (with AJAX long-polling), and I'm wondering what's the best strategy for the server-side.

Pseudocode:

def recent_activity_post_save():
    notify_view()

[in the view]
while not new_activity():
    sleep(1)
return HttpResponse(new_activity())

The first thing that comes in mind is querying the DB every second. Not feasible. Other options:

1. using the cache as a notification service
2. using a specialized tool, like Celery (I'd rather not do it, because it seems like overkill)

What's the best way to go here?

Answer 1

I would suggest keeping it simple...

Create a database table to store your events, insert into that table when appropriate, then just implement a simple ajax polling technique to hit the server every x seconds on the client side.

I have concerns with other solutions considering using a push-notification approach or using a noSql data store. It's a whole lot more complicated than a traditional pull-notification system using the tools that are built in to the Django framework, and except for very rare exceptions, is overkill. Unless you specifically require a strict real-time solution, keep it simple and use the tools that already exist in the framework, and for people with objections based on database or network performance, all I have to say is that premature optimization is the root of all evil.

Build a model that contains recent activity data specific to your application then, whenever your application does something that should log new activity you can just insert into this table.

Your view would simply be like any other view, pulling the top x rows from this RecentActivity table (optionally based on query parameters and whatever).

Then, on the client side, you'd just have a simple ajax poller hitting your view every x seconds. There is no shortage of complicated plugins and technologies you can use, but writing your own isn't that complicated either:

function simplePoll() {
  $.get("your-url", {query-parameters}, function(data){
    //do stuff with the data, replacing a div or updating json or whatever
    setTimeout(simplePoll, delay);
  });
}

My opinion is that performance issues aren't really issues until your site is successful enough for them to be an issue. A traditional relational database can scale up fairly well until you start reaching the level of success like Twitter, Google, etc. Most of us aren't at that level :)

Answer 2

Have you considered using Signals? You could send a signal in recent_activity_post_save() and there could be a listener which stores the information in cache.

The view would just refer to the cache to see if there are new notifications. Of course you don't need Signals, but IMHO it would be a bit cleaner that way, as you could add more "notification handlers".

This seems optimal because you don't need to poll the DB (artificial load), the notifications are "visible" almost immediately (only after the time required to process signals and interact with cache).

So the pseudocode would look like this:

# model
def recent_activity_post_save():
    post_save_signal.send()

# listener
def my_handler( ... ):
    cache.set( 'notification', .... )

post_save_signal.connect( my_handler )

# view
def my_view( request ):
    new_notification = None
    while not new_notification:
        sleep(1)
        new_notification = cache.get( 'notification' )
    return HttpResponse(...)

Answer 3

You could use a comet solution, like the Ape project. This kind of project is designed to send real-time data to the browser, and can make use of modern browsers web sockets feature.