Progress notifications from HTTP/REST service

Question

I'm working on a web application that submits tasks to a master/worker system that farms out the tasks to any of a series of worker instances. The work queue master runs as a separate process (on a separate machine altogether) and tasks are submitted to the master via HTTP/REST requests. Once tasks are submitted to the work queue, client applications can submit another HTTP request to get status information about tasks.

For my web application, I'd like it to provide some sort of progress bar view that gives the user some indication of how far along task processing has come. The obvious way to implement this would be an AJAX progress meter widget that periodically polls the work queue for status on the tasks that have been submitted. My question is, is there a better way to accomplish this without the frequent polling?

I've considered having the client web application open up a server socket on which it could listen for notifications from the work master. Another similar thought I've had is to use XMPP or a similar protocol for the status notifications. (Of course, the master/worker system would need to be updated to provide notifications either way but I own the code for that so can make any necessary updates myself.)

Any thoughts on the best way to set up a notification system like this? Is the extra effort involved worth it, or is the simple polling solution the way to go?

Answer 1

Polling

The client keeps polling the server to get the status of the response.

Pros

• Being really RESTful means cacheable and scaleable.

Cons

• Not the best responsiveness if you do not want to poll your server too much.

---

Persistent connection

The server does not close its HTTP connection with the client until the response is complete. The server can send intermediate status through this connection using HTTP multiparts.

Comet is the most famous framework to implement this behaviour.

Pros

• Best responsiveness, almost real-time notifications from the server.

Cons

• Connection limit is limited on a web server, keeping a connection open for too long might, at best load your server, at worst open the server to Denial of Service attacks.

---

Client as a server

Make the server post status updates and the response to the client as if it were another RESTful application.

Pros

• Best of every worlds, no resources are wasted waiting for the response, either on the server or on the client side.

Cons

• You need a full HTTP server and web application stack on the client
• Firewalls and routers with their default "no incoming connections at all" will get in the way.

---

Feel free to edit to add your thoughts or a new method!