Refreshing RSS feeds at near-realtime intervals

Question

I've got a system that fetches a few hundred RSS feeds. Currently they're on a 10 minute refresh cycle, but I'd preferably like to make that faster. What is a strategy to fetch the RSS sources at near-realtime/push intervals?

Some solutions I've come across:

• do a fetch at 1 minute; if no changes, fetch again at 2, then 4, then 8, etc.
• find the average time-between-updates interval/variance of the RSS feed, and put them in a bucket (this one updates every 3 mins, so do a check every 1 minute; this one updates every week, so do a check every day, etc.)

Answer 1

There is no way to make "pulling" quick and efficient. You will either poll more often (and be less efficient) or be more efficient by polling less often.

The only way to acheive near realtime experience is to poll at the right time :)

Luckily some publihsers (more and more!) use PubSubHubbub to update their feeds and let subscribers know. Other services like Superfeedr (I work for Superfeedr) use different techniques to learn when is the best time to fetch a feed (based on historic updates, updates in related feeds... etc).