How to get all Twitter links?

Question

There are lots of sites offering additional services to Twitter:

• hashtags.org
• tweetmeme.com
• repeets.com
• dailyrt.com
• backtweets.com

They all have something in common: They seem to get all updates from Twitter. Otherwise, they couldn't offer such complex services with high quality results.

But how do they do this? Twitter has a XMPP feed (Firehose) but not everybody is allowed to get data from it.

Are there any approaches you can recommend or which you think they are using? The normal API is limited too much, isn't it?

Answer 1

Quoted from http://apiwiki.twitter.com/Rate-limiting

Whitelisting

Some applications find that the default limit proves insufficient. Under such circumstances, we offer whitelisting. It is possible to whitelist both accounts and IP addresses. Each whitelisted entity, whether an account or IP address, is allowed 20000 requests per hour. If you are developing an application that should be considered for whitelisting, please fill out the whitelisting request form. Our manual review process can take up to a week. If you have a whitelisting that needs to be updated through the addition or removal of IP addresses, reapply with an explanation of the change. Approval or rejection for whitelisting requests is emailed to the email address associated with the account that filed the application.

Answer 2

Twitter has a new Streaming API which it looks like anyone can access, it's still under alpha but this is what tweetmeme.com is probably using.