tweepy wait_on_rate_limit not working

Question

So, first off, I realize there's a number of questions regarding handling the twitter rate limits. I have no idea why, but none of the ones's I've found so far work for me.

I'm using tweepy. I'm trying to get a list of all the followers of the followers of a user. As expected, I can't pull everything down all at once due to twitter's rate limits. I have tweepy v 3.5 installed and thus am referring to http://docs.tweepy.org/en/v3.5.0/api.html. To get the list of followers of the originating user I use:

auth = tweepy.OAuthHandler(consumer_key,consumer_secret)
auth.set_access_token(access_token, access_secret)
api = tweepy.API(auth)

followerIDs = []
for page in tweepy.Cursor(api.followers_ids, screen_name=originatingUser, wait_on_rate_limit = True, wait_on_rate_limit_notify = True).pages():
    followerIDs.extend(page)

followers = api.lookup_users(follower)

This works a for a bit but quickly turns into:

tweepy.error.TweepError: [{u'message': u'Rate limit exceeded', u'code': 88}]

My theory, would then to retrieve the followers of each user for each followerID using something like this:

for followerID in followerIDs:
        for page in tweepy.Cursor(api.followers_ids, id=followerID, wait_on_rate_limit = True, wait_on_rate_limit_notify = True).pages():
                followerIDs.extend(page)

The other problem I have is when I'm trying to look up the user names. For this, It use the grouper function from itertools to break the followers up into groups of 100 (api.lookup_users can only accept 100 id's at a time) and use

followerIDs = grouper(followerIDs,100)
for followerGroup in followerIDs:
        followerGroup=filter(None, followerGroup)
        followers = api.lookup_users(followerGroup,wait_on_rate_limit = True)
        for follower in followers:
                print (originatingUser + ", " + str(follower.screen_name))

That gets a different error, namely:

 TypeError: lookup_users() got an unexpected keyword argument 'wait_on_rate_limit'

which I'm finding confusing, becuase the tweepy api suggests that that should be an accepted argument.

Any ideas as to what I'm doing wrong?

Cheers Ben.

Answer 1

I know this might be a little late, but here goes.

You pass the wait_on_rate_limit argument in the Cursor constructor, while the tweepy documentation states that it should be passed on the API() constructor.

Answer 2

The wait_on_rate_limit argument is to be passed in the API() constructor. In your code it would look like:

api = tweepy.API(auth,wait_on_rate_limit=True)

There's also another argument wait_on_rate_limit_notify, which informs you when tweepy is waiting for your rate limit to refresh. Adding both would finally make the line:

api = tweepy.API(auth,wait_on_rate_limit=True,wait_on_rate_limit_notify=True)