How To Obtain All Twitter Followers Without Hitting API Limit

Question

I imagine it's pretty easy to do, but I can't figure out what I'm doing wrong. I'm using Abraham's OAuth to gain access. I'm building a database with my follower's information: screen name, user name and twitter ID. Nothing too special.

I referenced Twitter's "cursoring" page, especially the pseudo code, to make my code. For those who don't want to click the link to see said pesudo code, it looks like the following:

cursor = -1

api_path = "https://api.twitter.com/1.1/endpoint.json?screen_name=targetUser"

do {

    url_with_cursor = api_path + "&cursor=" + cursor      

    response_dictionary = perform_http_get_request_for_url( url_with_cursor )

    cursor = response_dictionary[ 'next_cursor' ]

}

while ( cursor != 0 )

With every request, the end user gets a "cursor" which allows them to navigate through "pages" of results. Each page holds 20, and if you have 200 followers you have to go through 10 pages. I have over 900 followers. I modified it to look like the following:

 include('config.php');  //db connection
 include('twitter_oauth.php'); //oauth connection

 $followers = "";

$cursor = -1;
echo '<pre>';   
do {

    $consumerKey = 'xxx';
    $consumerSecret = 'xxx';
    $OAuthToken = 'xxx';
    $OAuthSecret = 'xxx';

    $tweet = new TwitterOAuth($consumerKey, $consumerSecret, $OAuthToken, $OAuthSecret);

    $followers = $tweet->get('followers/list', array('screen_name' => 'my_screen_name', 'cursor' => $cursor));

    print_r($followers);

    if (isset($followers->error)) {
        echo $followers->next_cursor_str;
        break;
    } 

    foreach($followers->users as $users) {

        $followersQ = mysql_query("SELECT * FROM followers WHERE tw_id = '".$users->id."'") or die(mysql_error());
        $num_rows = mysql_num_rows($followersQ);

        if ($num_rows == 0) {
            $followersQ2 = "INSERT INTO followers 
                                        (screen_name, name, tw_id)
                                        VALUES
                                        ('".$users->screen_name."', '".$users->name."', '".$users->id."')";
            $followersR = mysql_query($followersQ2) or die(mysql_error());
            echo 'done one set<br>';
        }

    }


    $cursor = $followers->next_cursor_str;

}

while ( $cursor != 0 );
echo '</pre>';

?>

The above code calls the twitter followers/list and gets the first 20 users. It then gets a cursor and goes to the next one, and repeats. Only, it seems after about 80 users it gives me the lovely:

[errors] => Array
    (
        [0] => stdClass Object
            (
                [message] => Rate limit exceeded
                [code] => 88
            )

    )

I could manually get the next cursor, wait 15 minutes for the rate limit to go down, call the function again with the cursor, get the next 80 items, then get that key and repeat, but I want to set up a script that can call it over and over.

I feel I'm doing something wrong, either with my function where I call oAuth, or outside of it somewhere. Can somebody point me in the right direction?

Thank you.

Answer 1

This a way faster, but there's a limitation concerns also :

1- make a request to get all followers ids ... paging with 5000 id in the page https://developer.twitter.com/en/docs/twitter-api/v1/accounts-and-users/follow-search-get-users/api-reference/get-followers-ids

2- loop on the ids and send each 100 id in a comma separated string to get their info https://developer.twitter.com/en/docs/twitter-api/v1/accounts-and-users/follow-search-get-users/api-reference/get-users-lookup

3- now u can get 1500 user object instead of 300 user object every 15 minutes

But you need also to set a timer every 15 request in case the followers list is more than 1500

Answer 2

I don't think there is any way round the limitations imposed. Even tweetbot has this limitation, as it's a limitation twitter impose. You could create a note in the database of the current status and set a cron job to run every 15 minutes which would run a group of requests again. It will take time, but it could notify you via email when it's finished. That's what services like socialbro do. You'd cache those results in your database of course.