Issue with fetching tweets from timeline as Twitter doesnt return 'next_results'

Question

I am trying to fetch tweets from Twitter timeline using the max_id field present in next_results as demonstrated here.(Twython search API with next_results)

For most of the queries I get the desired number(500+) of tweets but for some I get less than 200 tweets.

When looking into what is returned in 'next_results I found that after returning just 100 tweets, the response has next_results field missing. There cannot be just 200 tweets for the queried topic as I am searching for this topic when it is trending and on Twitter I can see more than 200 tweets.

Is anyone else also facing this issue. Is there any workaround ?

Below is the output where in the second response the 'next_results' field is missing.

TEST_PROGRAM >> Calling twitter to get tweets for  #TheWorstLieEver

{u'count': 100, u'completed_in': 0.066, u'max_id_str': u'432942833725497345', u'since_id_str': u'0', **u'next_results': u'?max_id=432540545630494719&q=%23TheWorstLieEver&count=100&include
_entities=1&result_type=mixed'**, u'refresh_url': u'?since_id=432942833725497345&q=%23TheWorstLieEver&result_type=mixed&include_entities=1', u'since_id': 0, u'query': u'%23TheWorstLieEver
', u'max_id': 432942833725497345L}

{u'count': 100, u'completed_in': 0.111, u'max_id_str': u'432540545630494719', u'since_id_str': u'0', u'refresh_url': u'?since_id=432540545630494719&q=%23TheWorstLieEver&result_type=mixe
d&include_entities=1', u'since_id': 0, u'query': u'%23TheWorstLieEver', u'max_id': 432540545630494719L}

Answer 1

I got the answer. If you are looking for volume, you need to use Streaming API instead. Twitter search API focuses on relevance of query and hence might not return all tweets. Twitter documentation says Before getting involved, it's important to know that the Search API is focused on relevance and not completeness. This means that some Tweets and users may be missing from search results. If you want to match for completeness you should consider using a Streaming API instead.