TwitteR: 'searchTwitter' only returns a small set of tweets

Question

I am trying to retrieve ~3000 tweets with keyword "nba" or hashtag "#nba" using twitteR function 'searchTwitter' but it only returns 299 tweets for "nba" and 398 tweets for "#nba" between 2013-01-01 and 2014-02-25. I am really confused, is this normal? Has anyone else experienced similar problem using twitteR? Please help. Much appreciated!

library(twitteR)
library(plyr)
library(stringr)

load("~/twitter_authentication.Rdata")
registerTwitterOAuth(cred)

nbahash_tweets = searchTwitter("#nba",since='2013-01-01', until='2014-02-25',n=3000)

nba_tweets = searchTwitter("nba",since='2013-01-01', until='2014-02-25',n=3000)


Warning message:
In doRppAPICall("search/tweets", n, params = params, retryOnRateLimit = retryOnRateLimit,  :
  3000 tweets were requested but the API can only return 398

and then

Warning message:
In doRppAPICall("search/tweets", n, params = params, retryOnRateLimit = retryOnRateLimit,  :
  3000 tweets were requested but the API can only return 299

Answer 1

This is due to the limitations of the Search API of Twitter. In the FAQ of Twitter I found the following Question: Why are the Tweets I'm looking for not in Twitter Search, the Search API, or Search widgets?

There it's written:

Due to capacity constraints, the index currently only covers about a week's worth of tweets.

Hence, even if you want to download tweets starting from 1st of January 2013 you will only get tweets from at most one week ago. So the number of tweets won't be that big.

Furthermore, they say:

Our search service is not meant to be an exhaustive archive of public tweets and not all tweets are indexed or returned.

Therefore, there will be even less tweets available.

So, as far as I can see, your code should be OK and the small number of downloaded tweets is due to the Search API.