How to get more than 100 results from Google Custom Search API

Question

I am trying to use Google Custom Search API for research purposes in Java. As a result I'm gonna need a big result set for each query. However it seems that I'm limited with first 100 results which is much less than what I need. I use the list method like this:

list.setStart(90L);

And when I set it to start from index 100 I get this error:

com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request

{   "code" : 400,   
    "errors" : [ {
    "domain" : "global",
    "message" : "Invalid Value",
    "reason" : "invalid"   
  } ],   
    "message" : "Invalid Value" 
}

Is there any way to remove this limitation?

Answer 1

There's a difference between two limitations: a) max 100 queries per day; and b) max 100 results per searched phrase (even split into 10 queries of 10 results per query). The limitation (a) can be solved by paying to Google, whereas limitation (b) cannot be solved at all, it seems to be a feature of the CSE product when searching the whole web (i.e. not just a site search, which can give more results), see e.g. https://productforums.google.com/forum/#!topic/customsearch/VM8_6trCxGU