403 error while getting the google result using jsoup [duplicate]

Question

I'm trying to get Google results using the following code:

Document doc = con.connect("http://www.google.com/search?q=lakshman").timeout(5000).get();

But I get this exception:

org.jsoup.HttpStatusException: HTTP error fetching URL. Status=403,URL=http://www.google.com/search?q=lakshman

A 403 error means the server is forbidding access, but I can load this URL in a web browser just fine. Why does Jsoup get a 403 error?

Answer 1

You just need to add the UserAgent property to HTTP header as follows:

Jsoup.connect(itemUrl)
     .userAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/33.0.1750.152 Safari/537.36")
     .get()

Answer 2

Google doesn't allow robots, you couldn't use jsoup to connect google. You can use the Google Web Search API (Deprecated) but the number of requests you may make per day will be limited.

Answer 3

Actually, you can evade 403 error by just adding a user-agent

doc = Jsoup.connect(url).timeout(timeout)
                    .userAgent("Mozilla")

But that is against the google policy I think.

EDIT: Google catches robots quicker than you think. You can however, use this as a temporary solution.