How to add proxy support to Jsoup?

Question

I am a beginner to Java and my first task is to parse some 10,000 URLs and extract some info out of it, for this I am using Jsoup and it's working fine.

But now I want to add proxy support to it. The proxies have a username and password too.

Answer 1

You can easily set proxy

System.setProperty("http.proxyHost", "192.168.5.1"); System.setProperty("http.proxyPort", "1080"); Document doc = Jsoup.connect("www.google.com").get(); 

Answer 2

Jsoup 1.9.1 and above: (recommended approach)

// Fetch url with proxy Document doc = Jsoup //                .connect("http://www.example.com/") //                .proxy("127.0.0.1", 8080) // sets a HTTP proxy                .userAgent("Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2") //                .header("Content-Language", "en-US") //                .get(); 

You may use also the overload Jsoup#proxy which takes a Proxy class (see below).

Before Jsoup 1.9.1: (verbose approach)

// Setup proxy Proxy proxy = new Proxy(                                      //         Proxy.Type.HTTP,                                      //         InetSocketAddress.createUnresolved("127.0.0.1", 8080) // );  // Fetch url with proxy Document doc = Jsoup //                .connect("http://www.example.com/") //                .proxy(proxy) //                .userAgent("Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2") //                .header("Content-Language", "en-US") //                .get(); 

References:

• Connection#proxy(String,int)
• Connection#proxy(Proxy)
• Proxy class