403 response with HttpClient but not with browser

Question

I’m having a problem with the HttpClient library in java.

The target web site is on SSL (https://www.betcris.com), and I can load the index page from that site just fine .

However, the different pages showing odds for the different sports returns a 403 response code with HttpClient, but loading the same pages in a browser works just fine.

Here is such a page : https://www.betcris.com/en/live-lines/soccer.

I started troubleshooting this page with the information gathered by HttpFox (a Firefox add-on that resembles LiveHttpHeaders), making sure I had all the correct request headers and cookies, but I couldn’t get it to load using HttpClient. I also determined that cookies have nothing to do with the problem, as I can remove all cookies for that web site within my browser, and then hit the page directly and it will load.

I confirmed that there’s something special going on with these pages by using the online tool at http://www.therightapi.com/test. This tool allows you to input the url of a page along with any Request header you want, and shows you the response you get from the target web site. Using that tool, I can load https://www.google.com just fine, but I get the same 403 error when trying to load https://www.betcris.com/en/live-lines/soccer.

Here's my setup at therightapi :

And the response :

Does anyone know what’s going on here ?

Thanks.

EDIT : I've created a test project, here's the java code, followed by the maven dependency you should have in your pom :

package com.yourpackage;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;

import org.apache.http.HttpResponse;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.HttpClient;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.HttpClientBuilder;

public class TestHttpClient {
    public static void main(String[] args) {
        String url = "https://www.betcris.com/en/live-lines/soccer";

        HttpClient client = HttpClientBuilder.create().build();
        HttpGet request = new HttpGet(url);

        // add request header
        request.addHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:55.0) Gecko/20100101 Firefox/55.0");
        try {
            HttpResponse response = client.execute(request);
            System.out.println("Response Code : "
                    + response.getStatusLine().getStatusCode());

            BufferedReader rd = new BufferedReader(
                    new InputStreamReader(response.getEntity().getContent()));

            StringBuffer result = new StringBuffer();
            String line = "";
            while ((line = rd.readLine()) != null) {
                result.append(line);
            }
        } catch (ClientProtocolException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

    }
}



    <!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient -->
<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.5.3</version>
</dependency>

Answer 1

I have solved this problem (avoiding 403) by setting up User-Agent property while making a request as like follow:

• If you use HttpClient

  HttpGet httpGet = new HttpGet(URL_HERE);
  httpGet.setHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) 
      AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
  

• If you use HttpURLConnection

  HttpURLConnection connection = (HttpURLConnection) url.openConnection();    
  connection.setRequestProperty("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) 
      AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");
  

Answer 2

I use the following code to consume HTTPS Urls:

import org.apache.http.HttpResponse;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.conn.ssl.NoopHostnameVerifier;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.ssl.SSLContextBuilder;

...

    SSLContext sslContext =
            new SSLContextBuilder().loadTrustMaterial(null, (certificate, authType) -> true).build();

    try (CloseableHttpClient httpClient = HttpClients.custom().setSSLContext(sslContext)
                                                     .setSSLHostnameVerifier(new NoopHostnameVerifier()).build()) {
        HttpGet httpGet = new HttpGet("YOUR_HTTPS_URL");
        httpGet.setHeader("Accept", "application/xml");
        httpGet.setHeader("User-Agent",
                          "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.95 Safari/537.11");

        HttpResponse response = httpClient.execute(httpGet);

        logger.info("Response: " + response);

    }

pom.xml:

 <dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.5.3</version>
 </dependency>