What might cause a CircularRedirectException when accessing a password-protected URL with Apache DefaultHttpClient?

Question

I am trying to access a page that requires authentication. So I passed my username and password in the code. And this is the below output and error I am getting. First of all it execute the request http://me.somehost.com/ and I get the error as Caused by: org.apache.http.client.CircularRedirectException: Circular redirect to. But afterwards it tries to access http://me.somehost.com/robots.txt and I get the response back from the server since it authenticates my username and password. The response I am getting is the actual response If I type my username and password in to the browser with that link.. Then why it is happening with this link http://me.somehost.com/

----------------------------------------
executing requestGET http://qhome.somehost.com/ HTTP/1.1
org.apache.http.client.ClientProtocolException
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:822)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:754)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:732)
    at edu.uci.ics.crawler4j.url.WebURL.setURL(WebURL.java:113)
    at edu.uci.ics.crawler4j.crawler.CrawlController.addSeed(CrawlController.java:207)
    at edu.uci.ics.crawler4j.example.advanced.Controller.main(Controller.java:31)
Caused by: org.apache.http.client.CircularRedirectException: Circular redirect to 'http://me.somehost.com/net/pages/Home.xhtml'
    at org.apache.http.impl.client.DefaultRedirectStrategy.getLocationURI(DefaultRedirectStrategy.java:168)
    at org.apache.http.impl.client.DefaultRedirectStrategy.getRedirect(DefaultRedirectStrategy.java:193)
    at org.apache.http.impl.client.DefaultRequestDirector.handleResponse(DefaultRequestDirector.java:1021)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:482)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
    ... 5 more
----------------------------------------
executing requestGET http://me.somehost.com/robots.txt HTTP/1.1
HTTP/1.1 200 OK
# TS Application Portfolio: http://cm.somehost.com/cm/
# TS Email ID: [email protected]
# ITOC Qwiki TS Apps Section:  http://ki.somehost.com/itall/ITOC_Esion#QBAT-TS
User-agent: *
Disallow: /departments/
Disallow: /Mnet/pages/
Disallow: /Mnet/themes/
Disallow: /wps/
 INFO [main] Number of pages fetched per second: 0
----------------------------------------
executing requestGET https://login.somehost.com/siteminderagent/64219/smgetcred.scc?TYPE=16777217&REALM=-SM-somehost%202B7NS3b0k0Fk&TARGET=-SM-http%3a%2f%2fqhome%2esomehost%2ecom%2frobots%2etxt HTTP/1.1
HTTP/1.1 200 OK
# TS Application Portfolio: http://cm.somehost.com/cm/
# TS Email ID: [email protected]
# ITOC wiki TS Apps Section:  http://ki.somehost.com/itall/ITOC_Escalation#QBAT-TS
User-agent: *
Disallow: /departments/
Disallow: /net/pages/
Disallow: /net/themes/
Disallow: /wps/

And this is my authentication code..

DefaultHttpClient client = null;

        try
        {
            // Set url
            //URI uri = new URI(url.toString());

            client = new DefaultHttpClient();

            client.getCredentialsProvider().setCredentials(
                    new AuthScope(AuthScope.ANY_HOST, AuthScope.ANY_PORT, AuthScope.ANY_REALM),
                    new UsernamePasswordCredentials("test", "test"));

            // Set timeout
            //client.getParams().setParameter(CoreConnectionPNames.SO_TIMEOUT, 5000);

            URL url1 = new URL (url);
            HttpURLConnection connection = (HttpURLConnection) url1.openConnection();
            connection.setFollowRedirects(true);
            HttpGet request = new HttpGet(url);

            System.out.println("----------------------------------------");
            System.out.println("executing request" + request.getRequestLine());
            HttpResponse response = client.execute(request);
            HttpEntity entity = response.getEntity();


            System.out.println(response.getStatusLine());





                    InputStream content = entity.getContent();
                    BufferedReader in   = 
                        new BufferedReader (new InputStreamReader (content));
                    String line;
                    while ((line = in.readLine()) != null) {
                        System.out.println(line);
                    }
                } catch(Exception e) {
                    e.printStackTrace();
                }

What wrong I am doing. Since it accepts the username and password for one link and throws the error for the second link.. Any suggestions will be appreciated...

Answer 1

This question may address the very same issue. Have you tried hitting the same page in a regular browser, while monitoring the requests and responses with a tool like Firebug (my personal favorite).

There may actually not be an issue at all. If that's the case, then you can set ALLOW_CIRCULAR_REDIRECTS in the client params.

UPDATE:

To allow for circular redirects, your code would look like this...

...
HttpClient client = new DefaultHttpClient();
client.getParams().setParameter(ClientPNames.ALLOW_CIRCULAR_REDIRECTS, true);
...