Java HTTPUrlConnection timeout does not work

Question

I've written a programm that opens a httpurlconnection to a website through random proxies. My httpurlconnection is called conn. Now I know, that some of those proxies might be too slow, so i've set the timeout of the connection to 40000 milliseconds with conn.setConnectTimeout(40000) and conn.setReadTimeout(40000).

After doing so, i got this code:

long diff = 0;
    long starttime = 0;
    long endtime = 0;

    try
    {
        starttime = System.currentTimeMillis();
        conn.connect();

        endtime = System.currentTimeMillis();

        diff = endtime - starttime;

        if (endtime <= starttime + conn.getConnectTimeout())
        {
            //Trying to read sourecode
            InputStreamReader isrConn = new InputStreamReader(conn.getInputStream());
            BufferedReader brConn = new BufferedReader(isrConn);

            line = brConn.readLine();

            while (line != null)
            {

                response += line + "\t";
                try
                {
                    line = brConn.readLine();
                } catch (IOException e)
                {
                    printError("Reading sourcecode failed.");
                }

            }
        }
        else
        {
            response = "blabla.";
        }




    // If conn.connect failed   
    } catch (IOException e)
    {
        endtime = System.currentTimeMillis();
        diff = endtime - starttime;

        response = "Message: "+e.getMessage() +" MyTimeout:"+ conn.getConnectTimeout() +" Actual time passed:  "+ diff;
               e.printStackTrace();


    }

There are reasons why the connection could fail, so in many cases i get to the last catch-block and get the following output:

---

Message: Connection timed out: connect MyTimeout:40000 Actual time passed: 21012

Message: Connection timed out: connect MyTimeout:40000 Actual time passed: 21016

Message: Connection timed out: connect MyTimeout:40000 Actual time passed: 21010

Message: Connection timed out: connect MyTimeout:40000 Actual time passed: 21009

---

So my question would be: I have set the timeout to 40000 milliseconds, but i get a "Connection timed out"-response after about 21000 milliseconds, does any of you know why that is?

EDIT: im using windows 7 and i now added the e.printStackTrace() to the catch-block, like told in the comments. thanks so far. the output now is (example):

java.net.ConnectException: Connection timed out: connect
    at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
    at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
    at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
    at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
    at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
    at java.net.PlainSocketImpl.connect(Unknown Source)
    at java.net.Socket.connect(Unknown Source)
    at sun.net.NetworkClient.doConnect(Unknown Source)
    at sun.net.www.http.HttpClient.openServer(Unknown Source)
    at sun.net.www.http.HttpClient$1.run(Unknown Source)
    at sun.net.www.http.HttpClient$1.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.net.www.http.HttpClient.privilegedOpenServer(Unknown Source)
    at sun.net.www.http.HttpClient.openServer(Unknown Source)
    at sun.net.www.http.HttpClient.<init>(Unknown Source)
    at sun.net.www.http.HttpClient.New(Unknown Source)
    at sun.net.www.http.HttpClient.New(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
    at sun.net.www.protocol.http.HttpURLConnection.connect(Unknown Source)
    at TestThread.getSourcePage(TestThread.java:361)
    at TestThread.aChecker(TestThread.java:216)
    at TestThread.getNextProxy(TestThread.java:169)
    at TestThread.getNextC(TestThread.java:157)
    at TestThread.aChecker(TestThread.java:273)
    at TestThread.getNextProxy(TestThread.java:169)
    at TestThread.aChecker(TestThread.java:295)
    at TestThread.getNextProxy(TestThread.java:169)
    at TestThread.getNextC(TestThread.java:157)
    at TestThread.run(TestThread.java:103)
    at java.lang.Thread.run(Unknown Source)

Message: Connection timed out: connect MyTimeout:40000 Actual time passed:  21015

Answer 1

Look at the exception you got:
The biggest clue: You are getting java.net.ConnectException As per javadoc, java.net.ConnectException signifies that connection was refused remotely for reasons such as no process is listening on the port.

public class ConnectException
extends SocketException

Signals that an error occurred while attempting to connect a socket to a remote address and port. 
Typically, the connection was refused remotely (e.g., no process is listening on the remote 
address/port)

What you configured in HttpUrlConnection:
The timeout for connection (given that the remote port accepts connection). If the connection timeout expires, you get a java.net.SocketTimeoutException and not a java.net.ConnectException.

So, what is causing java.net.ConnectException?
I tried the following test cases:

   +------------+------------+----------------+------------------+---------------------------------+ 
   | Valid Host | Valid Port | Valid Proxy IP | Valid Proxy Port | Exception                       | 
   +------------+------------+----------------+------------------+---------------------------------+ 
#1 | yes        | yes        | -NA-           | -NA-             | -- none --                      |
#2 | yes        | no         | -NA-           | -NA-             | java.net.ConnectException       |
   +------------+------------+----------------+------------------+---------------------------------+ 
#3 | yes        | yes        | yes            | yes              | -- none --                      |
#4 | yes        | no         | yes            | yes              | java.net.SocketTimeoutException |
#5 | yes        | yes        | yes            | no               | java.net.ConnectException       |
   +------------+------------+----------------+------------------+---------------------------------+ 

• Case #1, #3 are the happy paths in which all configs are correct
• In case #4, we get a java.net.SocketTimeoutException because java process is able to establish connection (to the proxy port), but does not get any data to read as target host's port number is invalid
• In case #2, #5 we get java.net.ConnectException because the port at which java process attempts to write/read is invalid
• The connection timeout value does not hold good for cases where no process is listening on the port at which java process is attempting to connect. That's why you get ConnectException before timeout expires

Message: Connection refused: connect MyTimeout:10000 Actual time passed: 6549 java.net.ConnectException: Connection refused: connect at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method) at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85) .... ....

Conclusion:

• Some of the proxies you were trying to connect to, must be down. Hence java process threw java.net.ConnectException
• It's better to catch java.net.ConnectException and mark the proxy as invalid/down