Java 6 NTLM proxy authentication and HTTPS - has anyone got it to work?

Question

I have a Java application (not an applet) that needs to access a web service. Proxies for the web service have been generated with JAX-WS, and seem to work fine. In one scenario it needs to talk through a web proxy server (actually Squid 3.0), which is set to require NTLM authentication.

Running on Sun's JRE 1.6.0_14, everything works fine for accessing HTTP URLs, without requiring any changes: the built-in NTLM authenticator kicks in and it all works seemlessly. If, however, the web service URL is a HTTPS URL, the web service call fails deep inside Sun's code:

com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: java.lang.NullPointerException
        at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:121)
        at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:142)
        at com.sun.xml.internal.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:83)
        at com.sun.xml.internal.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:105)
        at com.sun.xml.internal.ws.api.pipe.Fiber.__doRun(Fiber.java:587)
        at com.sun.xml.internal.ws.api.pipe.Fiber._doRun(Fiber.java:546)
        at com.sun.xml.internal.ws.api.pipe.Fiber.doRun(Fiber.java:531)
        at com.sun.xml.internal.ws.api.pipe.Fiber.runSync(Fiber.java:428)
        at com.sun.xml.internal.ws.client.Stub.process(Stub.java:211)
        at com.sun.xml.internal.ws.client.sei.SEIStub.doProcess(SEIStub.java:124)
        at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:98)
        at com.sun.xml.internal.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:78)
        at com.sun.xml.internal.ws.client.sei.SEIStub.invoke(SEIStub.java:107)
        ... our web service call ...
Caused by: java.lang.NullPointerException
        at sun.net.www.protocol.http.NTLMAuthentication.setHeaders(NTLMAuthentication.java:175)
        at sun.net.www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:1487)
        at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:164)
        at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:896)
        at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:230)
        at com.sun.xml.internal.ws.transport.http.client.HttpClientTransport.getOutput(HttpClientTransport.java:109)
        ... 16 more

Looking in Sun's bug database turns up a few exceptions in such classes, but all of them seem to have been fixed. Has anyone come across anything like this? Has anyone got this to work?

Answer 1

After some debugging, this seems to be a flaw in the JRE class libraries, specifically in sun.net.www.protocol.http.HttpURLConnection.

Studying the HTTP requests and responses in the cases of HTTP and HTTPS endpoints showed that, in the successful HTTP case, the requests had a header Proxy-Connection=keep-alive, which was missing on the failing HTTPS case. Reading more generally, there seems to be some confusion on whether one should use "Proxy-Connection" or just "Connection", too ...

Anyway, it is notable that in the HTTP case, the code goes through HttpURLConnection.writeRequests(), which contains the following code snippet

    /*
     * For HTTP/1.1 the default behavior is to keep connections alive.
     * However, we may be talking to a 1.0 server so we should set
     * keep-alive just in case, except if we have encountered an error
     * or if keep alive is disabled via a system property
     */

    // Try keep-alive only on first attempt
    if (!failedOnce && http.getHttpKeepAliveSet()) {
    if (http.usingProxy) {
        requests.setIfNotSet("Proxy-Connection", "keep-alive");
    } else {
        requests.setIfNotSet("Connection", "keep-alive");
    }

There's no such code when creating a tunnel through the proxy for HTTPS, which causes Squid to get upset during the NTLM authentication conversation.

To work around this, in HttpURLConnection.sendCONNECTRequest(), I added

if (http.getHttpKeepAliveSet()) {
    if (http.usingProxy) {
        requests.setIfNotSet("Proxy-Connection", "keep-alive");
    }
}

just before

setPreemptiveProxyAuthentication(requests);
http.writeRequests(requests, null);

I inject my modified HttpURLConnection.class into the JRE using the "-Xbootclasspath/p" flag, and now it works! Not exactly elegant, but there we are.