TLSv1 handshake failure

Question

(Disclaimer: I am by no stretch of the imagination a security expert nor a windows expert for that matter)

Setup:

• server on our end: java 1.6 (already added bouncycastle to the security file) on windows 2003 server
• third party client: windows 2008 server with biztalk
• all renegotiation system properties introduced due to the renegotiation attack are "enabled" on the server side (not safe I know)

Ideally we want to fix this at our end but it is possible to propose a fix to the client if necessary.

The client server has to connect to our server over a HTTPS connection but it always fails, wireshark shows the following conversation:

> TLSv1: Client Hello
< TLSv1: Alert (21): Unexpected Message

As per the RFC (http://www.ietf.org/rfc/rfc2246.txt) the alert(21) refers to a failed decryption and from what I can see in wireshark, none of the ciphers proposed by the client are actually supported by JRE 1.6 (as per http://docs.oracle.com/javase/6/docs/technotes/guides/security/SunProviders.html#SupportedCipherSuites) In an effort to reproduce the error to be able to examine it closer, I tested with some other software:

• wfetch on windows xp with "https" selected will perform the initial client handshake in SSLv2, the server will switch to TLSv1 to answer, this works
• wfetch on windows xp with configured to use "TLSv1" for the initial handshake will fail in the same way as the biztalk server
• wfetch on windows 2008 with configured "https" will use "TLSv1" for the initial handshake and fail in the same way as the biztalk server
• IE (on windows xp) will initially try a TLSv1 handshake with the same failed result but immediately tries again using SSLv3 which works (at this point I figure all microsoft software uses a central configuration available at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel)
• firefox uses SSLv3 for the entire conversation, so no problem there
• OpenSSL performs an initial handshake in SSLv2, and the server switches to TLSv1 when it answers, no problem there
• OpenSSL can be forced to do the initial handshake in TLSv1 as well, it offers a list of 27 ciphers (as opposed to the 11 ciphers proposed by windows-based software) and can connect without a problem

To my untrained eye this reinforces the idea that an incompatible cipher proposition is the root cause where windows only supports cipher suites that are not supported by JVM (for TLSv1). I have installed bouncy castle as an additional provider in the java.security file to no avail. I have searched high and low and only found a reference that maybe websphere supports the windows ciphers for TLSv1 but no way of downloading a standalone provider to test it. JRE 1.7 is not supported by the software we run on our JVM, so upgrading is not an option (perhaps the security provider can be downgraded safely? I haven't found a download for it yet though) I have found no way to add a cipher to windows short of writing c++ code (I've played around with the above mentioned registry settings without effect).

So in conclusion I wonder if one of the following things would fix it and how they should be accomplished:

• add a provider to the jvm that can work with the ciphers for TLSv1 that are proposed by windows
• somehow force the client to do the initial handshake in SSLv3 (preferably not SSLv2) or at least retry if the TLSv1 handshake fails
• somehow add a JVM-supported cipher for TLSv1 to the client windows

Any other solutions are of course also appreciated.

EDIT

The Java version is Java version (64 bit): 1.6.0_19-b04.

The list of proposed ciphers is:

• TLS_RSA_WITH_RC4_128_MD5
• TLS_RSA_WITH_RC4_128_SHA
• TLS_RSA_WITH_3DES_EDE_CBC_SHA
• TLS_RSA_WITH_DES_CBC_SHA
• TLS_RSA_EXPORT1024_WITH_RC4_56_SHA
• TLS_RSA_EXPORT1024_WITH_DES_CBC_SHA
• TLS_RSA_EXPORT_WITH_RC4_40_MD5
• TLS_RSA_EXPORT_WITH_RC2_CBC_40_MD5
• TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
• TLS_DHE_DSS_WITH_DES_CBC_SHA
• TLS_DHE_DSS_EXPORT1024_WITH_DES_CBC_SHA

The unlimited strength cryptography policy files are installed. I have tried to set javax.net.debug=all and started the server from the console, no additional output appeared. I have set sun.security.ssl.allowUnsafeRenegotiation=true to no avail.

EDIT 2

It turns out the software we are using uses a custom stack for HTTPs instead of the default. A fix was issued which seems to solve the problem though I don't know exactly which part of the TLS request triggered the error (seeing as most TLSv1 handshakes did succeed).

Thanks for the feedback, it has been an interesting if futile search. Live and learn.

Answer 1

It turns out the software we are using uses a custom stack for HTTPs instead of the default. A fix was issued which seems to solve the problem though I don't know exactly which part of the TLS request triggered the error (seeing as most TLSv1 handshakes did succeed).

Thanks for the feedback, it has been an interesting if futile search. Live and learn.