How to troubleshoot TLSv1 handshake problem with mutual authentication?

Question

I am having a heck of a time getting my web service client talking to my web service which is protected by requiring client certificates. I am using JAX-WS 2.1 and the web service request first goes through IIS before being forwarded on to JBoss following authentication.

I am using a self-signed certificate for the client certificate and it is installed on the Windows 2003 server in its "Trusted Root Certification Authorities" section.

If I try to access the WSDL for the service with Internet Explorer, I am prompted for a certificate to send and I choose the one which I created and everything seems to work fine. This leads me to believe that all of the certificates are all right and trusted by all the right "people".

Below I can see that the server is including a reference to my "happyFunBall" as an authority that it trusts because of its inclusion in the CertificateRequest portion of the handshake :

*** CertificateRequest
Cert Types: RSA, DSS`
Cert Authorities:
   ...
<CN=Symantec Root CA, O=Symantec Corporation>
<CN=DoD Root CA 2, OU=PKI, OU=DoD, O=U.S. Government, C=US>
<CN=Microsoft Root Authority, OU=Microsoft Corporation, OU=Copyright (c) 1997 Microsoft Corp.>
<CN=happyFunBall, O=blah.blah.com, C=US>
<CN=DoD PKI Med Root CA, OU=PKI, OU=DoD, O=U.S. Government, C=US>
<CN=ECA Root CA 2, OU=ECA, O=U.S. Government, C=US>
<CN=Symantec Root 2005 CA, O=Symantec Corporation, C=US>
<CN=Microsoft Root Certificate Authority, DC=microsoft, DC=com>
   ...
*** ServerHelloDone

I'm pretty much a novice at this stuff, so I might be missing some very elementary stuff and I might not include all of the relevant information... Anyway, I generated my client certificate using keytool and it is of type "PKCS12". So when I launch my client, I have the following system properties defined:

javax.net.ssl.keyStore="C:\placeWhereFunBallLives\funBall.p12"
javax.net.ssl.keyStorePassword=<password>
javax.net.ssl.keyStoreType=PKCS12

I receive a 403 from the server when I make the call to the web service. It appears to me that the underlying TLS/SSL implementation is not finding the client certificate to send or for some reason is not sending it even if it does try to find it.

What should I see upon a successful handshake? After the above chunk there is this:

*** Certificate chain
***
*** ClientKeyExchange, RSA PreMasterSecret, TLSv1
main, WRITE: TLSv1 Handshake, length = 157
SESSION KEYGEN:
PreMaster Secret:
0000: 03 01 37 1B 02 E4 DB 34   87 A4 4C D0 03 83 74 0B  ..7....4..L...t.
0010: 8D 31 A0 B1 70 B8 31 F8   EB 72 AB 88 3B 69 B5 43  .1..p.1..r..;i.C
0020: 19 EA 24 BD 59 50 16 7D   C0 99 DC A6 EC 4F EF 64  ..$.YP.......O.d
CONNECTION KEYGEN:
Client Nonce:
0000: 4B 56 1E E4 66 09 1D 6C   EE 95 F1 47 3E 12 DA 22  KV..f..l...G>.."
0010: 63 8E 23 93 76 7D FB CB   27 7B 2E C5 8E DC 93 C2  c.#.v...'.......
Server Nonce:
0000: 4B 56 1E E4 A7 70 0F 2F   1A 17 0D 8F 2D 79 7D AE  KV...p./....-y..
0010: 70 0E C9 5C 16 A9 B5 25   B0 99 22 B3 F8 89 D8 EC  p..\...%..".....
Master Secret:
0000: C3 ED 84 D6 63 CD 6C 59   94 14 06 4B 37 EC EE C4  ....c.lY...K7...
0010: AE 99 97 1B 0E B9 61 25   AF DB C4 54 30 C5 4C 80  ......a%...T0.L.
0020: 47 74 47 E8 B0 B5 13 32   BA 93 62 33 B6 CA C4 A8  GtG....2..b3....
Client MAC write Secret:
0000: 3C E8 3A 6A B2 74 F0 ED   6C FE DE 70 77 E8 EB 36  <.:j.t..l..pw..6
Server MAC write Secret:
0000: BD 41 C5 EB 3B ED E9 E0   0C 61 28 C2 11 7A 75 1C  .A..;....a(..zu.
Client write key:
0000: 79 43 DD AD 44 B0 A0 61   1D EB 71 AB 4F 39 9D EF  yC..D..a..q.O9..
Server write key:
0000: C9 43 22 2A 48 50 FA 67   E0 01 1B 8A 48 0F C8 CF  .C"*HP.g....H...
... no IV used for this cipher
main, WRITE: TLSv1 Change Cipher Spec, length = 17
*** Finished
verify_data:  { 52, 94, 173, 217, 26, 70, 12, 243, 6, 71, 27, 163 }
***
main, WRITE: TLSv1 Handshake, length = 32
main, READ: TLSv1 Change Cipher Spec, length = 17
main, READ: TLSv1 Handshake, length = 32
*** Finished
verify_data:  { 56, 254, 211, 144, 48, 35, 4, 235, 65, 127, 237, 94 }
***
%% Cached client session: [Session-2, SSL_RSA_WITH_RC4_128_MD5]

Heeeeeeelp!

Answer 1

Ok, so I found out what the problem was (or at least a symptom of the problem) and thought I would share it here for historical-type purposes.

I used keytool thusly to generate my client keystore:

keytool -genkeypair -alias funBall -keyalg rsa -dname "CN=happyFunBall,O=thing.thing.com,C=US" -keystore <keystoreLocation>/funBall.p12 -keypass <password> -storetype PKCS12 -storepass <password>

and then I signed it with this keytool command:

keytool -selfcert -alias funBall -validity 1825 -keystore  <keystoreLocation>/funBall.p12 -keypass <password> -storetype PKCS12 -storepass <password>

Apparently, this is no good. The above "-selfcert" command needed a "-dname" parameter set to the same value as that used in the "-genkeypair", because otherwise the certificate chain or whatever (which I don't fully understand) is not correct and the certificate will not be recognized as a valid one for the list given by the server in its CertificateRequest.