SSL Client Authentication with smart card works in Java 6 but fails in Java 7

Question

the following code creates a client authenticated SSL context using PKCS#11 device (smart card). It all works great with Java 6:

// Configure the SunPkcs11 provider
String pkcs11config;
pkcs11config = "name = Cryptoki";
pkcs11config += "\nlibrary = /SCDriver/libbit4ipki.dylib";
InputStream confStream = new ByteArrayInputStream(pkcs11config.getBytes());
SunPKCS11 sunpkcs11 = new SunPKCS11(confStream);
Security.addProvider(sunpkcs11);

// Specify keystore builder parameters for PKCS#11 keystores
Builder scBuilder = Builder.newInstance("PKCS11", sunpkcs11, new KeyStore.CallbackHandlerProtection(new PasswordRetriever()));

// Create and init KeyManagerFactory
KeyManagerFactory factory = KeyManagerFactory.getInstance("NewSunX509");
factory.init(new KeyStoreBuilderParameters(scBuilder));

// create and init ssl context
m_ssl_context = SSLContext.getInstance("TLS");
m_ssl_context.init(factory.getKeyManagers(), new TrustManager[] {new PkTrustManager()}, null);      
SSLContext.setDefault(m_ssl_context);

The PkTrustManager is simply and 'empty' class, taking any server/client certificate for good, and PasswordRetriever just asks for password through a dialog box (By request I post source code for these). On Java 7 instead I get the following exception during SSL handshake of the ssl context:

java.security.InvalidKeyException: Class does not represent an RSA key: sun.security.pkcs11.P11Key$P11PrivateKey
    at iaik.pkcs.pkcs1.RSACipher.engineInit(Unknown Source)
    at iaik.pkcs.pkcs1.RSACipher.engineInit(Unknown Source)
    at iaik.security.rsa.RSA.init(Unknown Source)
    at iaik.security.rsa.RawRSASignature.engineInitSign(Unknown Source)
    at java.security.SignatureSpi.engineInitSign(SignatureSpi.java:103)
    at java.security.Signature.initSign(Signature.java:529)
    at sun.security.ssl.RSASignature.engineInitSign(RSASignature.java:125)
    at java.security.Signature$Delegate.engineInitSign(Signature.java:1136)
    at java.security.Signature.initSign(Signature.java:529)
    at sun.security.ssl.HandshakeMessage$CertificateVerify.<init>(HandshakeMessage.java:1556)
    at sun.security.ssl.ClientHandshaker.serverHelloDone(ClientHandshaker.java:982)
    ... 14 more

In the best case, I'd say something has changed in Java internals, but checking the Oracle documentation, I didn't find any supposed changes on the NewSunX509 key manager, or other components. I double checked the code and it seems compliant to the specifications (through there's for sure something I missed!).

I tried to add the configuration flags:

System.setProperty("javax.net.ssl.keyStoreType", "pkcs11");
System.setProperty("javax.net.ssl.keyStore", "NONE");
System.setProperty("javax.net.ssl.trustStoreType", "pkcs11");
System.setProperty("javax.net.ssl.trustStore", "NONE");
System.setProperty("javax.net.ssl.keyStoreProvider", sunpkcs11.getName() );
JCEMapper.setProviderId(sunpkcs11.getName());

But no change, same error... and they are not required in Java 6 where all works. Thanks in advance anyone can help or has any ideas!

PS: By request from @owlstead added -Djava.security.debug=sunpkcs11 and got the following output:

SunPKCS11 loading ---DummyConfig-1---
sunpkcs11: Initializing PKCS#11 library /SCDriver/libbit4ipki.dylib
Information for provider SunPKCS11-Cryptoki
Library info:
  cryptokiVersion: 2.20
  manufacturerID: bit4id srl                      
  flags: 0
  libraryDescription: bit4id PKCS#11                  
  libraryVersion: 1.02
All slots: 0
Slots with tokens: 0
Slot info for slot 0:
  slotDescription: bit4id miniLector-U38 00 00                                     
  manufacturerID: unknown                         
  flags: CKF_TOKEN_PRESENT | CKF_REMOVABLE_DEVICE | CKF_HW_SLOT
  hardwareVersion: 0.00
  firmwareVersion: 0.00
Token info for token in slot 0:
  label: CNS                             
  manufacturerID: ST Incard                       
  model: CNS (LB)        
  serialNumber: 7420057800291590
  flags: CKF_RNG | CKF_LOGIN_REQUIRED | CKF_USER_PIN_INITIALIZED | CKF_TOKEN_INITIALIZED
  ulMaxSessionCount: CK_EFFECTIVELY_INFINITE
  ulSessionCount: 0
  ulMaxRwSessionCount: CK_EFFECTIVELY_INFINITE
  ulRwSessionCount: CK_UNAVAILABLE_INFORMATION
  ulMaxPinLen: 8
  ulMinPinLen: 5
  ulTotalPublicMemory: 31988
  ulFreePublicMemory: CK_UNAVAILABLE_INFORMATION
  ulTotalPrivateMemory: 780
  ulFreePrivateMemory: CK_UNAVAILABLE_INFORMATION
  hardwareVersion: 0.00
  firmwareVersion: 0.00
  utcTime: 0000000000000000
Mechanism CKM_RSA_PKCS:
  ulMinKeySize: 1024
  ulMaxKeySize: 1024
  flags: 2561 = CKF_HW | CKF_DECRYPT | CKF_SIGN
Mechanism CKM_RSA_PKCS_KEY_PAIR_GEN:
  ulMinKeySize: 1024
  ulMaxKeySize: 1024
  flags: 65537 = CKF_HW | CKF_GENERATE_KEY_PAIR
Mechanism CKM_SHA1_RSA_PKCS:
  ulMinKeySize: 1024
  ulMaxKeySize: 1024
  flags: 2049 = CKF_HW | CKF_SIGN
Mechanism CKM_SHA_1:
  ulMinKeySize: 0
  ulMaxKeySize: 0
  flags: 1024 = CKF_DIGEST
Mechanism CKM_SHA256:
  ulMinKeySize: 0
  ulMaxKeySize: 0
  flags: 1024 = CKF_DIGEST
Mechanism CKM_SHA256_RSA_PKCS:
  ulMinKeySize: 1024
  ulMaxKeySize: 1024
  flags: 2049 = CKF_HW | CKF_SIGN
Mechanism CKM_SHA384:
  ulMinKeySize: 0
  ulMaxKeySize: 0
  flags: 1024 = CKF_DIGEST
Mechanism CKM_SHA384_RSA_PKCS:
  ulMinKeySize: 1024
  ulMaxKeySize: 1024
  flags: 2049 = CKF_HW | CKF_SIGN
Mechanism CKM_SHA512:
  ulMinKeySize: 0
  ulMaxKeySize: 0
  flags: 1024 = CKF_DIGEST
Mechanism CKM_SHA512_RSA_PKCS:
  ulMinKeySize: 1024
  ulMaxKeySize: 1024
  flags: 2049 = CKF_HW | CKF_SIGN
Mechanism CKM_RSA_X_509:
  ulMinKeySize: 1024
  ulMaxKeySize: 1024
  flags: 2561 = CKF_HW | CKF_DECRYPT | CKF_SIGN
Password per token PKCS11 [SunPKCS11-Cryptoki]: sunpkcs11: login succeeded
sunpkcs11: user already logged in
javax.net.ssl.SSLHandshakeException: Error signing certificate verify
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1886)
    *....... (continues with the exception as described above)*

Note: With the last update of Java 6, my code stops working on Java 6 as well :(

Answer 1

Solved by adding the line of code:

Security.removeProvider("IAIK");

before the line:

Security.addProvider(sunpkcs11);

This works correctly with all versions of java6 and java7 (hoping they won't screw again something in java8... :)

Looks like the IAIK provider inserts itself as a PKCS11 provider but then makes calls to the private key using it as a software key... because it's actually a software provider.