PKCS11 deriveKey() and encrypt() returning different results for 3DES

Question

I am working on a key derivation problem using an HSM and PKCS11 and currently I can't understand why I see completely different results depending on whether I use the deriveKey() method as opposed to using the encrypt() method. In both cases I'm attempting to use DESede/ECB/NoPadding algorithm for the result and yet depending on which method (deriveKey vs. encrypt) I use to generate the result, I see different outcomes.

Stepping back for a moment to give a high level overview... I am using a specific key derivation methodology used by Global Platform to diversify the master key for use on a smart card. The method starts with a master key and 8 bytes of data (diversification data) that are used to derive the new key. The div data bytes are literally just encrypted with the master key using DESede/ECB/NoPadding and the result is used as part of the new derived 3DES key. (In actuality, there are multiple encryptions performed and the results are concatenated together to form the new key, but that is not the issue here.)

I've verified the master key in both cases, verified that the diversification data is the same in both cases, verified that I'm using the same algorithm with the same padding. I've also tried changing the derived key template to DES, 2DES, 3DES. All yield similar results that only differ in length.

Currently I've implemented my test case using IAIK wrapper (a pkcs11 wrapper written in Java), which I will post here. The key is just a test key and the div data is sample div data, so there is no sensitive information here. I'm first creating a base key in the HSM, then attempt to derive a new key by calling session.deriveKey() using the diversification data. The derived key bytes are printed in hex (the key value is incorrect, based on my currently working implementation that derives the key in memory instead of within the HSM). Next, I simply initialize the session to perform an encryption using the master key and then encrypt the diversification data. This yields the expected value (again, checked against a working implementation).

What I'm looking for is any insight into why these operations are yielding different results, when I'm using the same base key, the same div data and the same encryption algorithm. I don't understand what deriveKey() is doing under the surface and I'm unable to find any documentation or source code to shed any light on this. I must be able to use the deriveKey() method since the derived key will not be available outside the HSM.

Any insight appreciated.

Mechanism keyGenerationMechanism = Mechanism.get(PKCS11Constants.CKM_DES3_KEY_GEN);

List supportedMechanisms = Arrays.asList(token.getMechanismList());
if (!supportedMechanisms.contains(Mechanism.get(PKCS11Constants.CKM_DES3_KEY_GEN))) {
    output_.println("Mechanism not supported: DES3_KEY_GEN");
    return;
}

// This is the master key that I want to diversify
DES3SecretKey baseKeyTemplate = new DES3SecretKey();
baseKeyTemplate.getValue().setByteArrayValue(new byte[] {0x3d, 0x20, 0x5b, 0x29, (byte) 0xfd, 0x04, (byte) 0xd9, (byte) 0x89, (byte) 0xd0, (byte) 0xfd, (byte) 0x85, (byte) 0xd5, (byte) 0xf7, (byte) 0xb3, 0x31, (byte) 0xd3,
        0x3d, 0x20, 0x5b, 0x29, (byte) 0xfd, 0x04, (byte) 0xd9, (byte) 0x89});
baseKeyTemplate.getDerive().setBooleanValue(Boolean.TRUE);
baseKeyTemplate.getToken().setBooleanValue(Boolean.TRUE);
baseKeyTemplate.getPrivate().setBooleanValue(Boolean.TRUE);
baseKeyTemplate.getSensitive().setBooleanValue(Boolean.FALSE);
baseKeyTemplate.getExtractable().setBooleanValue(Boolean.TRUE);
baseKeyTemplate.getLabel().setCharArrayValue("GP-3des-aba".toCharArray());
baseKeyTemplate.getObjectClass().setLongValue(PKCS11Constants.CKO_SECRET_KEY);
baseKeyTemplate.getKeyType().setLongValue(PKCS11Constants.CKK_DES3);
baseKeyTemplate.getEncrypt().setBooleanValue(Boolean.TRUE);
baseKeyTemplate.getDecrypt().setBooleanValue(Boolean.TRUE);
baseKeyTemplate.getWrap().setBooleanValue(Boolean.TRUE);
baseKeyTemplate.getUnwrap().setBooleanValue(Boolean.TRUE);


output_.println("baseKeyTemplate: " + baseKeyTemplate.toString());

SecretKey baseKey = (SecretKey) session.createObject(baseKeyTemplate);

System.out.println("Base key: ");
System.out.println(baseKey.toString());

output_
        .println("################################################################################");
output_.println("derive key");

//DES3 Key Template
DESSecretKey derived3DESKeyTemplate = new DESSecretKey();
SecretKey derivedKeyTemplate = derived3DESKeyTemplate;

derivedKeyTemplate.getSensitive().setBooleanValue(Boolean.FALSE);
derivedKeyTemplate.getToken().setBooleanValue(Boolean.TRUE);
derivedKeyTemplate.getExtractable().setBooleanValue(Boolean.TRUE);
derivedKeyTemplate.getPrivate().setBooleanValue(Boolean.FALSE);
derivedKeyTemplate.getKeyType().setLongValue(PKCS11Constants.CKK_DES);

// This represents the diversification data (.ie div bytes from some smart card)
byte[] data = new byte[] {0x00, (byte) 0x84, 0x30, (byte) 0x95, 0x35, 0x05,(byte)  0xf0, 0x01};

KeyDerivationStringDataParameters param = new KeyDerivationStringDataParameters(data);
Mechanism mechanism = Mechanism.get(PKCS11Constants.CKM_DES3_ECB);

if (!supportedMechanisms.contains(Mechanism
        .get(PKCS11Constants.CKM_DES3_ECB))) {
    output_.println("Mechanism not supported: CKM_DES3_ECB");
    return;
}

mechanism.setParameters(param);

System.out.println("Derivation Mechanism: ");
output_.println(mechanism.toString());
output_
        .println("--------------------------------------------------------------------------------");

Key derivedKey = session.deriveKey(mechanism, baseKey, derivedKeyTemplate);

if (derivedKey == null) {
    output_.println("Found NO key that can be used for encryption.");
    output_.flush();
    System.exit(0);
}
System.out.println("Derived key: ");
output_.println(derivedKey.toString());

output_
        .println("################################################################################");
output_.println("finished");

// initialize for encryption
Mechanism encryptionMechanism = Mechanism.get(PKCS11Constants.CKM_DES3_ECB);
session.encryptInit(encryptionMechanism, baseKey);
byte[] encryptedData = session.encrypt(data);

System.out.println("Encrypted data: " + new String(Hex.encodeHex(encryptedData)));

// This is the second part of the derived key, let's not worry about this yet since the first part isn't
// working.
//        data = new byte[] {0x00, (byte) 0x84, 0x30, (byte) 0x95, 0x35, 0x05,(byte)  0x0f, 0x01,
//                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
//                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00};
//
//        session.encryptInit(encryptionMechanism, baseKey);
//        encryptedData = session.encrypt(data);
//        System.out.println("Encrypted data: " + new String(Hex.encodeHex(encryptedData)));

session.closeSession();
pkcs11Module.finalize(null);

Here's the relevant output when I run the above code:

Base key:  
  Object Class: Secret Key  
  Token: true  
  Private: true  
  Modifiable: true  
  Label: GP-3des-aba  
  Key Type: DES3  
  ID: <NULL_PTR>  
  Start Date: 00.00.0000 (DD.MM.YYYY)  
  End Date: 00.00.0000 (DD.MM.YYYY)  
  Derive: true  
  Local: false  
  Key Generation Mechanism: <Information unavailable>  
  Allowed Mechanisms: <NULL_PTR>  
  Sensitive: false  
  Encrypt: true  
  Decrypt: true  
  Sign: false  
  Verify: false  
  Wrap: true  
  Unwrap: true  
  Extractable: true  
  Always Sensitive: false  
  Never Extractable: false  
  Check Value: <Attribute not present>  
  Wrap With Trusted: <Attribute not present>  
  Trusted: <Attribute not present>  
  Wrap Template: <Attribute not present>  
  Unwrap Template: <Attribute not present>  
  Value (hex): 3d205b29fd04d989d0fd85d5f7b331d33d205b29fd04d989  
################################################################################
derive key  
Derivation Mechanism:   
  Mechanism: CKM_DES3_ECB  
  Parameters:   
  String data (hex): 008430953505f001  
--------------------------------------------------------------------------------  
01/18/13 14:12:10   CALL: entering (in Java_iaik_pkcs_pkcs11_wrapper_PKCS11Implementation_C_1DeriveKey)  

[snip]  

Derived key:   
  Object Class: Secret Key  
  Token: true  
  Private: false  
  Modifiable: true  
  Label: <NULL_PTR>  
  Key Type: DES  
  ID: <NULL_PTR>  
  Start Date: 00.00.0000 (DD.MM.YYYY)  
  End Date: 00.00.0000 (DD.MM.YYYY)  
  Derive: false  
  Local: false  
  Key Generation Mechanism: CKM_DES3_ECB  
  Allowed Mechanisms: <NULL_PTR>  
  Sensitive: false  
  Encrypt: false  
  Decrypt: false  
  Sign: false  
  Verify: false  
  Wrap: false  
  Unwrap: false  
  Extractable: true  
  Always Sensitive: false  
  Never Extractable: false  
  Check Value: <Attribute not present>  
  Wrap With Trusted: <Attribute not present>  
  Trusted: <Attribute not present>  
  Wrap Template: <Attribute not present>  
  Unwrap Template: <Attribute not present>  
  Value (hex): 3efe0eab6d3db397      <--- call to deriveKey() value incorrect  
################################################################################  
finished  
01/18/13 14:12:12   CALL: entering (in Java_iaik_pkcs_pkcs11_wrapper_PKCS11Implementation_C_1EncryptInit)  
01/18/13 14:12:12   CALL: exiting  (in Java_iaik_pkcs_pkcs11_wrapper_PKCS11Implementation_C_1EncryptInit)  
01/18/13 14:12:12   CALL: entering (in Java_iaik_pkcs_pkcs11_wrapper_PKCS11Implementation_C_1Encrypt)  
01/18/13 14:12:12   CALL: exiting  (in Java_iaik_pkcs_pkcs11_wrapper_PKCS11Implementation_C_1Encrypt)  
Encrypted data: 3fff0faa6c3cb297    <--- call to encrypt() returns the expected value 

Answer 1

Turns out that the key generated in the HSM using DeriveKey and the key bytes I generated using session.encrypt() are essentially the same if one considers how DES parity works.

The least significant bit of each byte in a DES key is the parity bit, which is ignored in many implementations and is not used as part of the key, thus if I properly set the parity bits on my encrypt result then both results match.

0x3efe0eab6d3db397 <--- HSM generated value with correct odd parity
00111110 11111110 00001110 10101011 01101101 00111101 10110011 10010111

0x3fff0faa6c3cb297 <--- encryption result without parity bits set properly
00111111 11111111 00001111 10101010 01101100 00111100 10110010 10010111