Java vs Python HMAC-SHA256 Mismatch

Question

Based on recent feedback and findings on this problem, I've rewritten the question to get rid of noise.

I have 2 separate code paths, one in Java (Android), one and Python which accomplish the following for the purposes of negotiating a pairing between an Android device and a Python/Django.

Java:

• Generate a syncKey
• Hash a concatenated string of various values using the presharedKey (including the syncKey)
• Encrypt the syncKey using a presharedKey
• Send the Hash, encrypted syncKey, DeviceId and arbitrary variables to web server

Python

• Get the presharedKey from the deviceId
• Decrypt the encrypted syncKey
• Hash a concatenated string of various values using the presharedKey (including the decrypted syncKey)
• Make sure the hash matches, which confirms that the syncKey was decrypted successfully, and that the deviceId holds the correct presharedKey.

Now this process works if I send the syncKey unencrypted. The final hash matches, which proves the deviceId has the correct preshared-key, however as soon as I add the en/decryption into the process, the hash no longer matches, despite the fact that both the syncKey and concatenated string appear to match perfectly character for character from the debug output of both Java/Python.

One quirk of the process is that a 256bit key is necessary for the AES256 encryption algorithm, so I'm chopping the 512bit presharedKey in half. The alternative of using only a 256bit key across the board was requiring that I pass the key through encode('ascii') on the python side, or else it was throwing up errors during hashing with the shorter key.

Here is the relevant code:

Java:

String presharedKey = getKey();
// f8250b0d5960444e4de6ecc3a78900bb941246a1dece7848fc72b90092ab3ecd0c1c8e36fddba501ef92e72c95b47e07f98f7fd9cb63da75c008a3201124ea5d

String deviceId = getDeviceId();
// 1605788742789230

SyncKey syncKey = generateSyncKey();
// 824C1EE9EF507B52EA28362C71BD4AD512A5F82ACFAE80DEF531F73AC124CA814BA30CE805A157D6ADB9EC04FC99AAE6FDC4238FCD76B87CE22BC2FE33B2E5C9

String concat = syncKey.hexString();
// 824C1EE9EF507B52EA28362C71BD4AD512A5F82ACFAE80DEF531F73AC124CA814BA30CE805A157D6ADB9EC04FC99AAE6FDC4238FCD76B87CE22BC2FE33B2E5C9

String ALGORITHM = "HmacSHA256";
String hash = null;
try {
    SecretKeySpec keySpec = new SecretKeySpec(
        presharedKey.getBytes(),
        ALGORITHM);
    Mac mac = Mac.getInstance(ALGORITHM);
    mac.init(keySpec);
    byte[] result = mac.doFinal(concat.getBytes());
    hash = Base64.encodeToString(result, Base64.DEFAULT);
    // FpDE2JLmCBr+/rW+n/jBHH13F8AV80sUM2fQAY2IpRs=
} catch (NoSuchAlgorithmException x) {
} catch (InvalidKeyException x) {
}

String encKey = presharedKey.substring(0, presharedKey.length() / 2);
// f8250b0d5960444e4de6ecc3a78900bb941246a1dece7848fc72b90092ab3ecd

int len = encKey.length();
byte[] encKeyBytes = new byte[len / 2];
for (int i = 0; i < len; i += 2) {
    encKeyBytes[i / 2] = (byte) ((Character.digit(encKey.charAt(i), 16) << 4)
            + Character.digit(encKey.charAt(i+1), 16));
}

String encryptedSyncKey = null;
try {
    byte[] iv = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00 };
    AlgorithmParameterSpec ivSpec = new IvParameterSpec(iv);
    SecretKeySpec encKeySpec = new SecretKeySpec(encKeyBytes, "AES");
    Cipher cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
    cipher.init(Cipher.ENCRYPT_MODE, encKeySpec, ivSpec);
    byte[] encryptedSyncKeyBytes = cipher.doFinal(syncKey.hexString().getBytes());
    encryptedSyncKey = Base64.encodeToString(encryptedSyncKeyBytes, Base64.DEFAULT);
    /*
        Yrl0/SuTUUTC6oJ8o4TCOy65EwO0JzoXfEi9kLq0AOlf6rH+nN7+BEc0s5uE7TIo1UlJb/DvR2Ca
        ACmQVXXhgpZUTB4sQ0eSo+t32lg0EEb9xKI5CZ4l9QO5raw0xBn7r/tfIdVm8AIFkN9QCcthS0DF
        KH3oWhpwNS+tfEuibLPgGqP/zGTozmido9U9lb4n
    */
} catch (InvalidAlgorithmParameterException e) {
} catch (NoSuchAlgorithmException e) {
} catch (NoSuchPaddingException e) {
} catch (InvalidKeyException e) {
} catch (IllegalBlockSizeException e) {
} catch (BadPaddingException e) {
}

sendStuffToWeb(encryptedSyncKey, deviceId, hash);

Python:

hash = getHash(request)
# hash from Java: FpDE2JLmCBr+/rW+n/jBHH13F8AV80sUM2fQAY2IpRs=

encrypted_sync_key = getEncSyncKey(request)
# encryptedSyncKey from Java:
# Yrl0/SuTUUTC6oJ8o4TCOy65EwO0JzoXfEi9kLq0AOlf6rH+nN7+BEc0s5uE7TIo1UlJb/DvR2Ca
# ACmQVXXhgpZUTB4sQ0eSo+t32lg0EEb9xKI5CZ4l9QO5raw0xBn7r/tfIdVm8AIFkN9QCcthS0DF
# KH3oWhpwNS+tfEuibLPgGqP/zGTozmido9U9lb4n

device_id = getDeviceId(request)
# 1605788742789230

preshared_key = getPresharedKeyFromDevice(deviceId)
# f8250b0d5960444e4de6ecc3a78900bb941246a1dece7848fc72b90092ab3ecd0c1c8e36fddba501ef92e72c95b47e07f98f7fd9cb63da75c008a3201124ea5d

enc_key = preshared_key[:len(preshared_key)/2]
# f8250b0d5960444e4de6ecc3a78900bb941246a1dece7848fc72b90092ab3ecd

aes = AES.new(enc_key.decode('hex'), AES.MODE_CBC, IV="\x00"*16)
sync_key = aes.decrypt(base64.b64decode(encrypted_sync_key))
# 824C1EE9EF507B52EA28362C71BD4AD512A5F82ACFAE80DEF531F73AC124CA814BA30CE805A157D6ADB9EC04FC99AAE6FDC4238FCD76B87CE22BC2FE33B2E5C9

concat = sync_key
# 824C1EE9EF507B52EA28362C71BD4AD512A5F82ACFAE80DEF531F73AC124CA814BA30CE805A157D6ADB9EC04FC99AAE6FDC4238FCD76B87CE22BC2FE33B2E5C9

import hashlib
from hmac import new as hmac

verify_hash = hmac(preshared_key, concat, hashlib.sha256).digest().encode('base64')
# IoSc2w2sQ4/fwhJTdUQHw/Hdyjy+ranzQ1z3J5LfYbA=

From the debug output below you can see the syncKey is encrypted and decrypted successfully, and the concat is identical. However the resulting hash ends up being different.

Answer 1

Your Python code is wrong. I can reproduce, in Python, the answer you got in Java.

If I use your inputs:

>>> preshared_key_hex
b'f8250b0d5960444e4de6ecc3a78900bb941246a1dece7848fc72b90092ab3ecd0c1c8e36fddba501ef92e72c95b47e07f98f7fd9cb63da75c008a3201124ea5d'
>>> concat_hex
b'824C1EE9EF507B52EA28362C71BD4AD512A5F82ACFAE80DEF531F73AC124CA814BA30CE805A157D6ADB9EC04FC99AAE6FDC4238FCD76B87CE22BC2FE33B2E5C9'

I get the same value you get in Java:

>>> base64.b64encode(hmac.new(preshared_key_hex, concat_hex, hashlib.sha256).digest())
b'FpDE2JLmCBr+/rW+n/jBHH13F8AV80sUM2fQAY2IpRs='

However, that value is probably also wrong. You should almost certainly be hex decoding the input values.

I'm unable to reproduce what you got in Python; one of the values you're passing to hmac.new isn't what you think it is. print them immediately before calling hmac.new and you should see what doesn't match.