ECDH Shared secret derived from OpenSSL and BouncyCastle are not always the same although constants and domain parameters are the same for both

Question

I'm trying to implement AES cryptography between an iOS app and a java servlet. Java servlet uses BouncyCastle library while iOS app uses OpenSSL. Although I've used same public/private key pair and domain parameters for both side, shared secret generated by OpenSSL sometimes differs from what generates by BouncyCastle on server - side.

The procedure is as follows;

1. a public/private key pair generated in server with specified domain parameters (say server_public_key, server_private_key)
2. server_public_key is embedded in iOS app in form of an EC_POINT X and Y
3. at run-time iOS app generates its own public/private key pair (say client_key_curve which is an EC_KEY), and
4. then loads server_public_key and calculates shared secret (key_agreement) based on server_public_key and client_key_curve, and
5. then client_public_key (extracted from client_key_curve) as well as a ciphered message which is encrypted symmetrically using the derived shared secret (key_agreement) are sent to server
6. in server - side, shared secret again is calculated using client_public_key and server ECDH parameters which are the same as client side, and
7. then ciphered message decrypted using computed key_agreement

BUT the decrypted messages are not always the same as the client sent messages.

Since I've also developed an Android app which uses the same procedure but employs BouncyCastle for cryptography therefore I suspect the correctness of implemented code using OpenSSL, so the code is revealed here for others to help resolving the issue. What I've implemented to calculate the shared secret is as follows

- (void)calculateSharedSecret
{
    BN_CTX* bn_ctx;

    EC_KEY*     client_key_curve = NULL;
    EC_KEY*     server_key_curve = NULL;
    EC_GROUP*   client_key_group = NULL;
    EC_GROUP*   server_key_group = NULL;
    EC_POINT*   client_publicKey = NULL;
    EC_POINT*   server_publicKey = NULL;
    BIGNUM*     client_privatKey = NULL;

    BIGNUM* client_publicK_x = NULL;
    BIGNUM* client_publicK_y = NULL;
    BIGNUM* server_publicK_x = NULL;
    BIGNUM* server_publicK_y = NULL;

    NSException *p = [NSException exceptionWithName:@"" reason:@"" userInfo:nil];

    bn_ctx = BN_CTX_new();
    BN_CTX_start(bn_ctx);

    client_publicK_x = BN_CTX_get(bn_ctx);
    client_publicK_y = BN_CTX_get(bn_ctx);
    client_privatKey = BN_CTX_get(bn_ctx);
    server_publicK_x = BN_CTX_get(bn_ctx);
    server_publicK_y = BN_CTX_get(bn_ctx);

    // client

    if ((client_key_curve = EC_KEY_new_by_curve_name(NID_X9_62_prime256v1)) == NULL)
        @throw p;

    if ((client_key_group = (EC_GROUP *)EC_KEY_get0_group(client_key_curve)) == NULL)
        @throw p;

    if (EC_KEY_generate_key(client_key_curve) != 1)
        @throw p;

    if ((client_publicKey = (EC_POINT *)EC_KEY_get0_public_key(client_key_curve)) == NULL)
        @throw p;

    if (EC_KEY_check_key(client_key_curve) != 1)
        @throw p;

    client_privatKey = (BIGNUM *)EC_KEY_get0_private_key(client_key_curve);

    char *client_public_key = EC_POINT_point2hex(client_key_group, client_publicKey, POINT_CONVERSION_COMPRESSED, bn_ctx);
    char *client_privat_key = BN_bn2hex(client_privatKey);

    _clientPublicKey = [NSString stringWithCString:client_public_key encoding:NSUTF8StringEncoding];

    // server

    NSArray* lines = [self loadServerPublicKeyXY];

    NSString *public_str_x = [lines objectAtIndex:0];
    NSString *public_str_y = [lines objectAtIndex:1];

    BN_dec2bn(&server_publicK_x, [public_str_x UTF8String]);
    BN_dec2bn(&server_publicK_y, [public_str_y UTF8String]);

    if ((server_key_curve = EC_KEY_new_by_curve_name(NID_X9_62_prime256v1)) == NULL)
        @throw p;

    if ((server_key_group = (EC_GROUP *)EC_KEY_get0_group(server_key_curve)) == NULL)
        @throw p;

    if (EC_KEY_generate_key(server_key_curve) != 1)
        @throw p;

    if ((server_publicKey = EC_POINT_new(server_key_group)) == NULL)
        @throw p;

    if (EC_POINT_set_affine_coordinates_GFp(server_key_group, server_publicKey, server_publicK_x, server_publicK_y, bn_ctx) != 1)
        @throw p;

    if (EC_KEY_check_key(server_key_curve) != 1)
        @throw p;

    unsigned char *key_agreement = NULL;
    key_agreement = (unsigned char *)OPENSSL_malloc(SHA_DIGEST_LENGTH);
    if (ECDH_compute_key(key_agreement, SHA_DIGEST_LENGTH, server_publicKey, client_key_curve, KDF1_SHA1) == 0)
        @throw p;
    _symmetricKey = [NSData dataWithBytes:key_agreement length:16];
}

and

void *KDF1_SHA1(const void *input, size_t inlen, void *output, size_t *outlen)
{
    if (*outlen < SHA_DIGEST_LENGTH)
        return NULL;
    else
        *outlen = SHA_DIGEST_LENGTH;
    return SHA1(input, inlen, output);
}

_clientPublicKey and _symmetricKey are declared at class level

The same curve (named prime256v1 or secp256r1) is used on both side but the results are not always the same.

EDIT 1: In response to @PeterDettman, I’ve published server – side code for more clarification

public byte[] generateAESSymmetricKey(byte[] client_public_key_hex) throws InvalidRequest{
    try {
        // ECDH Private Key as well as other prime256v1 params was generated by Java "keytool" and stored in a JKS file
        KeyStore keyStore = ...;
        PrivateKey privateKey = (PrivateKey) keyStore.getKey("keyAlias", "keyStorePassword".toCharArray());
        ECPrivateKeyParameters ecdhPrivateKeyParameters = (ECPrivateKeyParameters) (PrivateKeyFactory.createKey(privateKey.getEncoded()));

        ECCurve ecCurve = ecdhPrivateKeyParameters.getParameters().getCurve();
        ECDomainParameters ecDomainParameters = ecdhPrivateKeyParameters.getParameters();
        ECPublicKeyParameters client_public_key = new ECPublicKeyParameters(ecCurve.decodePoint(client_public_key_hex), ecDomainParameters);

        BasicAgreement agree = new ECDHBasicAgreement();
        agree.init(ecdhPrivateKeyParameters);
        byte[] keyAgreement = agree.calculateAgreement(client_public_key).toByteArray();

        SHA1Digest sha1Digest = new SHA1Digest();
        sha1Digest.update(keyAgreement, 0, keyAgreement.length);
        byte hashKeyAgreement[] = new byte[sha1Digest.getDigestSize()];
        sha1Digest.doFinal(hashKeyAgreement, 0);

        byte[] server_calculatd_symmetric_key = new byte[16];
        System.arraycopy(hashKeyAgreement, 0, server_calculatd_symmetric_key, 0, server_calculatd_symmetric_key.length);
        return server_calculatd_symmetric_key;
    } catch (Throwable ignored) {
        return null;
    }
}

where client_public_key_hex is client_public_key that is converted to an array of byte. The expected result is that server_calculatd_symmetric_key equals symmetricKey for all the time. BUT they are not always the same.

EDIT 2: As a feedback to @PeterDettman answer, I made some changes to reflect his suggestion and although rate of inequality reduces, generated key agreements (shared secret) on either side are not still equal in all cases.

It is possible to reproduce one of inequality case with following data

• Public key : 02E05C058C3DF6E8D63791660D9C5EA98B5A0822AB93339B0B8815322131119C4C
• Privat key : 062E8AC930BD6009CF929E51B37432498075D21C335BD00086BF68CE09933ACA
• Generated Shared Secret by OpenSSL : 51d027264f8540e5d0fde70000000000
• Generated Shared Secret by BouncyCastle : 51d027264f8540e5d0fde700e5db0fab

So is there any mistake in the implemented code or procedure?

Thanks

Answer 1

There's a problem in the server code, in the way that the ECDH agreement value is converted to bytes:

byte[] keyAgreement = agree.calculateAgreement(client_public_key).toByteArray();

Try this instead:

BigInteger agreementValue = agree.calculateAgreement(client_public_key);
byte[] keyAgreement = BigIntegers.asUnsignedByteArray(agree.getFieldSize(), agreementValue);

This will ensure a fixed-size byte array as output, which is a requirement for converting EC field elements to octet strings (search "Field Element to Octet String Conversion Primitive" for more details).

I recommend you ignore the SHA1 key derivation part until you can get that Java keyAgreement byte array to exactly match the input to your KDF1_SHA1 function.