Diffie-Hellman with BIGNUM (OpenSSL) vs. BigInteger (C#)

Question

I'm implementing a Diffie-Hellman key exchange algorithm between an embedded device which uses OpenSSL libraries for BIGNUM operations and a C# software which uses System.Numerics.BigInteger methods to calculate the secret shared key generation.

But after Alice and Bob exchange keys, they calculate different shared secrets.

The keys are printed out on each side (PubA, PrivA, PubB, PrivB, DHPrime, DHGenerator) and I can see that they are the same.

I suspect there is an issue about little/big endianness, or maybe openssl doesn't care about negative numbers for the exponent, and I don't know how to debug those operations. I don't have the code with me right now, but all the operations are kept simple, like this.

C# side

BigInteger bintA = new BigInteger(baByteArrayReceived);
BigInteger bintb = new BigInteger(baRandomBobSecret);
BigInteger bintDHPrime = new BigInteger(baDHPrime2048);

BigInteger bintSharedSecret = bintA.ModPow(bintb,bintDHPrime);

C side

BIGNUM *bnB = BN_new();
BIGNUM *bna = BN_new();
BIGNUM *bnDHPrime = BN_new();
BIGNUM *bnResult = BN_new();
BN_CTX *bnctx = BN_CTX_new();

BN_bin2bn(baBReceived, 256,bnB);
BN_bin2bn(baRandomAliceSecret, 256,bna);
BN_bin2bn(baDHPrime2048, 256,bnDHPrime);

BN_mod_exp(bnResult,bnB,bna,bnDHPrime,bnctx);

Some additional info for C methods of OpenSSL: >

BIGNUM *BN_bin2bn(const uint8_t *in, size_t len, BIGNUM *ret);

BN_bin2bn sets |*ret| to the value of |len| bytes from |in|, interpreted as a big-endian number, and returns |ret|. If |ret| is NULL then a fresh |BIGNUM| is allocated and returned. It returns NULL on allocation failure.

 int BN_mod_exp(BIGNUM *r, const BIGNUM *a, const BIGNUM *p, 
     const BIGNUM *m, BN_CTX *ctx);

BN_mod_exp sets |r| equal to |a|^{|p|} mod |m|. It does so with the best algorithm for the values provided and can run in constant time if |BN_FLG_CONSTTIME| is set for |p|. It returns one on success or zero otherwise.

And they produce different results.

What should I do about this? What would be your next thing to check?

Thanks in advance.

Answer 1

Solved by using BouncyCastle Crypto Library's BigInteger methods. Couldn't tell System.Numerics.BigInteger that we are using big endian unsigned chars for initialization.