CryptoAPI: Using CryptVerifySignature to verify a signature from openssl with public key

Question

I am trying to port the AquaticPrime framework for Mac to Windows.

On the Mac, it uses the opensll library, and I try to understand how to port this to Windows, where I have to use the CryptoAPI, I guess.

I mainly need the code for validation of the generated signature with a given public key.

Here's how verification is done with openssl:

1. inputs: license data, public key and signature, both 128 bytes long.
2. A SHA1 digest is calculated from the license data.
3. A RSA context is set up with the public key data
4. RSA_public_decrypt() is called, given the RSA key and the signature, which returns a 20 byte long SHA1 digest - is this digest equal the one from step 2, the signature is valid.

So, how do I do this with CryptoAPI? I've gotten this far:

1. Start with CryptAcquireContext(ctx, 0, 0, PROV_RSA_FULL, CRYPT_VERIFYCONTEXT)
2. Use CryptImportKey with the help of this posting, with pubexp=3 and bitlen=1024. That all works, i.e. I get no errors, and I looked at the binary data to verify that it matches what the MSDN article shows.
3. Create a SHA1 digest from the license data. I've retrieved the resulting 20 byte hash value and see that it matches what I get with openssl on the Mac.

At this point, I call:

CryptVerifySignature (hashHdl, sig, sigLen, keyHdl, 0, 0)

This fails with error code ERROR_INVALID_PARAMETER.

The odd thing is that when I first accidentally had stored a twice as large public key into the PUBLICKEYBLOB structure, I received a NTE_BAD_SIGNATURE error instead. This might suggest that now the public key I am passing is correct.

Why the ERROR_INVALID_PARAMETER error now, then? I've verified that the hash value is correct, and the key appears to be accepted, too. And the "sig" parameter is just a pointer to the 128 bytes of the signature, and sigLen is 128.

So, what am I missing here?

Answer 1

OK, I solved the problem after lots of trial-and-error.

Both the signature and the public key data, when in their pure byte string form, need to be reversed, i.e. first byte to last position, and so on. Then the above works.