RSA_public_decrypt and MS Crypto API equivalent

Question

I'm trying to develop a license verification solution. Licenses are encoded on server using OpenSSL's RSA_private_encrypt function.

For Mac OX X I use RSA_public_decrypt and it works like a charm. On Windows I must use very tiny bit of code, so I can not link with OpenSSL or other lib AND I have to use MS Crypto API.

I have spent several days trying to figure out what is wrong, but with no luck. I can successfully import public key, but here my success ends. I'm aware that I need to reverse byte order with CAPI so this might not be the issue.

I have tried everything, including CryptVerifyMessageSignatureWithKey and CryptDecodeObject to load the blob with different params, but still no luck.

It always ends up with GetLastError() == CRYPT_E_ASN1_BADTAG, which I assume means that the BLOB is not ASN1 formatted... Google does not tell anything on the output format of RSA_private_encrypt... so I am completely lost here.

Here is the OS X code based on OpenSSL:

void cr_license_init(const char* lic) {
    __cr_license_ = lic;
    unsigned char lic_encoded[CR_LIC_LEN];

    BIO* b64 = BIO_new(BIO_f_base64());
    BIO* licIn = BIO_new_mem_buf((void*)lic, -1);
    licIn = BIO_push(b64, licIn);

    if(BIO_read(licIn, lic_encoded, CR_LIC_LEN) == CR_LIC_LEN) {

        const unsigned char* key_data = license_pub_der;
        RSA* r = d2i_RSA_PUBKEY(NULL, &key_data, sizeof(license_pub_der));

        if(r != NULL) {
            if(__cr_license_data_ != NULL) {
                free((void*)__cr_license_data_);
            }
            __cr_license_data_ = malloc(CR_LIC_LEN);
            if(RSA_public_decrypt(CR_LIC_LEN, lic_encoded,
    (unsigned char*)__cr_license_data_, r, RSA_PKCS1_PADDING) &lt= 0) {
                free((void*)__cr_license_data_);
                __cr_license_data_ = NULL;
            }
            RSA_free(r);
        }
    }
    BIO_free_all(licIn);
}

This part of code on windows works well, so I assume public key is not an issue.

__cr_license_ = lic;
unsigned char lic_encoded[CR_LIC_LEN];

DWORD dwSize;
if(CryptStringToBinaryA(__cr_license_, 0/*autocalculate*/, CRYPT_STRING_BASE64, lic_encoded, &dwSize, NULL, NULL) && dwSize == CR_LIC_LEN) {
HCRYPTPROV hProv;
if(CryptAcquireContext(&hProv, NULL, MS_ENHANCED_PROV, PROV_RSA_FULL, CRYPT_VERIFYCONTEXT)) {
    PCERT_PUBLIC_KEY_INFO pki = NULL;
    DWORD dwKeySize;
    if(CryptDecodeObjectEx(X509_ASN_ENCODING, X509_PUBLIC_KEY_INFO, license_pub_der, sizeof(license_pub_der), CRYPT_ENCODE_ALLOC_FLAG, NULL, &pki, &dwKeySize)) {
        HCRYPTKEY hKey = 0;
        if(CryptImportPublicKeyInfo( hProv, X509_ASN_ENCODING, pki, &hKey)) {

But after that anything I try to do with message leads to CRYPT_E_ASN1_BADTAG. I tried CryptMsgOpenToDecode with CryptMsgUpdate, CryptDecodeObject, CryptVerifyMessageSignatureWithKey - nothing works.

Basically I think that the problem is in pkcs1 and pkcs7 incompatibility as owlstead mentioned. Does anyone has experience working with pkcs1 format importing/converting/etc with MS CAPI?

Any help or even a clue is appreciated a lot! Thanks in advance!

Answer 1

You are mixing higher and lower level signature formats. OpenSSL asumes PKCS#1 v1.5 signatures by default, which contains of only the signature data. Windows seems to asume PKCS#7 containers. These may contain a PKCS#1 v1.5, but those and other data are wrapped using ASN.1 BER tag/length format. If the Microsoft API tries to decode this it will assume that the raw signature is the container format, and decoding will fail.