OpenSSL decrypt error - Padding versus Raw

Question

I am receiving an encrypted file and it's key from a partner. The Key has itself been encrypted using our Digital Certificate Public Key.

When I attempt to decrypt the key using the following and our private key, I get a padding error as shown below:

C:\openssl rsautl -decrypt -in xxxx_Key -inkey xxxxprivatekey.pem -hexdump -out aeskey.txt
Loading 'screen' into random state - done
RSA operation error
5612:error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding er
ror:.\crypto\rsa\rsa_pk1.c:273:
5612:error:04065072:rsa routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed:.\
crypto\rsa\rsa_eay.c:602:

If I add the -Raw switch to the decrypt, it appears to work but the resulting hexdump is WAY larger than I'm expecting. Can anyone offer advice as to what may be going on here? Thanks!

Answer 1

My guess is that you are decrypting with the wrong private key or your ciphertext is corrupted.

In RSA, padding is used to extend the length of the message being encrypted to be the same size as the modulus (so 1024 bit RSA pads messages to 1024 bits). PKCS1 type 2 is (I believe) another name for PKCS#1 v1.5 which adds the padding 0x00 || 0x02 || (random bytes) || 0x00 to the start of the message. When decrypting the first check that is done is that the start of the message is 0x00 0x02. Then all bytes up to and including the second 0x00 are stripped off, yielding the original message. If the start is not 0x00 0x02 or there is no second 0x00 byte then there is a padding error.

If you ignore the padding check you most likely will get a message the same size as the RSA modulus since no padding is stripped off. Considering most RSA moduli are at least 1024 bit this will be much larger than an AES key.