Can't decrypt RSA data with open SSL

Question

I try to encrypt some data in matlab using a public key I created with openssl

I created the keys using:

openssl genrsa -des3 -out private.pem 1024
openssl rsa -in private.pem -pubout -outform DER -out public.der

I encrypt my data using this matlab code (with Java libraries):

import java.security.spec.RSAPublicKeySpec
import javax.crypto.Cipher;
import java.security.KeyFactory
import java.math.BigInteger

fid = fopen('public.der');
a = fread(fid);
key = java.security.spec.X509EncodedKeySpec(a);
kf = KeyFactory.getInstance('RSA');
KEY = kf.generatePublic(key);

cipher = Cipher.getInstance('RSA/ECB/PKCS1Padding');
cipher.init(Cipher.ENCRYPT_MODE, KEY)

plaintextBytes = [24];
ciphertext = cipher.doFinal(plaintextBytes)' ;

fid2 = fopen('msg.txt','w');
fwrite(fid2,ciphertext);
fclose(fid2);

I try to decrypt it using:

openssl rsautl -decrypt -inkey private.pem  -in msg.txt -keyform PEM -pkcs

Then I get this error:

RSA operation error
80305:error:0407109F:rsa routines:RSA_padding_check_PKCS1_type_2:pkcs decoding error:/BuildRoot/Library/Caches/com.apple.xbs/Sources/OpenSSL098/OpenSSL098-59.40.2/src/crypto/rsa/rsa_pk1.c:267:
80305:error:04065072:rsa routines:RSA_EAY_PRIVATE_DECRYPT:padding check failed:/BuildRoot/Library/Caches/com.apple.xbs/Sources/OpenSSL098/OpenSSL098-59.40.2/src/crypto/rsa/rsa_eay.c:614:

Answer 1

Most of the time for such "RSA_padding_check_PKCS1_type_2 error ..." - you tends to see this with (1) Encoding errors: instead of decrypting binary data, decrypting is done on (maybe) Base64 encoded data. (2) Mismatched key pair or key itself: Public key does not matched to the Private key for decryption. http://hustoknow.blogspot.ca/2013/01/rsa-block-type-is-not-02-error.html

Maybe we can make sure the pair is not mismatch (2) too before saying the cipher load is not correct (1). Like below in ref to https://www.sslshopper.com/ssl-converter.html

Convert PEM to DER: openssl x509 -outform der -in certificate.pem -out certificate.der or if the cert is already in "der" format, can also be turn into "pem" e.g. Convert DER to PEM: openssl x509 -inform der -in certificate.cer -out certificate.pem

Convert PEM to PFX: openssl pkcs12 -export -out certificate.pfx -inkey privateKey.key -in certificate.crt -certfile CACert.crt or if there is only "pfx" then can get "pem" e.g. Convert PFX to PEM: openssl pkcs12 -in certificate.pfx -out certificate.cer -nodes

After making sure we have the "pem", we can try the encrypt and decrypt as stated in http://openssl.6102.n7.nabble.com/unable-to-decrypt-using-using-private-key-td15204.html

e.g. 1) openssl enc -base64 -d -in -out where was created and it had binary contents. e.g. 2) openssl rsautl -decrypt -inkey -out -pkcs but in this case, consider trying to use -raw instead of -pkcs to decrypt data with server private key