Openssl RSA encrypt and decrypt in C

Question

I've got a sample code that is encrypting a message using PEM private key and decrypting it using PEM public key but at the end the decrypted result is empty.

    const char * msg = "this is a test message";
    //********************Encrypt*******************************
    if ((pFile = fopen("private.pem", "rt")) &&
        (rsa = PEM_read_RSAPrivateKey(pFile, NULL, passwd_callback, (void*)pcszPassphrase)))
    {
        fprintf(stderr, "Private key read.\n");

        RSA_private_encrypt(strlen(msg), (unsigned char *)msg, encrypted, rsa, RSA_PKCS1_PADDING);
        fclose(pFile);
    }
    //********************Decrypt*******************************
    pFile = fopen("pubkey.pem", "rt");
    if (rsa = PEM_read_RSAPublicKey(pFile, NULL, NULL, NULL))
    {
        RSA_public_decrypt(strlen((char *)encrypted), encrypted, decrypted, rsa, RSA_PKCS1_PADDING);
        ERR_load_crypto_strings();
        char * err = (char *)malloc(130);
        ERR_error_string(ERR_get_error(), err);
        fprintf(stderr, "Error decrypting message: %s\n", err);
    }

as a result the output of RSA_public_decrypt is 1 but decrypted string is empty.

error message : Error decrypting message: error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding

Answer 1

Your input message msg is a null terminated string, but when you encrypt it to get encrypted buffer it will be binary buffer, strlen(encrypted) that you are passing to RSA_public_decrypt() will be invalid.

So change

RSA_public_decrypt(strlen((char *)encrypted), encrypted, 
decrypted, rsa, RSA_PKCS1_PADDING);

to

RSA_public_decrypt(RSA_size(rsa), encrypted, 
decrypted, rsa, RSA_PKCS1_PADDING);