Signing 20-byte message with 256-bit RSA key working with openssl.exe but not in code

Question

I have a 256-bit private key that I want to use to sign a SHA-1 digest (20 bytes). Using openssl directly it seems to work

echo doesntmatter | openssl dgst -sha1 -binary | openssl rsautl -sign -inkey 256bit_private_key.pem | openssl enc -base64

gives me a Base64 output as expected.

But doing it with the OpenSSL fails with "error:04075070:rsa routines:RSA_sign:digest too big for rsa key". As you can see below, I'm passing the 20-byte (SHA_DIGEST_LENGTH=20) SHA-1 digest as input to RSA_sign. Even with padding it shouldn't be more than the maximum of 32 bytes that I can encrypt with a 256 bit modulus key?!

unsigned char digest[SHA_DIGEST_LENGTH];
SHA1(message, messageSize, digest);

unsigned int privateKeySize = RSA_size(privateKey); // 256 bits = 32 bytes
unsigned char* signature = new unsigned char[privateKeySize];
unsigned int signatureSize;

int res = RSA_sign(NID_sha1, digest, SHA_DIGEST_LENGTH, signature, &signatureSize, privateKey);

if(res == 0)
{
    int err = ERR_get_error(); // 67588208
    char *s = ERR_error_string(err, 0); // error:04075070:lib(4):func(117):reason(112)

    delete [] signature;

    [...]
}

What am I doing wrong in the code?

Answer 1

check out this SO answer. rsautl is depreciated in favor of pkeyutl

Essentially, RSA signing should use RSA-PSS signature padding scheme to be secure, and the RSA-PSS will be strictly larger than the digest, because of the necessity to communicate a salt. additionally, RSA_sign does a digest as a part of the signing scheme, so your code is going to do a digest of a digest. you want to instead either pass the message in directly, or use RSA_private_encrypt combined with a suitable RSA_padding_add_x call.

and as I said in my comment, a 256-bit key is sufficiently secure for a symmetric algorithm, but it will be trivial to crack for an RSA private key (on the order of 4 minutes on a couple of year old machine). Using a bare minimum of a 1024 bit key (and probably a 2048 or 4096 bit key) will give you a security level roughly equivalent to a 128 bit symmetric algorithm.