Why OpenSSL RSA_sign function needs int type parameter?

Question

RSA signature generation is supposed to be modular exponentiation on big numbers; I understand why OpenSSL needs message digest and its size to calculate RSA signature, however, why it is required to know what is message digest algorithm used to generate input value to calculate its modular exponentiation and generate signature?

Answer 1

The signature is calculated from the hash and the hash algorithm formatted as a PKCS#1 v1.5 DigestInfo (See RFC2313, section 10.1.2). This is required, to ensure that the recipient can recalculate the hash themselves to verify the signature, and it must be included in the signature to ensure that the indicated hash algorithm hasn't been changed by an attacker.