OpenSSL RSA signature verification: hash and padding?

Question

I am trying to write code to verify some RSA signatures. The signatures were made using the OpenSSL command-line tool, using the equivalent of this command line:

openssl dgst -sha1 -sign private_key_file.pem < binary_data_file > sig

I am trying to use libtomcrypt to do the verify:

https://www.libtom.net/

Here is the calling signature of the RSA verification function in libtomcrypt:

int rsa_verify_hash_ex(
    const unsigned char *sig, unsigned long siglen,  // signature to verify
    const unsigned char *hash, unsigned long hashlen,  // hash value to check against sig
    int padding,  // defined constant value, see below
    int hash_idx,  // identifies which hash algorithm, see below
    unsigned long saltlen,  // specify salt length, see below
    int *stat,  // output parameter, returns whether verify succeeded or not
    rsa_key *key);  // RSA public key to use for verify

This function returns a 0 if it operates without error, otherwise returns an error code. If it operates without error, the stat output parameter indicates whether the signature verified.

Most of the arguments seem straightforward: pass in the signature to check, the hash value to use to compare it, and the RSA key to use for the check. hash_idx is clear from the example code included with libtomcrypt; it is an index into a table of supported hash algorithms, and I can find the correct value to use with this code snippet: hash_idx = find_hash("sha1")

But I'm wondering about the padding and saltlen values. padding doesn't worry me too much, as there are only two possible values, and I can just try them both. But what should I pass for saltlen?

The OpenSSL documentation for the OpenSSL functions for RSA verify don't show a saltlen parameter. The man page for openssl dgst (i.e. the result of man dgst) does not discuss salt.

So my questions:

• How can I determine the correct salt length to use?
• Does OpenSSL's dgst command insert any extra stuff in the input, such as: (stdin)=

(I found that (stdin)= thing by searching StackOverflow: Why are the RSA-SHA256 signatures I generate with OpenSSL and Java different?)

• libtomcrypt also has a function called pkcs_1_pss_decode() which is documented to "decode a PSS encoded signature block". Is there any chance that this is the function I need to call?

Thanks for any help you can give me.

EDIT: thanks to the help below, from @Jonathan Ben-Avraham, I was able to get this working today. The answers to my questions are, respectively:

• Use length 0 for the salt, no salt at all.
• No, OpenSSL did not insert anything extra such as (stdin)=
• I needed to call rsa_verify_hash_ex(), and I needed to specify the padding argument as LTC_LTC_PKCS_1_V1_5.

Answer 1

No salt:

First, generate a binary SHA1 hash of your data:

openssl dgst -sha1 -binary -out hash1 some_data_file

This is an SHA1 hash or digest. There is no salt prependended to the file some_data_file. The openssl dgst -sha1 itself does not add salt. Note that the output file is just a 20 byte SHA1 hash with no salt. If there were salt, the hash would have to include it, probably prepended before the last 20 bytes that hold the SHA1 hash.

Next, sign the SHA1 hash file hash1 with your private key:

openssl pkeyutl -sign -in hash1 -inkey privkey.pem -pkeyopt digest:sha1 -out sig1

Now sign the some_data_file with openssl dgst:

openssl dgst -sha1 -sign privkey.pem < some_data_file > sig2

Finally, compare the two signatures:

diff sig1 sig2

and you should see that they are the same. This tells us that signing the raw SHA1 hash of a file with no salt is the same as using the openssl dgst -sha1 -sign command to sign the file, so it must be that the openssl dgst -sha1 -sign command also did not use any salt when generating its SHA1 hash for sig2.

Note also that you cannot achieve the same result using the deprecated rsautl:

openssl rsautl -sign -in hash1 -inkey privkey.pem -out sig1

instead of openssl pkeyutl, because openssl rsautl -sign does not do the ASN.1 encoding of DigestInfo as required by RSASSA-PKCS1-v1_5 defined in e.g. RFC3447 section 9.2 step 2. (This is why you need -pkeyopt digest: even though pkeyutl -sign itself doesn't do any hashing.) See this SE post for details.

Answer 2

One thing to emphasize: Be sure to pass the hash instead of the actual data. That threw me off for some time. Here's a snippet that works (but use sha256):

void
verify_tomcrypt(unsigned char *keyblob, size_t klen,
                unsigned char *payload, size_t dlen,
                unsigned char *signature, size_t slen)
{
    rsa_key key;
    int stat;
    unsigned long len;
    unsigned char digest2[SHA256_DIGEST_LENGTH];

    ltc_mp = ltm_desc;
    register_hash(&sha256_desc);

    /* try reading the key */
    if (rsa_import(keyblob, klen, &key) != CRYPT_OK) {
        printf("Error reading key\n");
        exit(-1);
    }

    int hash_idx = find_hash("sha256");
    if (hash_idx == -1) {
        printf("LTC_SHA256 not found...?\n");
        exit(-1);
    }
    len = sizeof(digest2);
    if (hash_memory(hash_idx, payload, dlen, digest2, &len) != CRYPT_OK) {
        printf("sha256 fails...?\n");
        exit(-1);
    }

    if (rsa_verify_hash_ex(signature, slen, digest2, sizeof(digest2), LTC_LTC_PKCS_1_V1_5, hash_idx, 0, &stat, &key) == CRYPT_OK) {
        if (stat == 1)
            printf("Tomcrypt: Signature OK!\n");
        else
            printf("Tomcrypt: Signature NOK?\n");
    } else {
        printf("Tomcrypt: Signature error\n");
    }
}