OpenSSL C++ RSA sign is different from command line sign

Question

I'm trying to sign a text in C++ and then verify it in the command line. I'm using OpenSSL libraries. This is my command line for key generation:

openssl genrsa -out key.pem 1024

Now I have my private key. Then this is how I do to sign in command line:

echo "hola" | openssl rsautl -pkcs -sign -inkey key.pem > sign.txt

At this point all works like it seems to be, now I have a sign in sign.txt. Now I'm trying to do the same in C... This is my code:

RSA * rsaPrivKey;


RSA * createRSAWithFilename (const char * filename, int publicKey)
{
   FILE * fp = fopen (filename, "rb");

   if (fp == NULL)
   {
      printf ("Unable to open file %s \n", filename);
      return NULL;
   }
   RSA *rsa = RSA_new ();

   if (publicKey)
      rsa = PEM_read_RSA_PUBKEY (fp, &rsa, NULL, NULL);
   else
      rsa = PEM_read_RSAPrivateKey (fp, &rsa, NULL, NULL);

   return rsa;
}


void initRSA (void)
{
   rsaPrivKey = createRSAWithFilename ("key.pem", 0);

   unsigned char text[] = {"hola"};
   unsigned char encrypted[4098] = {};
   unsigned int outlen;

   unsigned char hash[20];
   if (!SHA1 (text, sizeof(text), hash)){
      printf ("SHA1 failed\n");
      exit (0);
   }
   if (!RSA_sign (NID_sha1, hash, 20, encrypted, &outlen, rsaPrivKey)){
      printf ("RSA_sign failed\n");
      exit (0);
   }

   printf ("Result:\n");
   for (int a = 0; a < outlen; a++)
      printf ("%c", encrypted[a]);

   exit (1);
}

When I call initRSA() it prints the generated signature.. but.. is not the same as in generated in command line.

Because not sure about if the sizeof is taking the real length of "text" I tried with length = 4 (hola have 4 chars) and 5 (perhaps computing \0) and the results are not the expected.

My knowledge in cryptography is very limited.. don't know where is the problem.

Answer 1

Apparently it's because you're using the wrong command line. You want this command line, which computes a digest of your input and signs it using the private key:

echo -n "hola" | openssl dgst -sha1 -binary -sign key.pem >sign.txt

Using pkeyutl does some other extra stuff as described here: Different signatures when using C routines and openssl dgst, rsautl commands

So if you use the command line I provided (use echo -n to avoid adding a newline) and change your code to sizeof(text)-1 to skip the null terminator, you should get the same output.