RSA Sign: OpenSSL

Question

I am trying to code with EVP interface of Openssl for RSA signing with SHA1. Later I would like to expand signing with different digest algorithms and different signature algorithms(sor of generic, and therefore the use of EVP).

I seem to get a segmentation fault whenever I try to retrieve the size of my private key

Can someone tell me how to correct this?

int rsaSign(char *in_file, char * sig_file){

        char *data = NULL;
        int data_len;
        unsigned int sig_len;
        unsigned char *sig;
        int err = -1;

        OpenSSL_add_all_digests();
        FILE *fd;
        EVP_PKEY *priv_key = EVP_PKEY_new();
        RSA *privkey = NULL;
           printf( "we are here..\n");

        if ((fd = fopen(PRIVKEY_FILE, "r")) == NULL){
            printf("error reading file\n");
            exit(0);
        }

        privkey = RSA_new();
        if (!PEM_read_PrivateKey(fd, &privkey, NULL, NULL))
        {
            fprintf(stderr, "Error loading RSA Private Key File.\n");
            return 2;
        }

        fclose(fd);

        if (!EVP_PKEY_assign_RSA (priv_key, privkey))
        {
            fprintf(stderr, "EVP_PKEY_assign_RSA: failed.\n");
            return 3;
        }

        if (!priv_key) {
            printf("no private key\n");
        }
        EVP_PKEY_set1_RSA(privkey, priv_key);


        EVP_MD_CTX *ctx = EVP_MD_CTX_create();

        const EVP_MD *md = EVP_get_digestbyname("SHA1");

        if (!md) {
            fprintf(stderr, "Error creating message digest");
            fprintf(stderr, " object, unknown name?\n");
            ERR_print_errors_fp(stderr);
            exit(1);
        }



        if (!EVP_SignInit(ctx, md))
            {
                fprintf(stderr, "EVP_SignInit: failed.\n");
                EVP_PKEY_free(priv_key);
                return 3;
            }
        printf( "now to sign update..\n");
        data = readFile(in_file);
        data_len = strlen(data);
        printf("data len = %d\n", data_len);

        if (!EVP_SignUpdate(ctx, data, data_len))
        {
            fprintf(stderr, "EVP_SignUpdate: failed.\n");
            EVP_PKEY_free(priv_key);
            return 3;
        }
        printf( "now to sign final..\n");

sig = malloc(EVP_PKEY_size(privkey)); //!!!!! SEGMENTATION FAULT HERE !!!!!


        if (!EVP_SignFinal(ctx, &sig, &sig_len, priv_key))
        {
            fprintf(stderr, "EVP_SignFinal: failed.\n");
            free(sig);
            EVP_PKEY_free(priv_key);
            return 3;
        }

        free(data);
        free(sig);
        EVP_PKEY_free(priv_key);
        return EXIT_SUCCESS;

}

Answer 1

You have several problems:

• PEM_read_PrivateKey() expects an EVP_PKEY **, not an RSA ** - in fact, you do not need an RSA * object at all;
• You should not be calling EVP_PKEY_assign_RSA();
• You should not be calling EVP_PKEY_set1_RSA().

To load the private key, you can simply do:

EVP_PKEY *priv_key = NULL;

if (!PEM_read_PrivateKey(rsa_pkey_file, &priv_key, NULL, NULL))
{
    fprintf(stderr, "Error loading Private Key File.\n");
    return 2;
}

You may now call EVP_PKEY_size(priv_key) to determine the key size.

Additionally, your EVP_SignFinal() call is wrong - you should be passing sig, not &sig.

Answer 2

I'm no expert in that particular library, but having perused the API docs, I think this:

EVP_PKEY_set1_RSA(privkey, priv_key);

should be:

EVP_PKEY_set1_RSA(priv_key, privkey);

and similarly shouldn't the line you've highlighted as SEGFAULTing refer to priv_key not privkey also:

sig = malloc(EVP_PKEY_size(priv_key));

Both these mistakes would be easier to see if you change your variable names to something easier to distinguish (rather than privkey and priv_key, use priv_key_RSA and priv_key_EVP

Having variables which differ only in their underscores is a really bad idea :)