PEM_read_RSAPrivateKey returns "Illegal Seek" when decrypting using OpenSSL libs in C

Question

This problem has been driving me nuts! :) I am generating a couple of public/private key pairs, but when i go to load the private key i get an error. I am using C. Note i do use the password "password" on the keys, it is set as a #define but i have tried putting it in manually as a string with no difference. Here is how i'm generating the keys:

FILE *fp;
OpenSSL_add_all_algorithms();
RSA *rsa=NULL;
unsigned char seed[KEYSIZE];
int i;

//Seed PRNG
srand(time(NULL));
RAND_bytes(seed, KEYSIZE - 1);

//Generate a key
if ((rsa=RSA_generate_key(KEYSIZE,65537,NULL,NULL)) == NULL) ERR_get_error();

//Write the public key
fp = fopen(pubkey,"w");
if (!PEM_write_RSA_PUBKEY(fp, rsa)) {
    printf("Error writing public key\n"); exit(1);
}
fclose(fp);

//Write the private key
fp = fopen(privkey,"w");
if (!PEM_write_RSAPrivateKey(fp, rsa, EVP_des_ede3_cbc(), NULL, 0, NULL, PASSWORD))
{
   printf("Error writing private key\n"); exit(1);
}
fclose(fp);

This completes successfully and i end up with the files in the appropriate places, and they look good. I've also successfully encrypted/decrypted on the command line using these files, so i know they work!

Here is how i'm loading the private key:

static void decrypt(int locale) {
FILE *key; 
RSA *rsa;

key = fopen(PRIVATEKEY, "r");
if (key == NULL) perror("Error reading private key");

rsa = PEM_read_RSAPrivateKey(key, NULL, NULL, PASSWORD); // THIS BREAKS!
if (rsa == NULL) perror("Private Key not valid");  // I always get this error :(

if (rsa != NULL) RSA_free(rsa);
fclose(key);}

Any suggestions? I went through the source file rsa.c and this is basically exactly how they do it in there, except they use BIOs instead of freads. I don't see why my way shouldn't work though!

Here is the relevant section of strace output:

open("/opt/evoting/keys/priv/mix1.priv", O_RDONLY) = 15
fstat64(15, {st_mode=S_IFREG|0644, st_size=1751, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb773c000
read(15, "-----BEGIN RSA PRIVATE KEY-----\n"..., 4096) = 1751
dup(2)                                  = 16
fcntl64(16, F_GETFL)                    = 0x2 (flags O_RDWR)
fstat64(16, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 2), ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb773b000
_llseek(16, 0, 0xbfba00d0, SEEK_CUR)    = -1 ESPIPE (Illegal seek)
write(16, "Private Key not valid: Resource "..., 56Private Key not valid: Resource     temporarily unavailable ) = 56
close(16)                               = 0
munmap(0xb773b000, 4096)                = 0
close(15)                               = 0
munmap(0xb773c000, 4096)                = 0

Also, perror produces the error "resource not available". I'm not sure what that's about, but i reset my hard and soft ulimits to be really high and it did nothing. I copied the function into its own file and compiled it and it doesn't produce the error there (it still fails to read the key, though).

Answer 1

Aha! Alright, i just learned a lot about troubleshooting this stuff, and hopefully it will help someone else along the line. Most of my info came from this thread: http://readlist.com/lists/openssl.org/openssl-users/2/10340.html

Basically i needed to not use perror and instead use ERR_print_errors_fp(stderr). This gave me an actual error message like so:

3077973640:error:0906B072:lib(9):func(107):reason(114):pem_lib.c:530:

I then used the command: openssl errstr 0906B072 and got:

error:0906B072:PEM routines:PEM_get_EVP_CIPHER_INFO:unsupported encryption

Which basically led me back to the creation of my key pair where i specified CBC: EVP_des_ede3_cbc() which is not supported for RSA. So there you go!