How to encrypt/decrypt long input messages with RSA? [Openssl, C]

Question

I wrote a simple test program that encrypts/decrypts a message.

I have a keylength:

int keylength = 1024; // it can also be 2048, 4096

and max input length:

int maxlen = (keylength/8)-11;

and I know that my input size should be < than maxlen, something like this:

if(insize >= maxlen)
        printf("cannot encrypt/decrypt!\n");

My question is simple - is it possible (if so, how can I do this) to encrypt/decrypt with RSA messages LONGER than maxlen?

Main code is also, very simple but works only when insize < maxlen:

   if((encBytes=RSA_public_encrypt(strlen(buff1)+1, buff1, buff2, keypair, RSA_PKCS1_PADDING)) == -1)
    {
        printf("error\n");
    }

    if((decBytes=RSA_private_decrypt(encBytes, buff2, buff3, keypair, RSA_PKCS1_PADDING)) == -1)
    {
        printf("error\n");
    }

Answer 1

Encrypting long messages requires combined scheme - RSA algorithm encrypts session key (i.e. AES key), and data itself is encrypted with that key. I would recommend to not invent another bicycle and use well established scheme, i.e. PKCS#7/CMS or OpenPGP, depending on your needs.

Answer 2

You would be able to encrypt long messages with RSA the same way as it is done with block ciphers. That is, encrypt the messages in blocks and bind the blocks with an appropriate chaining mode. However, this is not the usual way to do it and you won't find support for it (RSA chaining) in the libraries you use.

Since RSA is quite slow, the usual way to encrypt large messages is using hybrid encryption. In hybrid encryption you use a fast symmetric encryption algorithm (like AES) for encrypting the data with a random key. The random key is then encrypted with RSA and send along with the symmetric key encrypted data.

EDIT:

As fore your implementation, you have insize = 1300 and keylength = 1024 which gives maxlen = 117. To encrypt the full message you those needs 12 encrypts, that each produce 128 bytes, giving an encrypted size of 1536 bytes. In your code you only allocates buffers of 1416 bytes. Also, you don't seem to allow for 128 bytes output as you only increment with 117 in:

RSA_public_encrypt(maxlen, buff1+i, buff2+i, keypair, RSA_PKCS1_PADDING)

and

i += maxlen;