Encrypting AES key with RSA public key

Question

I'm writing a small application for transferring files, more or less as a way to learn more of the programmatic encryption underpinnings. The idea is to generate an RSA keypair, exchange public keys, and send the AES iv and key over for further decryption. I want to encrypt the AES key with the receivers RSA public key, like so:

// encode the SecretKeySpec
private byte[] EncryptSecretKey ()
{
    Cipher cipher = null;
    byte[] key = null;

    try
    {
        cipher = Cipher.getInstance("RSA/ECB/NOPADDING");
        // contact.getPublicKey returns a public key of type Key
        cipher.init(Cipher.ENCRYPT_MODE, contact.getPublicKey() );
        // skey is the SecretKey used to encrypt the AES data
        key = cipher.doFinal(skey.getEncoded());
    }
    catch(Exception e )
    {
        System.out.println ( "exception encoding key: " + e.getMessage() );
        e.printStackTrace();
    }
    return key;
}

I then write out the key value to the receiver, and decrypt it like so:

private SecretKey decryptAESKey(byte[] data )
{
    SecretKey key = null;
    PrivateKey privKey = null;
    Cipher cipher = null;

    System.out.println ( "Data as hex: " + utility.asHex(data) );
    System.out.println ( "data length: " + data.length );
    try
    {
        // assume this loads our private key
        privKey = (PrivateKey)utility.loadLocalKey("private.key", false);

        cipher = Cipher.getInstance("RSA/ECB/NOPADDING");
        cipher.init(Cipher.DECRYPT_MODE, privKey );
        key = new SecretKeySpec(cipher.doFinal(data), "AES");

        System.out.println ( "Key decrypted, length is " + key.getEncoded().length );
        System.out.println ( "data: " + utility.asHex(key.getEncoded()));
    }
    catch(Exception e)
    {
        System.out.println ( "exception decrypting the aes key: " + e.getMessage() );
        e.printStackTrace();
        return null;
    }

    return key;
}

In console, on the other side, I get this as output:

read_bytes for key: 16
data length: 16
Data as hex: <hex string>
Key decrypted, length is 256
java.security.InvalidKeyException: Invalid AES key length: 256 bytes

Furthermore, if I create a byte array of size 16 and put the cipher.doFinal(data) output into it, the array is seemingly resized to 256 bytes (.length says so, at least). Why would this be, and further, what am I doing incorrectly?

edit
I solved this, and thought I'd post the issue in case someone runs into this. The problem, it turns out, was the RSA/ECB/NOPADDING. For some odd reason it was screwing up my creation of the SecretKey when I transferred it over to the client. It might have something to do with how I'm generating keypairs (i'm using getInstance("RSA") for that), but I'm not entirely sure.

Answer 1

As owlstead mentioned, you cannot just use "raw" RSA without padding for encryption/decryption. For one it is very insecure, and for another, the Java libraries do not even support it. Below is the working code for the encryption/decryption of the AES key using RSA keypairs.

private byte[] EncryptSecretKey ()
{
    Cipher cipher = null;
    byte[] key = null;

    try
    {
        // initialize the cipher with the user's public key
        cipher = Cipher.getInstance("RSA/ECB/PKCS1Padding");
        cipher.init(Cipher.ENCRYPT_MODE, contact.getPublicKey() );
        key = cipher.doFinal(skey.getEncoded());
    }
    catch(Exception e )
    {
        System.out.println ( "exception encoding key: " + e.getMessage() );
        e.printStackTrace();
    }
    return key;
}

Decryption of the AES key looks like this:

private SecretKey decryptAESKey(byte[] data )
{
    SecretKey key = null;
    PrivateKey privKey = null;
    Cipher cipher = null;

    try
    {
        // this is OUR private key
        privKey = (PrivateKey)utility.loadLocalKey(
                                ConfigFrame.privateKeyLocation, false);

        // initialize the cipher...
        cipher = Cipher.getInstance("RSA/ECB/PKCS1Padding");
        cipher.init(Cipher.DECRYPT_MODE, privKey );

        // generate the aes key!
        key = new SecretKeySpec ( cipher.doFinal(data), "AES" );
    }
    catch(Exception e)
    {
        System.out.println ( "exception decrypting the aes key: " 
                                               + e.getMessage() );
        return null;
    }

    return key;
}

Answer 2

You cannot just use "raw" RSA to encrypt data without any padding. You need some kind of padding scheme, if only for security reasons. Normally "RSA/ECB/PKCS1Padding" is used. This can encrypt data up to 11 bytes less than the key size. It makes sure that the data fits in the modulus, and adds at least 8 bytes of random data to make sure that encrypting e.g. the word "Yes" twice does not result in two identical cipher texts. Finally, it makes sure you can find out the size in octets of the encrypted data, so you can simply encrypt the 16, 24 or 32 bytes that make up the AES key.