Re-Implementing RSA decryption in Java

Question

I'm using a private RSA key to encrypt a random AES key with the default Java RSA implementation:

Cipher cipher = Cipher.getInstance("RSA");
cipher.init(Cipher.ENCRYPT_MODE, privateKey);
byte[] cipherBytes = cipher.doFinal(plainText.getBytes());

Since we need a public key anyway, this is a convenient method to disguise the key and make sure it had been encrypted with our private key. The decryption is done similarly:

Cipher cipher = Cipher.getInstance("RSA");
cipher.init(Cipher.DECRYPT_MODE, publicKey);
byte[] plainBytes = cipher.doFinal(cipherBytes);

This works fine with Oracle's JDK, but with IBM's this fails because IBM thinks using the private key for encryption is not a valid use case. Unfortunately, I have to support both JDKs, so I'm trying to re-implement the RSA decryption myself.

This is the code I have so far:

BigInteger big = new BigInteger(cipherBytes);
big = big.modPow(pub.getPublicExponent(), pub.getModulus()); 
System.out.println(new String(big.toByteArray()));

It almost works, but there seems to be a padding issue. Most of the time I'm getting the original text with a string of dot-like symbols in front of it, but sometimes it's only random bytes.

Unfortunately, I wasn't able to figure out which padding scheme is used by default. Does anyone know what's missing in my code or can at least give a hint with which algorithm the padding is handled?

Here is an example of input and output values, as requested. I have used 512 bit keys to avoid too huge numbers.

Public modulus :  8117919732251191237549784557538073836207094968952416063837701691514861428726690140363567956265691836505266266364256892197254736023284927189008247933889303
Public exponent:  65537
Plaintext:        teststring
Plaintext as BN:  549665952565679142563431
Ciphertext as BN: 6304229782339071167863563708554898540621778162930150363326921290545577949349781053660336996882823758722402137580193903457839924005473545992074817339077456
"Decrypted" BN:   409173825987017733751648712103449894027080255755383098685411421012016724550584319360408761540738019643860835515945008876151848132891805352276483731047
Resultstring: ˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇˇteststring

To address the discussion why am I doing this:

The public key is hard-coded into my software. I use the private key to encrypt another key for AES. Therefore, to actually decode anything with AES, you need the AES key first. To get this key, you have to decrypt it with the public key first. Since the public key cannot be modified without serious manipulation, only AES keys encrypted with the private key work. You may extract the public key somehow and decrypt the AES key, but that's elaborate and only gets you the AES key to decrypt the secured content. There is also a signature calculated with the private key, which is verified with the public key as well. So manipulations aren't possible.

So yes, technically the signature is sufficient, because there are methods to read the content. But those are elaborate and I don't mind if anyone really takes all the trouble, but I don't want to make things easy.

Answer 1

Public keys are for encrypting and verifying signatures. Private keys are for decrypting and signing. Public keys are intended to be just that: public. If you're doing things right, there should be no reason to hide a public key.