RSA Encryption on Windows using openssl C++ / Error in decryption

Question

I generate rsa public/private on Android using :

KeyPairGenerator kpg;
KeyPair kp;
PublicKey publicKey;
PrivateKey privateKey;
kpg = KeyPairGenerator.getInstance("RSA");
kpg.initialize(2048);
kp = kpg.genKeyPair();
publicKey = kp.getPublic();
privateKey = kp.getPrivate();

Then I send the public key to Windows and encrypt data using OpenSSL library :

int resEncryption = RSA_public_encrypt(length, in, encrypted, rsaPkey, RSA_PKCS1_OAEP_PADDING);

Then I transform encrypted to HEX on Windows :

stringstream ss;
for (int i = 0; i < resEncryption; i++)
    ss << std::hex << (int)encrypted[i] << std::setfill('0');

string hexEncrypted = ss.str();

I get hexEncrypted on Android and put it in string :

StringBuilder sb = new StringBuilder();
char[] hexData = hex.toCharArray();
for (int count = 0; count < hexData.length - 1; count += 2) {
    int firstDigit = Character.digit(hexData[count], 16);
    int lastDigit = Character.digit(hexData[count + 1], 16);
    int decimal = firstDigit * 16 + lastDigit;
    sb.append((char)decimal);
}
String hexDecrypted = sb.toString();

And to decrypt this data I do :

byte[] encryptedBytes = hexDecrypted.getBytes();
Cipher cipher1 = Cipher.getInstance("RSA/None/OAEPwithSHA-1andMGF1Padding");//also tried RSA/ECB/OAEPwithSHA-1andMGF1Padding
cipher1.init(Cipher.DECRYPT_MODE, privateKey);
decryptedBytes = cipher1.doFinal(encryptedBytes);

But I get an error in the engineDoFinal method in CipherSpi.class :

protected byte[] engineDoFinal(byte[] input, int inputOffset, int inputLen) throws IllegalBlockSizeException, BadPaddingException {
    if(input != null) {
        this.bOut.write(input, inputOffset, inputLen);
    }

    if(this.cipher instanceof RSABlindedEngine) {
        if(this.bOut.size() > this.cipher.getInputBlockSize() + 1) {
            throw new ArrayIndexOutOfBoundsException("too much data for RSA block");
        }
    } else if(this.bOut.size() > this.cipher.getInputBlockSize()) {
        throw new ArrayIndexOutOfBoundsException("too much data for RSA block");//Error here 
    }

    return this.getOutput();
} 

Answer 1

It looks like I had a problem with the hex conversion with C++, this solved my problem. Now I get hex string on Android with the expected length.

string asciiResEnc(reinterpret_cast<char*>(encrypted), resEncryption);
stringstream ss;
for(int i = 0; i < asciiResEnc.size(); i++)
    ss << std::hex << setw(2) << setfill('0') << (int) (unsigned char)asciiResEnc[i];