RSA Decryption exception: The length of the data to decrypt is not valid for the size of this key

Question

I have got an Angular + Net Core application with an (RSA + AES) encrypted connection. All requests from client are coming via POST. (You will be given an example below.

The script provided below works quite well but throws in 5% cases an exception:

The length of the data to decrypt is not valid for the size of this key in the line:

var decryptedAesKey = Encoding.UTF8.GetString(rsaCng.Decrypt(Convert.FromBase64String(request.k), RSAEncryptionPadding.Pkcs1));

Encryption part (Front-end)

encrypt(requestObj:any):any {  
var rsaEncrypt = new JsEncryptModule.JSEncrypt();
var key = this.generateAesKey(32); //secret key
var iv = this.generateAesKey(16); //16 digit

var stringifiedRequest = CryptoJS.enc.Utf8.parse(JSON.stringify(requestObj));
var aesEncryptedRequest = CryptoJS.AES.encrypt(stringifiedRequest, 
CryptoJS.enc.Utf8.parse(key), 
{ 
  keySize: 128 / 8,
  iv: CryptoJS.enc.Utf8.parse(iv),
  padding: CryptoJS.pad.Pkcs7,
  mode: CryptoJS.mode.CBC
 });

rsaEncrypt.setPrivateKey(this.publicPemKey);
var encryptedKey = rsaEncrypt.encrypt(key);
var encryptedIV  = rsaEncrypt.encrypt(iv);

var encryptedRequestObj = {
    k: encryptedKey,
    v: encryptedIV,
    r: aesEncryptedRequest.toString()
 };

return encryptedRequestObj;

}

Decryption part (C# Back-end)

var decryptedAesKey = Encoding.UTF8.GetString(rsaCng.Decrypt(Convert.FromBase64String(request.k), 
RSAEncryptionPadding.Pkcs1));
var decryptedAesIV = Encoding.UTF8.GetString(rsaCng.Decrypt(Convert.FromBase64String(request.v), RSAEncryptionPadding.Pkcs1));

byte[] encryptedBytes = request.r;
AesCryptoServiceProvider aes = new AesCryptoServiceProvider()
{
    Mode = CipherMode.CBC,
    Padding = PaddingMode.PKCS7,
    Key = Encoding.UTF8.GetBytes(decryptedAesKey),
    IV = Encoding.UTF8.GetBytes(decryptedAesIV)
};

ICryptoTransform crypto = aes.CreateDecryptor(aes.Key, aes.IV);
byte[] secret = crypto.TransformFinalBlock(encryptedBytes, 0, encryptedBytes.Length);
crypto.Dispose();

requestJson = Encoding.UTF8.GetString(secret);

Example, a user wants to open a page by id.

Front-end:
1) encrypts request Id using AES
2) encrypts AES' key & iv using RSA
3) sends to Back-end

Back-end:
1) decrypts AES' key & value using RSA <--- BREAKS HERE
2) decrypts request Id using AES' key & iv
3) decrypts and get id as if there was no encryption

This logic works quite well but breaks sometimes...

EXAMPLE OF FAILING request:

{ "k":"L+ikMb/JGvFJmhBpADMGTVLFlkHOe69dZUVSQ5r7yHCvWSwY2x6KMR274ByflF0lDMYdCmywo+Nfq6JUybRctDqmAp8UFHXnhwBAv49d99mF5x2yGbJr/j0cn6EZyhweNK4p97i5yMM6MQtluZTIErpsUa22Cajtj8F+xl0jJPUMXIf8cs2X+ooFr5VP/p/vlbPmnEY3K/hMCRZRdXMkEqaCWoA5EnYMTQABtRXPZWgLSQwJpr4dqEAhGCBtga1AGsKF3dQCsKO92NYyst0ngkBiKwFNfy1QDwbk4SzKAKeBckaY17SHt526NMvpEv08BGV6btBxcM+ypsmpB4o0",
"v":"LIndJOjUgKHDlXqwpg7uSmDuut3oi5z9L/GKm2KgU7P2EXmf/JIpXM0JgpTXPJL7wUTndq3F9UMlMdU70JBOV56x/4uIBRbHbyvaG2JZYxbBZblwyYgdo1ZcK1OSE4k5oesQmMEGNEk9RVu+EZO4xAme6+mlyd2/Y/709jaC90PuiOG/k/4JMTTI/2q4s7tk6IgSxLBT8ZiOtgJVGdasSaAksEBMRHyUkzAIr5tSUw1VXedwJFPfwQT2nOD5dU2cxiNJKOwtO9uAYXly0U0FDoa/nkWskca8zaU+4EiPikJ6Km7phViH9JvwZFgHhBj+8FM6Jof+AdrY3q1dcMLFlg==",
"r":"OJnA3wFoKKG+iu4FciXyJg=="
}

EXAMPLE OF CORRECT REQUEST:

{   "k":"uW8d7vIzlgkEkKTkDnHbBZeqKwdgoG+1BVZ/NUiC0pZ/LqZM9aUasQSx+qDg+X50ur30uRnEyAyIZXruYeHQb8cacx5mvr9LWLud+wueJXsOlEEdocD/4A1DfE9TDFdnTaVcMSIwhSVlLPUjO7ubJdANY9yK4S+vb0IyPbsrYpAT7ho01mDkvsH1rZsId/TmzQadmsGhThowu+mrQlz78rrdlN8nI5LnUQHXRNWMUgBvuteTpVBmyrfnIELIKoo/jI6Nj4rGPQBf7+2OOoZPs0Y1GtjXxUCTAt7madNLKSOdaPjdWjaOfGSwnymDNeEFyJQOmAwHZoOGYNd2B/UhQQ==",
"v":"IimiJFcKv5ZHWHljJixX0LUgV4I2GWAWPbk7dWHVhwmHEhTHA/hCdih/E1wiWFS+0KaL05ZobiZInyK7gCwYPHaz0aRCSQtVeBPiFg4f7L0gwfvk1GHwJ1wZjqNJZaYf0elXJzc2l5BwN+aXNWaNJDPA7M6kfK6UPkq84IV3ohCQcTuC8zPM7aMJHxpz9IudcrMmYIkeqrj9Do88CkTLv8yg5hk3EASPk9HqsUieuQixggv/8ZlHnp00iftc62LJlIuCkGn4WR3FkMdFdqpKXf6Ebj8PU1HOmokEtKtYJiOZ5JxieZO5Pnd+ez6sO7khIbdRFDhAQ20chsxKUypezw==",
"r":"2mbUgU44JFFDlWu8As2RIw=="
}

Answer 1

In the case of the failed request, the Base64 decoded encrypted AES key has a length of 255 bytes. For a 2048 bit RSA key it should actually be 256 bytes, as it is for the remaining data.

For the RSA-encryption JSEncrypt is used, which has a known bug that sporadically causes too short ciphertexts and which is probably responsible for your issue, see here. This bug was opened in July 2019 and is not fixed yet.

Within JSEncrypt the too short ciphertexts are processed correctly, so that no error occurs. Cross platform however, this is often not the case, because the too short ciphertexts are strictly speaking invalid and therefore some programming languages identify them as invalid, e.g. Python, apparently C# is another one.

If the too short ciphertext is manually padded from the left to the length of the modulus with 0x00, the ciphertext should also be decryptable in the C# code.

Update:

• I have successfully tested the suggested fix using your code. The ciphertext can be fixed in the JavaScript or C# code. A possible implementation for the JavaScript side is e.g. for the key:

  encryptedKey = btoa(atob(encryptedKey).padStart(256, "\0"));
  

  where encryptedKey is the Base64 encoded ciphertext as returned by JSEncrypt#encrypt. To ensure that this correction isn't applied to ciphertexts that already have the correct length, a length check is useful: A Base64 encoded ciphertext of length 4 * Math.ceil(256 / 3) doesn't need to be fixed because it corresponds to a ciphertext of the correct length of 256 bytes, see here.

• You apply the method setPrivateKey in the JSEncrypt part when setting the public key for the encryption, correct would be setPublicKey, see here. However, JSEncrypt seems to fix this internally, because it works as well. Nevertheless it should be changed, because it's misleading.

• As already mentioned in the comments by @kelalaka, the IV is no secret and doesn't need to be encrypted.