RSA in javascript no longer supports ASCII/byte arrays

Question

I am using rsa.js v1.0 from http://www-cs-students.stanford.edu/~tjw/jsbn/ to encrypt an ASCII string in a browser. The string is actually a 16 byte array that contains a double-length TripleDes key. With rsa v1.0 this works. The byte array is correctly decrypted on the server (using Bouncy Castle or a Thales HSM) as a 16 byte array.

e.g.

var zpk = hex2a("E0F8AD4092F81FC401E60ECB7F5B8F1A");
var rsa = new RSAKey();
rsa.setPublic(modulus, exponent);
var res = rsa.encrypt(zpk);
if (res) {
    document.rsatest.zpkrsa.value = hex2b64(res);
}

When moving the rsa.js v1.4 this not longer works. Bouncy castle decrypts the data, but instead of a 16 byte array, it is now a 25 byte array.

The key difference I can see in the rsa.js library is in the v1.1 release notes:

Added support for utf-8 encoding of non-ASCII characters when PKCS1 encoding and decoding JavaScript strings.

The PKCS#1 padding in v1.0 is:

// PKCS#1 (type 2, random) pad input string s to n bytes, and return a bigint
function pkcs1pad2(s, n) {
    if (n < s.length + 11) {
        alert("Message too long for RSA");
        return null;
    }
    var ba = new Array();
    var i = s.length - 1;
    while (i >= 0 && n > 0) ba[--n] = s.charCodeAt(i--);
    ba[--n] = 0;
    var rng = new SecureRandom();
    ...
    return new BigInteger(ba);
}

The PKCS#1 padding function in v1.1 and later is:

// PKCS#1 (type 2, random) pad input string s to n bytes, and return a bigint
function pkcs1pad2(s,n) {
  if(n < s.length + 11) { // TODO: fix for utf-8
    console.error("Message too long for RSA");
    return null;
  }
  var ba = new Array();
  var i = s.length - 1;
  while(i >= 0 && n > 0) {
    var c = s.charCodeAt(i--);
    if(c < 128) { // encode using utf-8
      ba[--n] = c;
    }
    else if((c > 127) && (c < 2048)) {
      ba[--n] = (c & 63) | 128;
      ba[--n] = (c >> 6) | 192;
    }
    else {
      ba[--n] = (c & 63) | 128;
      ba[--n] = ((c >> 6) & 63) | 128;
      ba[--n] = (c >> 12) | 224;
    }
  }
  ba[--n] = 0;
  ...
  return new BigInteger(ba);
}

rsa.js v1.0 treated each character as a 1 byte character. Since v1.1 characters are tested to see if they are multi-byte utf-8 or not.

It seems my only options are to either:

1. Stick with rsa.js v1.0
2. Create a modified version of rsa.js (and rsa2.js) that allow me to disable to utf-8 character detection.
3. (Edited) Alter code to use defensivejs.com which supports PKCS#1 v2 (oaep).

Ideas?

Answer 1

1. This code is implementing PKCS #1 v1.5 padding in both cases, the only difference is the utf-8 support. For it to work with a recipient library, that library would need to decode the content the same way he encodes it. Good luck on that, I don't think you will find anything that does that.

2. PKCS #1 v1.5 padding is insecure due to an attack illustrated by Daniel Bleichenbacher around 1999. Nowadays it is recommended to use PKCS #1 v2.x. Wu's code does not support this.

3. If you really wanted to use this library (I recommend against it), probably the cleanest approach is to send in the key hex encoded before you encrypt it ("E0F8AD4092F81FC401E60ECB7F5B8F1A") and make sure recipient hex decodes it after decrypt: that would work around Wu's UTF-8 adjustments. You could also use base64 encoding/decoding.

4. SJCL is a much better JavaScript crypto library, and you will unlikely run into problems like this. To my knowledge, Wu's code was designed as a PoC of his wonderful authentication protocol, whereas SJCL is designed for more general use and is being maintained by a community.