How to check if the decryption is correct?

Question

I'm working on chat room that encrypt messages for more than one users and each user might have different encryption and key/password. The user's key won't work with all the message, therefore; return errors.

var message ="secret message";
var encrypted = CryptoJS.AES.encrypt(message, "Secret Passphrase");
try {
    var decrypted = CryptoJS.AES.decrypt(encrypted, "Secret Passphrase123").toString(CryptoJS.enc.Utf8);
    if (decrypted.length > 0) {
        alert(decrypted);
    } else {
        alert("false");
    }
} catch(e) {
    alert("false");
}

I'm currently catching the error, but sometimes the decryption returns with jumbled up letters and symbols. The current way I'm doing this is not efficient. Is there a better way?

Answer 1

What is needed it authenticated encryption, see Wikipedia.

Essentially HMAC the encrypted data with the encryption key and append the result to the encrypted data.

Prior to decryption HMAC the encrypted data with the decryption key and compare to the appended HMAC value. (Use a constant time comparison function.)

Answer 2

Put a prefix at the beginning of each message, e.g.

encrypt("ABCD" + message);

Then when you decrypt, check if the result begins with "ABCD".

This isn't foolproof, since it's possible that an incorrect decryption could have the same prefix, but it's very unlikely (make it longer to reduce the chance).

However, there's a security implication, since if someone knows that every message begins with the same prefix, it can help them analyze and try to break the encryption (similar to how you can use known letter frequencies when trying to break a simple Caesar cypher).

A slightly better method would be to generate a random string, and then put it in two places in the string before encrypting. Then check that they match.

random = random_string(10);
encrypt(random + message + random);

Then after decrypting, check if the first and last 10 characters match.