Codec.Crypto.RSA: (decrypt . encrypt) /= id when PKCS#1 v1.5 padding is used?

Question

I'm using Codec.Crypto.RSA to encrypt a random string that is passed to an external processes over a socket in base64 representation. The external process (a ruby program using openssl for decryption) sometimes fails to decrypt the message.

To debug this I set up a simple script in haskell that encrypts and decrypts a fixed message, all without base64 encoding/decoding. What baffles me is that this very simple program results in a failure after a few iterations. The decrypted ciphertext does not equal the original message, though the message is contained in the decryption (after some unprintable characters).

Here's the code:

import Crypto.Random
import qualified Codec.Crypto.RSA as RSA
import qualified Data.ByteString.Lazy.Char8 as L

m :: L.ByteString
m = L.pack "11111222223333344444555556666600"

main = do
  gen <- newGenIO :: IO SystemRandom
  let (pub, priv, _) = RSA.generateKeyPair gen 1024
  doStuff pub priv

doStuff pub priv = do
  gen <- newGenIO :: IO SystemRandom
  let (e,_) = RSA.encrypt' RSA.UsePKCS1_v1_5 gen pub m
  let d = RSA.decrypt' RSA.UsePKCS1_v1_5 priv e

  if (m == d)
    then do
      putStrLn "SUCCESS"
      doStuff pub priv
    else do
      putStrLn "FAILED"
      putStrLn $ "expected: " ++ show m
      putStrLn $ "got:      " ++ show d

As the test suite for Codec.Crypto.RSA passes, there certainly must be a problem with my program.

After replacing RSA.encrypt' RSA.UsePKCS1_v1_5 with RSA.encrypt (defaulting to OAEP¹) and RSA.decrypt' RSA.UsePKCS1_v1_5 with RSA.decrypt, the failure is no longer triggered.

Does anyone see what's wrong here?

---

[1] I plan to use OAEP later, but the generated ciphertext cannot be decrypted with echo ciphertext | openssl rsautl -oaep -inkey keypair.pem -decrypt for some reason, but that's another problem.

Update: To make OAEP work with OpenSSL one has to use SHA-1 as the hash function:

cryptOptions :: RSA.EncryptionOptions
cryptOptions = RSA.UseOAEP sha1' (RSA.generate_MGF1 sha1') BS.empty
  where sha1' = bytestringDigest . sha1

-- then, to encrypt
enc = RSA.encrypt' cryptOptions gen pubkey

Answer 1

There's nothing wrong with your code, it's a bug in a used library.

The problem is that

generate_random_bytestring :: CryptoRandomGen g => g -> Int64 -> (ByteString, g)
generate_random_bytestring g 0 = (BS.empty, g)
generate_random_bytestring g x = (BS.cons' first rest, g'')
 where
  (rest, g')   = generate_random_bytestring g (x - 1)
  (first, g'') = throwLeft $ crandomR (1,255) g'

which is supposed to generate a random ByteString of given length with no 0-byte, doesn't.

Hacking the source of Codec.Crypto.RSA to test for it, I consistently get a 0-byte error very soon.

That means the decoded message is thought to be longer than it actually is and you get some rubbish in front of it.

The concrete bug is that crandomR sometimes produces a 0-byte due to a bug in crandomR_Num:

        Right (bs, g') ->
                let res = fromIntegral $ fromIntegral low + (bs2i bs .&. mask)
                in if res > high then go g' else Right (res, g')

Here, mask is 0xFF (255), low is 1. If the generated unrestricted byte is 255, then

res = fromIntegral 256

which is 0 and thus not > high.

The bug should be has been fixed in the next release (0.4.1) of monadcryptorandom which is due soon already on hackage.

The OAEP methods are not affected, as far as I can see, because they use a different padding scheme to fill the chunks to the required length.