Why are the RSA-SHA256 signatures I generate with OpenSSL and Java different?

Question

I want to generate a RSA-SHA256 signature in Java, but I can't get it to produce the same signature as with OpenSSL on the console.

This is what I did with OpenSSL (following this tutorial):

Generate key pair:

openssl genrsa -out private.pem 1024 

Extract public key:

openssl rsa -in private.pem -out public.pem -outform PEM -pubout 

Create hash of data:

echo 'data to sign' > data.txt openssl dgst -sha256 < data.txt > hash 

The generated hash file starts with (stdin)= what I removed by hand (first forgot to mention it, thanks mata).

Sign hash:

openssl rsautl -sign -inkey private.pem -keyform PEM -in hash  > signature 

To reproduce the results in Java I first converted the private key from PEM to DER:

openssl pkcs8 -topk8 -inform PEM -outform DER -in private.pem -nocrypt > private.der 

Now I wrote this Java class to generate the same signature:

public class RSATest {      public static void main(String[] args) throws IOException,             NoSuchAlgorithmException, InvalidKeySpecException,             InvalidKeyException, SignatureException {          byte[] encodedPrivateKey = readFile("private.der");         byte[] content = readFile("data.txt");          KeyFactory keyFactory = KeyFactory.getInstance("RSA");         PKCS8EncodedKeySpec keySpec = new PKCS8EncodedKeySpec(encodedPrivateKey);         RSAPrivateKey privateKey = (RSAPrivateKey) keyFactory                 .generatePrivate(keySpec);          Signature signature = Signature.getInstance("SHA256withRSA");         signature.initSign(privateKey);         signature.update(content);         byte[] signatureBytes = signature.sign();          FileOutputStream fos = new FileOutputStream("signature-java");         fos.write(signatureBytes);         fos.close();     }      private static byte[] readFile(String filename) throws IOException {         File file = new File(filename);         BufferedInputStream bis = new BufferedInputStream(new FileInputStream(                 file));         byte[] bytes = new byte[(int) file.length()];         bis.read(bytes);         bis.close();         return bytes;     }  } 

Unfortunately the results are not the same, so I think I must have done something wrong, but I can't figure out what. Can someone of you help me to find the bug?

Answer 1

openssl dgst -sha256 < data.txt 

produces something like:

 (stdin)= b39eaeb437e33087132f01c2abc60c6a16904ee3771cd7b0d622d01061b40729 

notice the (stdin)='? you don't want that to be part of your hash, if you need to create a digest, use the -binary option.

try using this to sign your data:

openssl sha -sha256 -sign private.pem < data.txt 

This does everything you need.

---

edit - a little more explanations:

let's create a digest and show it

$ openssl dgst -sha256 -binary < data.txt > digest $ hd digest 00000000  26 3b 0a a1 2e b9 32 db  b8 dc d3 6f 37 94 0b 05  |&;....2....o7...| 00000010  71 9c ba 79 46 34 28 9f  5c 5b 98 9a 64 61 c9 ec  |q..yF4(.\[..da..| 

now we take this digest and sign int using rsautl:

$ openssl rsautl -sign -inkey private.pem < digest > sign1 $ hd sign1 00000000  1b 7a cf a4 8d 41 8e 04  ed 3a bb ba 86 f1 f8 e0  |.z...A...:......| 00000010  df f7 47 3e d7 a7 f4 90  7a 05 f8 7f 45 e5 29 e7  |..G>....z...E.).| 00000020  9f f4 2c 91 97 2f e7 26  69 9f 6a 07 a3 48 1b 85  |..,../.&i.j..H..| 00000030  2e f8 ee 44 4d 25 9f ae  05 95 81 c9 e3 07 68 ad  |...DM%........h.| 

now let's sign the same file using dgst directly:

$ openssl dgst -sha256 -sign private.pem < data.txt > sign2 $ hd sign2 00000000  15 c2 94 87 eb e6 cb 45  c8 63 0c 97 60 d3 07 f3  |.......E.c..`...| 00000010  dc 65 32 ad 44 1c c2 2a  7f a3 e1 fc dd 84 27 8c  |.e2.D..*......'.| 00000020  77 a6 97 2b 33 6b c6 d7  7d e1 1d 39 5c 48 b6 48  |w..+3k..}..9\H.H| 00000030  cb 18 be bf 6a 66 90 d3  88 89 52 6c dd d1 b9 99  |....jf....Rl....| 

So what's different here? To see that, we can verify the signature and show the raw output. Both files do contain the digest, but the metadata and padding is different:

$ openssl rsautl -raw -verify -inkey private.pem < sign1 | hd 00000000  00 01 ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |................| 00000010  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff 00  |................| 00000020  26 3b 0a a1 2e b9 32 db  b8 dc d3 6f 37 94 0b 05  |&;....2....o7...| 00000030  71 9c ba 79 46 34 28 9f  5c 5b 98 9a 64 61 c9 ec  |q..yF4(.\[..da..|  $ openssl rsautl -raw -verify -inkey private.pem < sign2 | hd 00000000  00 01 ff ff ff ff ff ff  ff ff ff ff 00 30 31 30  |.............010| 00000010  0d 06 09 60 86 48 01 65  03 04 02 01 05 00 04 20  |...`.H.e....... | 00000020  26 3b 0a a1 2e b9 32 db  b8 dc d3 6f 37 94 0b 05  |&;....2....o7...| 00000030  71 9c ba 79 46 34 28 9f  5c 5b 98 9a 64 61 c9 ec  |q..yF4(.\[..da..| 

To see this more clearly, we can try to use the -asn1parse flag, which won't work for the first signature, but for the second it shows the correct structure of the signature:

$ openssl rsautl -verify -inkey private.pem -asn1parse < sign1 Error in encoding 139931349546656:error:0D07209B:asn1 encoding routines:ASN1_get_object:too long:asn1_lib.c:142:  $ openssl rsautl -verify -inkey private.pem -asn1parse < sign2     0:d=0  hl=2 l=  49 cons: SEQUENCE               2:d=1  hl=2 l=  13 cons:  SEQUENCE               4:d=2  hl=2 l=   9 prim:   OBJECT            :sha256    15:d=2  hl=2 l=   0 prim:   NULL                  17:d=1  hl=2 l=  32 prim:  OCTET STRING             0000 - 26 3b 0a a1 2e b9 32 db-b8 dc d3 6f 37 94 0b 05   &;....2....o7...       0010 - 71 9c ba 79 46 34 28 9f-5c 5b 98 9a 64 61 c9 ec   q..yF4(.\[..da..