Signed message using OpenSSL; can't verify with Android Java

Question

I'm using SHA256 and RSA to sign a message on my Ubuntu machine using OpenSSL. My goal is to verify this message on Android using Android's Java.

Following commands were used on ubuntu:

openssl genrsa -out private.pem 1024 
openssl rsa -in private.pem -out public.pem -outform PEM -pubout
echo 'foobar' > data.txt
openssl dgst -sha256 < data.txt > hash
openssl rsautl -sign -inkey private.pem -keyform PEM -in hash  > signature
openssl rsa -in private_key.pem -pubout -outform DER -out public_key.der
openssl enc -base64 -in signature -out base64_signature

I have now created keys, signed the message, created a .der file for the public key that should be able to be accessed in Java and encoded the message with Base64. I then place the .der public key on my device and successfully load the key into the class PublicKey.

This method is used to verify the message:

public static boolean verify(PublicKey publicKey,String data,String verification){
    java.security.Signature sig;
    try {
        sig = java.security.Signature.getInstance("SHA256WithRSA");
        sig.initVerify(publicKey);
        try {
            sig.update(verification.getBytes());
        } catch (Exception e) {
            ...
        }

        if (!sig.verify(Base64.decode(data, Base64.DEFAULT))) {
            return false;
        }
        return true;
    }
    catch ....
    return false;
}

Parameters when calling the method:

verify(PublicKey, Base64 encoded data in a String that is to be verified, "foobar");

Obviously the verification fails, but I can't understand why. I'm guessing it has to do something with the encoding(?).

---

Update! So I managed to write the results of Base64.decode(data, Base64.DEFAULT))to a file and compare it with the original signature file using a hexeditor. Completely different!

Answer 1

Java produces and expects to receive signatures in slightly different form. Hash of the message must be encoded in DER, then padded with PKCS#1 and only then signed with private key. And Openssl has a command for that (because it's actually a standard procedure). Instead of

openssl dgst -sha256 < data.txt > hash
openssl rsautl -sign -inkey private.pem -keyform PEM -in hash  > signature

you do

openssl dgst -sha256 -binary -sign private.pem data.txt > signature

Also note:

• your data.txt contains a newline, don't forget it in String verification variable
• sig.update(verification.getBytes()) should explicitly indicate a charset - the same charset, that was used to fill the data.txt file, for example: sig.update(verification.getBytes("UTF-8"))

The rest of your commands/code seems OK.

---

UPD - to answer @GilCol about the differences:

The padding is the same for both signed messages (PKCS#1). But the messages are different.

When you use openssl dgst -sha256 < data.txt > hash, hash will contain (depending on openssl version):

(stdin)= aec070645fe53ee3b3763059376134f058cc337247c978add178b6ccdfb0019f

or

aec070645fe53ee3b3763059376134f058cc337247c978add178b6ccdfb0019f

It is just plain text and it is the message you will sign using openssl rsautl -sign .... We can see that with openssl rsautl -verify ...:

# raw message as-is - we can see the padding
$ openssl rsautl -in signature -pubin -inkey public.pem -verify -raw -hexdump
0000 - 00 01 ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0010 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0020 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0030 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0040 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0050 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0060 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0070 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0080 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0090 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
00a0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
00b0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff 00 61   ...............a #
00c0 - 65 63 30 37 30 36 34 35-66 65 35 33 65 65 33 62   ec070645fe53ee3b #
00d0 - 33 37 36 33 30 35 39 33-37 36 31 33 34 66 30 35   3763059376134f05 # your plain-text message
00e0 - 38 63 63 33 33 37 32 34-37 63 39 37 38 61 64 64   8cc337247c978add #
00f0 - 31 37 38 62 36 63 63 64-66 62 30 30 31 39 66 0a   178b6ccdfb0019f. # we can even see newline char (0a) at the end

# strip the padding
$ openssl rsautl -in signature -pubin -inkey public.pem -verify -pkcs -hexdump
0000 - 61 65 63 30 37 30 36 34-35 66 65 35 33 65 65 33   aec070645fe53ee3
0010 - 62 33 37 36 33 30 35 39-33 37 36 31 33 34 66 30   b3763059376134f0
0020 - 35 38 63 63 33 33 37 32-34 37 63 39 37 38 61 64   58cc337247c978ad
0030 - 64 31 37 38 62 36 63 63-64 66 62 30 30 31 39 66   d178b6ccdfb0019f
0040 - 0a                                                .

If you use openssl dgst -sha256 -binary < data.txt > hash to get hash in binary (pure) form, and then sign it, the result will be better, but still not right:

# raw message as-is - we can see the same padding
$ openssl rsautl -in signature -pubin -inkey public.pem -verify -raw -hexdump
0000 - 00 01 ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0010 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0020 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0030 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0040 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0050 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0060 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0070 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0080 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0090 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
00a0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
00b0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
00c0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
00d0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff 00   ................
00e0 - ae c0 70 64 5f e5 3e e3-b3 76 30 59 37 61 34 f0   ..pd_.>..v0Y7a4. # the hash - now in binary form
00f0 - 58 cc 33 72 47 c9 78 ad-d1 78 b6 cc df b0 01 9f   X.3rG.x..x...... #

# strip the padding
$ openssl rsautl -in signature -pubin -inkey public.pem -verify -pkcs -hexdump
0000 - ae c0 70 64 5f e5 3e e3-b3 76 30 59 37 61 34 f0   ..pd_.>..v0Y7a4. # just the hash, nothing else
0010 - 58 cc 33 72 47 c9 78 ad-d1 78 b6 cc df b0 01 9f   X.3rG.x..x...... #

But when you use openssl dgst -sha256 -sign ..., the message is different - it's now a standard ASN.1 structure for message digests (hashes). Let's see:

# raw message as-is - we can see the same padding
$ openssl rsautl -in signature -pubin -inkey public.pem -verify -raw -hexdump
0000 - 00 01 ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0010 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0020 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0030 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0040 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0050 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0060 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0070 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0080 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
0090 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
00a0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
00b0 - ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff   ................
00c0 - ff ff ff ff ff ff ff ff-ff ff ff ff 00 30 31 30   .............010 #
00d0 - 0d 06 09 60 86 48 01 65-03 04 02 01 05 00 04 20   ...`.H.e.......  # the message - it's different
00e0 - ae c0 70 64 5f e5 3e e3-b3 76 30 59 37 61 34 f0   ..pd_.>..v0Y7a4. # <- we can see the hash (in binary form) starting at this line
00f0 - 58 cc 33 72 47 c9 78 ad-d1 78 b6 cc df b0 01 9f   X.3rG.x..x...... #

# strip the padding
$ openssl rsautl -in signature -pubin -inkey public.pem -verify -pkcs -hexdump
0000 - 30 31 30 0d 06 09 60 86-48 01 65 03 04 02 01 05   010...`.H.e.....
0010 - 00 04 20 ae c0 70 64 5f-e5 3e e3 b3 76 30 59 37   .. ..pd_.>..v0Y7
0020 - 61 34 f0 58 cc 33 72 47-c9 78 ad d1 78 b6 cc df   a4.X.3rG.x..x...
0030 - b0 01 9f                                          ...

# parse the message and show the underlying ASN.1 structure
$ openssl rsautl -in signature -pubin -inkey public.pem -verify -pkcs -asn1parse 
    0:d=0  hl=2 l=  49 cons: SEQUENCE          
    2:d=1  hl=2 l=  13 cons:  SEQUENCE          
    4:d=2  hl=2 l=   9 prim:   OBJECT            :sha256               # type of hash
   15:d=2  hl=2 l=   0 prim:   NULL              
   17:d=1  hl=2 l=  32 prim:  OCTET STRING      
      0000 - ae c0 70 64 5f e5 3e e3-b3 76 30 59 37 61 34 f0   ..pd_.>..v0Y7a4. # the hash in binary form
      0010 - 58 cc 33 72 47 c9 78 ad-d1 78 b6 cc df b0 01 9f   X.3rG.x..x...... # and no extra newline chars

As you can see, only the last signature file had proper ASN.1 structure, previous two were just "some arbitrary" messages, signed with RSA private key.