How to Calculate Fingerprint From SSH RSA Public Key in Java?

Question

As title, How to Calculate Fingerprint From SSH RSA Public Key in Java? I got an rsaPublicKey object from sample.pub and I calculated the fingerprint by using library Apache Commons Codec DigestUtils.sha256Hex(rsaPublicKey.getEncoded()); But I got a different fingerprint when using ssh-keygen command ssh-keygen -E sha256 -lf sample.pub sample.pub as below ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAsuVPKUpLYSCNVIHD+e6u81IUznkDoiOvn/t56DRcutRc4OrNsZZ+Lmq49T4JCxUSmaT8PeLGS/IC946CNQzFwMh++sVoc19UUkZtRaDgiYn+HkYk8VW4IFI1dKfXomKSbX/lB+ohzLzXLVP2/UJgfBmdaE10k+6b+/Yd8YGXIeS8/Z9zToHPo0ORNSGIolgq3xMXUtfAOK/0KC6IFc/FuvuOSAG1UWup91bcm5GSXv4BWWjgFtOxCLIknYjsDah4qfrP8Olp5eUDhn/65xRcZsmRXoYe1ylhlSjJoPDFWXVs9npwqQmi3JaZtgg7xJxMu1ZcdpYxoj280zM9/6w1Lw==

Answer 1

Your main problem is that the XDR-style encoding used by SSH for publickey, which OpenSSH uses to compute the fingerprint, is not the same as the encoding used by Java crypto, which is an ASN.1 DER format defined by X.509 formally called SubjectPublicKeyInfo. In fact I'm very surprised you were able to read an OpenSSH .pub file in Java; there is no direct way to do so. See numerous existing Qs on this at ssh-keygen and openssl gives two different public keys (disclosure: mine) but on a quick check I don't think any of them are Java so you'll need to do something like:

byte[] n = rsapubkey.getModulus().toByteArray(); // Java is 2sC bigendian
byte[] e = rsapubkey.getPublicExponent().toByteArray(); // and so is SSH
byte[] tag = "ssh-rsa".getBytes(); // charset very rarely matters here
ByteArrayOutputStream os = new ByteArrayOutputStream();
DataOutputStream dos = new DataOutputStream(os);
dos.writeInt(tag.length); dos.write(tag);
dos.writeInt(e.length); dos.write(e);
dos.writeInt(n.length); dos.write(n);
byte[] encoded = os.toByteArray();
// now hash that (you don't really need Apache) 
// assuming SHA256-base64 (see below)
MessageDigest digest = MessageDigest.getInstance("SHA256");
byte[] result = digest.digest(encoded);
String output = Base64.getEncoder().encodeToString(result);

(Aside: Thanks linc01n for catching the bug -- I try to always compile before posting and I'm not sure how I missed this one.)

The second problem is that OpenSSH has never displayed SHA256 fingerprints in hex. It originally used MD5 fingerprints in hex with colons; in 6.8 it switched by default to SHA256 in base64 (using the traditional alphabet not the 'URLsafe' one preferred by JSON) although you can still get the older form (in ssh use -oFingerprintHash=md5 or the equivalent config setting; in ssh-keygen -l use -E md5). Determine which one(s?) you want and code accordingly.

Or, if you have the .pub file, just read the second space-separated field of the one line, convert from base64 to byte[], hash that, and display.