BCECPublicKey to fingerprint

Question

When connecting to a "new" SSH server, with the command line, a fingerprint will be shown:

The authenticity of host 'test.com (0.0.0.0)' can't be established.
ECDSA key fingerprint is SHA256:566gJgmcB43EXimrT0exEKfxSd3xc7RBS6EPx1XZwYc.
Are you sure you want to continue connecting (yes/no)?

I understand that the fingerprint is a Base64 string of the SHA256 hash of the public key.

I know how to generate this fingerprint with a RSAPublicKey:

    RSAPublicKey publicKey = ...;

    ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
    DataOutputStream dataOutputStream = new DataOutputStream(byteArrayOutputStream);

    dataOutputStream.writeInt("ssh-rsa".getBytes().length);
    dataOutputStream.write("ssh-rsa".getBytes());
    dataOutputStream.writeInt(publicKey.getPublicExponent().toByteArray().length);
    dataOutputStream.write(publicKey.getPublicExponent().toByteArray());
    dataOutputStream.writeInt(publicKey.getModulus().toByteArray().length);
    dataOutputStream.write(publicKey.getModulus().toByteArray());

    MessageDigest digest = MessageDigest.getInstance("SHA256");
    byte[] result = digest.digest(byteArrayOutputStream.toByteArray());

    String fingerprint = Base64.getEncoder().encodeToString(result);

But how can I do this with a BCECPublicKey?

UPDATE
I found out that the BCECPublicKey isn't similar to RSAPublicKey at all. I never knew that the SSH server public key is ECDSA and the client public key is RSA.

Also the way the bytes are structured is way different. The RSA public key starts with a header(ssh-rsa). The header length can be read from the first 4 bytes(readInt()). But when I do this with the ECDSA the length is way to long to represent a header...

Addition to answer
Something to copy paste:

    BCECPublicKey publicKey = ...;

    byte[] point = SubjectPublicKeyInfo.getInstance(ASN1Sequence.getInstance(publicKey.getEncoded())).getPublicKeyData().getOctets();

    ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
    DataOutputStream dataOutputStream = new DataOutputStream(byteArrayOutputStream);

    dataOutputStream.writeInt("ecdsa-sha2-nistp256".getBytes().length);
    dataOutputStream.write("ecdsa-sha2-nistp256".getBytes());
    dataOutputStream.writeInt("nistp256".getBytes().length);
    dataOutputStream.write("nistp256".getBytes());
    dataOutputStream.writeInt(point.length);
    dataOutputStream.write(point);

    MessageDigest digest = MessageDigest.getInstance("SHA256");
    byte[] result = digest.digest(byteArrayOutputStream.toByteArray());

    String fingerprint = Base64.getEncoder().encodeToString(result);

Answer 1

The OpenSSH publickey format (and the SSH wire format on which it is based) does start with the type, but for ECDSA the type includes the curve id. As an example, one of my test systems has an ecdsa/p256 key as follows:

$ awk '{print $2}' <id_ecdsa.pub |openssl base64 -d -A |xxd
0000000: 0000 0013 6563 6473 612d 7368 6132 2d6e  ....ecdsa-sha2-n
0000010: 6973 7470 3235 3600 0000 086e 6973 7470  istp256....nistp
0000020: 3235 3600 0000 4104 8141 9c28 53e7 532e  256...A..A.(S.S.
0000030: 8c4b 9781 c6a5 1820 f41a bc95 4e62 13a9  .K..... ....Nb..
0000040: 8356 a517 be55 6ebc fbf4 de74 e216 8f17  .V...Un....t....
0000050: 6222 011c 5920 a3fc caed c880 4298 46d5  b"..Y ......B.F.
0000060: dd39 396e d72d 1e40                      .99n.-.@

That consists of:
4 bytes 00000013 bigendian int = 19: length of type
19 bytes 'ecdsa-sha2-nistp256' type
4 bytes 00000008 bigendian int = 8: length of curveid
8 bytes 'nistp256' curveid (redundant, but that's the wire format)
4 bytes 00000041 bigendian int = 65: length of pub point
65 bytes beginning 04: pub point in X9.62 format, copied more conveniently in SEC1, which is 1 byte 04=uncompressed, N bytes X-coordinate, N bytes Y-coordinate where N is the (fixed) size needed to represent the curve's underlying field as unsigned.

These are mostly defined in rfc5656 section 3.1 and the curveid's in 6.1. The RFC allows compressed point format, but OpenSSH doesn't use that option.

BCECPublicKey.getEncoded() (like all Java PublicKey classes) returns a so-called X.509 (really SubjectPublicKeyInfo, SPKI) encoding which for EC includes the public point in X9.62 uncompressed format, but you need some parsing to extract it. Since you have BC, it's easiest to use that:

byte[] point = SubjectPublicKeyInfo.getInstance(ASN1Sequence.getInstance(encoded)).getPublicKeyData().getOctets();

Alternatively .getW() and .getQ() get the point as a JCE or BC class, from either of which you can get the (affine) X and Y coordinates as BigInteger resp. ECFieldElement which in turn yields BigInteger, and each BigInteger can be converted to a variable-sized byte array which you must then left-pad or left-truncate to the correct size.

The result is the data to be hashed. In case you aren't aware, only OpenSSH versions 6.8 up use base64(sha256(pubkey)) for the fingerprint (by default). Before that it was hex-with-colons(md5(pubkey)), and newer versions can do the old fingerprint for compatibility (see option FingerprintHash in ssh_config for ssh and flag -E in ssh-keygen).

And to be clear this is only the OpenSSH fingerprint. Key fingerprints are also used in the PGP and X.509/PKIX (SSL/TLS, CMS/SMIME, etc) worlds and they are entirely different.