Java - Get Key failed: java.security.InvalidKeyException: Invalid RSA private key and DerInputStream.getLength(): Redundant length bytes found

Question

I am having some trouble with XML digital signature with a PFX file. I get an exception when I run this code:

KeyStore ks = KeyStore.getInstance("PKCS12");
        
fs = new FileInputStream(file);

ks.load(fs, "password".toCharArray());

// this line!
KeyStore.PrivateKeyEntry keyEntry = (KeyStore.PrivateKeyEntry) ks.getEntry("alias", new KeyStore.PasswordProtection("password".toCharArray()));

This is the exception:

java.security.UnrecoverableKeyException: Get Key failed: 
java.security.InvalidKeyException: Invalid RSA private key
  at sun.security.pkcs12.PKCS12KeyStore.engineGetKey(PKCS12KeyStore.java:435)
  at sun.security.pkcs12.PKCS12KeyStore.engineGetEntry(PKCS12KeyStore.java:1306)
  at java.security.KeyStore.getEntry(KeyStore.java:1521)
  at app.ubl.xml.GenerateSignature.makeSignatureXML(GenerateSignature.java:88)

Caused by: java.io.IOException: DerInputStream.getLength(): Redundant length bytes found
  at sun.security.util.DerInputStream.getLength(DerInputStream.java:606)
  at sun.security.util.DerInputStream.getLength(DerInputStream.java:569)
  at sun.security.util.DerInputStream.getPositiveBigInteger(DerInputStream.java:220)
  at sun.security.rsa.RSAPrivateCrtKeyImpl.parseKeyBits(RSAPrivateCrtKeyImpl.java:205)

The real problem is that the code works in java 1.8.0_111, but any higher version the error shows.

Exception when running keytool

Another issue that I found is that when I run this command using keytool:

keytool -list -keystore file.pfx -storepass password -storetype PKCS12 -v

to show the details of the PFX file then again it only works on java 1.8.0_111. Otherwise I get this exception:

java.util.IllegalFormatConversionException: d != java.lang.String
    at java.util.Formatter$FormatSpecifier.failConversion(Formatter.java:4302)
    at java.util.Formatter$FormatSpecifier.printInteger(Formatter.java:2793)
    at java.util.Formatter$FormatSpecifier.print(Formatter.java:2747)
    at java.util.Formatter.format(Formatter.java:2520)
    at java.util.Formatter.format(Formatter.java:2455)
    at java.lang.String.format(String.java:2940)
    at sun.security.tools.keytool.Main.withWeak(Main.java:3076)
    at sun.security.tools.keytool.Main.printX509Cert(Main.java:3125)
    at sun.security.tools.keytool.Main.doPrintEntry(Main.java:1924)
    at sun.security.tools.keytool.Main.doPrintEntries(Main.java:2236)
    at sun.security.tools.keytool.Main.doCommands(Main.java:1123)
    at sun.security.tools.keytool.Main.run(Main.java:366)
    at sun.security.tools.keytool.Main.main(Main.java:359)

I don't know if this is relevant, but that is all I got.

Any thoughts?

PD: Sorry for my bad english.

Answer 1

Stricter DER checking in 8u121

This is a change that was introduced in the Java versions after Java 8u111. The next version was Java 8u121.

As a result some encodings in the underlying ASN.1 format are no longer being accepted.

And this change is mentioned in the 8u121 release notes as such:

security-libs
More checks added to DER encoding parsing code
More checks are added to the DER encoding parsing code to catch various encoding errors. In addition, signatures which contain constructed indefinite length encoding will now lead to IOException during parsing. Note that signatures generated using JDK default providers are not affected by this change.
JDK-8168714 (not public)

Reason: CVE

This specific change was because of a vulnerability: CVE-2016-5546.

Relevant source code change

The error messages you saw (Redundant length bytes found) was introduced as part this changeset. The summary was this:

8168714: Tighten ECDSA validation
Summary: Added additional checks to DER parsing code

Unfortunately as of now (2018) the bug mentioned in the changeset "Bug 8168714" is STILL marked private. (You won't be able to view it without a login.)

But here's another bug that is public: OpenJDK: JDK-8175251: Failed to load RSA private key from pkcs12. (A less complete version of this bug is also on java.com.)

And this bug links to the source code changeset mentioned above.

Workaround: Run through OpenSSL washing machine to clean up P12 file

The encoding in some P12 files seems to be broken somehow. A workaround is to unwrap the P12's contents to PEM with OpenSSL and then rewrap them into a new P12 file. (The real solution would be to fix/update the software that generates the P12 file.)

The following command is mentioned in the Java Bug ticket:

Openssl is able to remove the redundant 0s when extracting the private key. We can use the following 2 commands to normalize an affected pkcs12 file:

1. openssl pkcs12 -in pkcs12-file -out key-and-cert -nodes -passin pass:abcXYZ

2. openssl pkcs12 -in key-and-cert -export -out new-pkcs12-file -passout pass:abcXYZ

Further reading: Bouncy Castle Mailing List

Update 2018-02-12Mon: I just realized that the issue discussed on the Bouncy Castle mailing list deals with a DIFFERENT type of encoding error. Namely: in ASN.1 you have T-L-V (Tag-Length-Value) encoding.

• And OP's question had to do with a non-standard/faulty encoding of the LENGTH attribute. (Error message: Redundant length bytes found.)
• While the issue discussed on the BC-ML had to do with a non-standard/faulty encoding of the VALUE attribute. (Error message: Invalid encoding: redundant leading 0s.)

But both of these issues are related to unnecessarily (and therefore illegally) padding with 0x00.

This change was discussed on the mailing list for the Java "Bouncy Castle" cryptography library:

• [email protected] mailing list, 2017-06-27, Strict Public Key checking leads to broken certificates (Archived here. Official archive here.)
  • And as a result a special configuration parameter (org.bouncycastle.asn1.allow_unsafe_integer) was introduced in BC version 1.58. (See release notes section 2.2.3.)

Related questions

• 2017-06-27, SO: Unable to open keystore in AndroidStudio - "Redundant length bytes found"
• 2017-03-17, SO: Signing android app throws IOException: Redundant length bytes found