SSL handshake exception: "Algorithm constraints check failed: MD5withRSA"

Question

I tried to install Oracle Entitlements Server Client. When I call

config.cmd -smConfigId Sample-SM -prpFileName C:\oracle\product\11.1.2\as_1\oessm\SMConfigTool\smconfig.java.controlled.prp 

I got this Exception:

    javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Algorithm constraints check failed: MD5withRSA
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1884)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:276)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:270)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1341)
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:153)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
        at sun.security.ssl.Handshaker.process_record(Handshaker.java:804)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1016)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
        at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:702)
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122)
        at java.io.OutputStream.write(OutputStream.java:75)
        at oracle.security.oes.enroll.EnrollmentClient.writeToSocket(EnrollmentClient.java:330)
        at oracle.security.oes.enroll.EnrollmentClient.enroll(EnrollmentClient.java:161)
        at oracle.security.oes.enroll.EnrollmentClient.main(EnrollmentClient.java:478)
        at oracle.security.oes.tools.EnrollmentTool.doEnroll(EnrollmentTool.java:103)
        at oracle.security.oes.tools.SMConfigTool.doEnrollment(SMConfigTool.java:1192)
        at oracle.security.oes.tools.SMConfigTool.run(SMConfigTool.java:617)
        at oracle.security.oes.tools.SMConfigTool.main(SMConfigTool.java:546)
    Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Algorithm constraints check failed: MD5withRSA
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:350)
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:260)
        at sun.security.validator.Validator.validate(Validator.java:260)
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326)
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231)
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:126)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1323)
        ... 15 more
    Caused by: java.security.cert.CertPathValidatorException: Algorithm constraints check failed: MD5withRSA
        at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:159)
        at sun.security.provider.certpath.PKIXCertPathValidator.doValidate(PKIXCertPathValidator.java:351)
        at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:191)
        at java.security.cert.CertPathValidator.validate(CertPathValidator.java:279)
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:345)
        ... 21 more
sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Algorithm constraints check failed: MD5withRSA

Can you help me to find a reason?

Answer 1

The problem is caused by Oracle disabling hash algorithms which are no longer considered to be secure. Take a look at

JRE_HOME/lib/security/java.security

It contains the following properties:

jdk.certpath.disabledAlgorithms
jdk.tls.disabledAlgorithms

You can adjust them appropriately. For example, remove MD5 from the former and MD5withRSA from the latter.

Hint for docker images:

there is additional config file /etc/crypto-policies/back-ends/java.config in some docker images like keycloak in my case which overrides values in java.security

Answer 2

keyser gave direction for answer in comment.

Problem was in key's length. In short: "Starting from 7u40, the use of x.509 certificates with RSA keys less than 1024 bits in length is restricted."

So the right way to solve this problem it is using certificates with key's length at least 2048 bits.