Why does SSL handshake give 'Could not generate DH keypair' exception?

Question

The problem is the prime size. The maximum-acceptable size that Java accepts is 1024 bits. This is a known issue (see JDK-6521495).

The bug report that I linked to mentions a workaround using BouncyCastle's JCE implementation. Hopefully that should work for you.

UPDATE

This was reported as bug JDK-7044060 and fixed recently.

Note, however, that the limit was only raised to 2048 bit. For sizes > 2048 bit, there is JDK-8072452 - Remove the maximum prime size of DH Keys; the fix appears to be for 9.

The "Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files" answer did not work for me but The BouncyCastle's JCE provider suggestion did.

Here are the steps I took using Java 1.6.0_65-b14-462 on Mac OSC 10.7.5

1) Download these jars:

• bcprov-jdk15on-154.jar

• bcprov-ext-jdk15on-154.jar

2) move these jars to $JAVA_HOME/lib/ext

3) edit $JAVA_HOME/lib/security/java.security as follows: security.provider.1=org.bouncycastle.jce.provider.BouncyCastleProvider

restart app using JRE and give it a try

Here is my solution (java 1.6), also would be interested why I had to do this:

I noticed from the javax.security.debug=ssl, that sometimes the used cipher suite is TLS_DHE_... and sometime it is TLS_ECDHE_.... The later would happen if I added BouncyCastle. If TLS_ECDHE_ was selected, MOST OF the time it worked, but not ALWAYS, so adding even BouncyCastle provider was unreliable (failed with same error, every other time or so). I guess somewhere in the Sun SSL implementation sometimes it choose DHE, sometimes it choose ECDHE.

So the solution posted here relies on removing TLS_DHE_ ciphers completely. NOTE: BouncyCastle is NOT required for the solution.

So create the server certification file by:

echo |openssl s_client -connect example.org:443 2>&1 |sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p'

Save this as it will be referenced later, than here is the solution for an SSL http get, excluding the TLS_DHE_ cipher suites.

package org.example.security;

import java.io.BufferedInputStream;
import java.io.BufferedReader;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.InetAddress;
import java.net.Socket;
import java.net.URL;
import java.net.UnknownHostException;
import java.security.KeyStore;
import java.security.cert.Certificate;
import java.security.cert.CertificateFactory;
import java.security.cert.X509Certificate;
import java.util.ArrayList;
import java.util.List;

import javax.net.ssl.HttpsURLConnection;
import javax.net.ssl.SSLContext;
import javax.net.ssl.SSLParameters;
import javax.net.ssl.SSLSocket;
import javax.net.ssl.SSLSocketFactory;
import javax.net.ssl.TrustManagerFactory;

import org.apache.log4j.Logger;

public class SSLExcludeCipherConnectionHelper {

    private Logger logger = Logger.getLogger(SSLExcludeCipherConnectionHelper.class);

    private String[] exludedCipherSuites = {"_DHE_","_DH_"};

    private String trustCert = null;

    private TrustManagerFactory tmf;

    public void setExludedCipherSuites(String[] exludedCipherSuites) {
        this.exludedCipherSuites = exludedCipherSuites;
    }

    public SSLExcludeCipherConnectionHelper(String trustCert) {
        super();
        this.trustCert = trustCert;
        //Security.addProvider(new BouncyCastleProvider());
        try {
            this.initTrustManager();
        } catch (Exception ex) {
            ex.printStackTrace();
        }
    }

    private void initTrustManager() throws Exception {
        CertificateFactory cf = CertificateFactory.getInstance("X.509");
        InputStream caInput = new BufferedInputStream(new FileInputStream(trustCert));
        Certificate ca = null;
        try {
            ca = cf.generateCertificate(caInput);
            logger.debug("ca=" + ((X509Certificate) ca).getSubjectDN());
        } finally {
            caInput.close();
        }

        // Create a KeyStore containing our trusted CAs
        KeyStore keyStore = KeyStore.getInstance("jks");
        keyStore.load(null, null);
        keyStore.setCertificateEntry("ca", ca);

        // Create a TrustManager that trusts the CAs in our KeyStore
        String tmfAlgorithm = TrustManagerFactory.getDefaultAlgorithm();
        tmf = TrustManagerFactory.getInstance(tmfAlgorithm);
        tmf.init(keyStore);
    }

    public String get(URL url) throws Exception {
        // Create an SSLContext that uses our TrustManager
        SSLContext context = SSLContext.getInstance("TLS");
        context.init(null, tmf.getTrustManagers(), null);
        SSLParameters params = context.getSupportedSSLParameters();
        List<String> enabledCiphers = new ArrayList<String>();
        for (String cipher : params.getCipherSuites()) {
            boolean exclude = false;
            if (exludedCipherSuites != null) {
                for (int i=0; i<exludedCipherSuites.length && !exclude; i++) {
                    exclude = cipher.indexOf(exludedCipherSuites[i]) >= 0;
                }
            }
            if (!exclude) {
                enabledCiphers.add(cipher);
            }
        }
        String[] cArray = new String[enabledCiphers.size()];
        enabledCiphers.toArray(cArray);

        // Tell the URLConnection to use a SocketFactory from our SSLContext
        HttpsURLConnection urlConnection =
            (HttpsURLConnection)url.openConnection();
        SSLSocketFactory sf = context.getSocketFactory();
        sf = new DOSSLSocketFactory(sf, cArray);
        urlConnection.setSSLSocketFactory(sf);
        BufferedReader in = new BufferedReader(new InputStreamReader(urlConnection.getInputStream()));
        String inputLine;
        StringBuffer buffer = new StringBuffer();
        while ((inputLine = in.readLine()) != null) 
            buffer.append(inputLine);
        in.close();

        return buffer.toString();
    }

    private class DOSSLSocketFactory extends javax.net.ssl.SSLSocketFactory {

        private SSLSocketFactory sf = null;
        private String[] enabledCiphers = null;

        private DOSSLSocketFactory(SSLSocketFactory sf, String[] enabledCiphers) {
            super();
            this.sf = sf;
            this.enabledCiphers = enabledCiphers;
        }

        private Socket getSocketWithEnabledCiphers(Socket socket) {
            if (enabledCiphers != null && socket != null && socket instanceof SSLSocket)
                ((SSLSocket)socket).setEnabledCipherSuites(enabledCiphers);

            return socket;
        }

        @Override
        public Socket createSocket(Socket s, String host, int port,
                boolean autoClose) throws IOException {
            return getSocketWithEnabledCiphers(sf.createSocket(s, host, port, autoClose));
        }

        @Override
        public String[] getDefaultCipherSuites() {
            return sf.getDefaultCipherSuites();
        }

        @Override
        public String[] getSupportedCipherSuites() {
            if (enabledCiphers == null)
                return sf.getSupportedCipherSuites();
            else
                return enabledCiphers;
        }

        @Override
        public Socket createSocket(String host, int port) throws IOException,
                UnknownHostException {
            return getSocketWithEnabledCiphers(sf.createSocket(host, port));
        }

        @Override
        public Socket createSocket(InetAddress address, int port)
                throws IOException {
            return getSocketWithEnabledCiphers(sf.createSocket(address, port));
        }

        @Override
        public Socket createSocket(String host, int port, InetAddress localAddress,
                int localPort) throws IOException, UnknownHostException {
            return getSocketWithEnabledCiphers(sf.createSocket(host, port, localAddress, localPort));
        }

        @Override
        public Socket createSocket(InetAddress address, int port,
                InetAddress localaddress, int localport) throws IOException {
            return getSocketWithEnabledCiphers(sf.createSocket(address, port, localaddress, localport));
        }

    }
}

Finally here is how it is used (certFilePath if the path of the certificate saved from openssl):

try {
            URL url = new URL("https://www.example.org?q=somedata");            
            SSLExcludeCipherConnectionHelper sslExclHelper = new SSLExcludeCipherConnectionHelper(certFilePath);
            logger.debug(
                    sslExclHelper.get(url)
            );
        } catch (Exception ex) {
            ex.printStackTrace();
        }

The answer above is correct, but in terms of the workaround, I had problems with the BouncyCastle implementation when I set it as preferred provider:

java.lang.ArrayIndexOutOfBoundsException: 64
    at com.sun.crypto.provider.TlsPrfGenerator.expand(DashoA13*..)

This is also discussed in one forum thread I found, which doesn't mention a solution. http://www.javakb.com/Uwe/Forum.aspx/java-programmer/47512/TLS-problems

I found an alternative solution which works for my case, although I'm not at all happy with it. The solution is to set it so that the Diffie-Hellman algorithm is not available at all. Then, supposing the server supports an alternative algorithm, it will be selecting during normal negotiation. Obviously the downside of this is that if somebody somehow manages to find a server that only supports Diffie-Hellman at 1024 bits or less then this actually means it will not work where it used to work before.

Here is code which works given an SSLSocket (before you connect it):

List<String> limited = new LinkedList<String>();
for(String suite : ((SSLSocket)s).getEnabledCipherSuites())
{
    if(!suite.contains("_DHE_"))
    {
        limited.add(suite);
    }
}
((SSLSocket)s).setEnabledCipherSuites(limited.toArray(
    new String[limited.size()]));

Nasty.

You can disable DHE completely in your jdk, edit jre/lib/security/java.security and make sure DHE is disabled, eg. like

jdk.tls.disabledAlgorithms=SSLv3, DHE.

You can installing the provider dynamically:

1) Download these jars:

• bcprov-jdk15on-152.jar
• bcprov-ext-jdk15on-152.jar

2) Copy jars to WEB-INF/lib (or your classpath)

3) Add provider dynamically:

import org.bouncycastle.jce.provider.BouncyCastleProvider;

...

Security.addProvider(new BouncyCastleProvider());