OpenSSL SSL_read Failure (error:00000005:lib(0):func(0):DH lib)

Question

I'm using OpenSSL version 1.1.0f to encrypt the connection between my client and a relaying server for which I don't have any access. Some times the client's connection is getting terminated abruptly with the readLine failure on client side.

Here is the error code:

readLine: readString returned -1 , Error Msg - error:00000005:lib(0):func(0):DH lib

I did search for the appropriate reasons for this kind of error code, everybody (Example this thread) says that there seems to be a problem with the Diffie-Hellman key exchange during the connection establishment. But The connection was going fine for few minutes(some times it may not even occur and it may occur very frequently once after 2 to 3 minutes of connection success) until it get terminated abruptly.

Here are some more errors from the logs that I have

SSL_read failed with error - 5, bytes received -1, error string error:00000005:lib(0):func(0):DH lib, wsaError 0

SSL Error - 5:error:FFFFFFFF:lib(255):func(4095):reason(4095)

Here is the socket read code from the SSL client

if ((isSecureMode() == true) && (lpSSL != NULL))
{
    bytesReceived = SSL_read(lpSSL, receiveBuf, bufferSize) ;

    if ((bytesReceived <= 0))
    {
        int sslErrorCode = lpSSL ? SSL_get_error(lpSSL, bytesReceived) : -1;

        char sslErrorString[MAX_ERROR_MSG_LEN] = {'\0'};

        ERR_error_string(sslErrorCode, sslErrorString);
        setLastError(sslErrorCode, std::string(sslErrorString));

        int wsaError = WSAGetLastError();

        if(isShutdownInitiated == false)
        {
            if (Logger)
            {
                Logger->log(LOG_WARNING, "receiveString - SSL_read failed with error - %d, bytes received %d, error string %s, wsaError %d", sslErrorCode, bytesReceived, std::string(sslErrorString), wsaError);
            }
                                                    // Always return -1 incase of failure
            bytesReceived = -1;
        } 
        else
        {
            if (Logger)
            {
                Logger->log(LOG_WARNING, "receiveString - Socket was Shutdown, SSL_read failed with error - %d, bytes received %d, error string %s , wsaError %d", sslErrorCode, bytesReceived, std::string(sslErrorString), wsaError);
            }
                                    
            setLastError(ERROR_ALREADY_SHUTDOWN);
            bytesReceived = -1;
        }
    }
}

I'm not an expert in OpenSSL and I doubted that the problem may be with the server but there are no error messages that can prove it except for the above ones.

I did analyze a lot and spent weeks without any improvement.

Update : Upon analyzing the error messages the error code SSL_ERROR_SYSCALL along with the wsa error_code 10054 says that the connection has been terminated by the remote party. The same error message is being written on server logs. That leads to the doubt for the connection being reset by the intermediate devices (may be routers that reside in between the client and the server) as per this thread. I'm trying to analyze further and sort out the problem.

Update 2: I also found out that the problem happens some times that is only on some networks. It works perfectly fine on few networks.The above mentioned thread also tells the same.

Update 3: I had figured out one of the reasons for connection termination, The error 10054 was actually due to firewall and the network that has been resolved now.

Debugging the readline failure with error

SSL_read failed with error - 5, bytes received -1, error string error:00000005:lib(0):func(0):DH lib, wsaError 0

Any help would be appreciated.

Answer 1

The return values of SSL_get_error should not be passed to ERR_error_string. The value 5 in sslErrorCode is probably SSL_ERROR_SYSCALL.

You should loop over the local threads error queue with ERR_get_error to find the reason for the failure, and you also should empty the error queue before calling SSL_read.

(All this is also covered in https://stackoverflow.com/a/37980911/1478356)

There have also been many reports of SSL_read returning -1 (with SSL_ERROR_SYSCALL from SSL_get_error) when errno = 0 (WSAGetLastError on windows) and the error queue being empty, when the underlying connection received an EOF. See https://github.com/openssl/openssl/issues/1903 for a long discussion; especially this comment explains how to manually check for EOF.

There have been claims this got fixed in 1.1.0d, but there were other reports with this error in 1.1.0f (e.g. https://redmine.lighttpd.net/issues/2784).