WebRTC DTLS-SRTP OpenSSL Server Handshake Failure

Question

Here is my procedure in OpenSSL Server Mode,

Initialization Part of SSL and BIO variables:

map<int, SSL> m_SSLMap;
map<int, BIO> m_BioWriteMap;
map<int, BIO> m_BioReadMap;
int InitializeServerNegotiationMode(int iFd)
{
        SSL *pServSslFd;
        BIO *pWb, *pRb;

        pServSslFd = SSL_new(m_pCtx);
        assert(pServSslFd);

        if ( SSL_version(pServSslFd) == DTLS1_VERSION)
        {
            pWb = BIO_new(BIO_s_mem());
            pRb = BIO_new(BIO_s_mem());
            assert(pWb);
            assert(pRb);
            SSL_set_bio(pServSslFd, pRb, pWb);
            SSL_set_accept_state(pServSslFd);
        }
        m_SSLMap[iFd] = *pServSslFd;
        m_BioReadMap[iFd] = *pRb;
        m_BioWriteMap[iFd] = *pWb;

        return INITIALIZATION_SUCCESS;
 }

Server Mode Negotiation Operations when DTLS data comes to the server:

int ServerModeDTLSNegotiation(int iChannel, const char *pBuff, const int iLen, int iFd)
{

    SSL *pServSslFd;
    BIO *pRbio;
    BIO *pWbio;
    pServSslFd = &m_SSLMap[iFd];
    pRbio = &m_BioReadMap[iFd];
    pWbio = &m_BioWriteMap[iFd];


    char buff[4096];
    memset(buff, 0, strlen(buff));

    BIO_write(pRbio, pBuff, iLen);

    if(!SSL_is_init_finished(pServSslFd))
    {
        int iRet = SSL_do_handshake(pServSslFd);
    }

    int iNewLen = BIO_read(pWbio, buff, 2048);
    if(iNewLen>0)
    {
        char *pNewData = new char[iNewLen+1];
        for(int i=0;i<iNewLen;i++)
        pNewData[i] = buff[i];
         m_pEventHandler->SendReply(iChannel, (unsigned char *)pNewData, iNewLen);
    }
    else
    {
         printf("[DTLS]:: HandShaking Response failed for this data, 
         return -1;
    }
    return NEGOTIATION_SUCCESS;

}

Here I am attaching Wireshark TCP-Dump for better monitoring about the issue.

https://www.dropbox.com/s/quidcs6gilnvt2o/WebRTC%20DTLS%20Handshake%20Failure.pcapng?dl=0

Now, I am confident about my initialization of SSL_CTX variable. Because, Sometimes Handshake successfully negotiate for every port. But sometimes Handshake fails for one or two port. I am working for 5 days to solve WebRTC DTLS Server Mode Negotiation for Google Chrome. But I haven't found the root cause for this problem.

Answer 1

The link for TCP-Dump is not working. Anyway, it seems your solution should work.

As it's a server program, it's definitely multi threaded. But it's really dangerous to initialize SSL variables or to perform handshake procedure without locking. In that case so many things can happen if these two methods are processed by multiple thread.

My suggestion is to add locking mechanism for these methods.