SSL_CTX_set_tlsext_servername_callback callback function not being called

Question

I am writing a https server and i need to get host name from client before ssl_accept() using SNI. I am using SSL_CTX_set_tlsext_servername_callback() to receive host name but the callback not at all get called. Here is part of my code

// Server Name Indication callback from OpenSSL
static int serverNameCallback(SSL *ssl, int *ad, void *arg) 
{
    if (ssl == NULL)
        return SSL_TLSEXT_ERR_NOACK;

    const char* servername = SSL_get_servername(ssl, TLSEXT_NAMETYPE_host_name);
    printf("ServerName: %s\n", servername);
}

int main()
{
    //some code 
    const SSL_METHOD *method;
    SSL_CTX *ctx;

    method = SSLv3_server_method();  
    ctx = SSL_CTX_new(method);   
    if ( ctx == NULL )
    { 
        printf("SSL_ctx_new() error\n");
    }

    int ret = SSL_CTX_set_tlsext_servername_callback(ctx, serverNameCallback);
}

Answer 1

I am writing a https server and i need to get host name from client before ssl_accept() using SNI.

You don't call SSL_CTX_set_tlsext_servername_callback. The OpenSSL library will invoke it for you during the handshake.

Your SNI callback is not called for SSLv2 and SSLv3 clients. That's because SNI is a TLS extension.

If the client is Windows XP (or similar that don't use the extension with TLS), then your SNI callback will be invoked, but servername will be NULL. In this case, you should use the default server context.

If the client is using TLS and send s the server name, then your SNI callback will be invoked. In the callback, you should (1) determine if the default certificate and context is OK. If it is OK, then return SSL_TLSEXT_ERR_OK. (2) If you can provide a more appropriate certificate, then use SSL_set_SSL_CTX to swap in the new context and return SSL_TLSEXT_ERR_OK.

If you experience an error (like a NULL servername), then you should return SSL_TLSEXT_ERR_NOACK. There's two other error codes you can return. Both are fatal to the connection, IIRC.

Implementation details of a callback are given at Serving multiple domains in one box with SNI.