SSL_read fails with SSL_ERROR_SSL

Question

I am writing a https server. I have created csr and signed it using my root certificate for a test domain. When a client connects, SSL_accept() is done successfully. I am using Non- Blocking IO. So I will receive the data first in a char buffer asychronously using WSARecv() and IOCP in windows. From that char buffer, i am writing it to a BIO (BIO_write returns number of bytes written) and on trying to decrypt contents of that BIO using SSL_read() it returns ssl_error_ssl and error string as error:00000001:lib(0):func(0):reason(1).

I have added structure of my code here.

const SSL_METHOD *method;
SSL_CTX *ctx;

method = SSLv23_method();    /* create new server-method instance */
ctx = SSL_CTX_new(method);   /* create new context from method */
if ( ctx == NULL )
{
    printf("SSL Context Creation failed\n");
}

//create bio
BIO *bioIn = BIO_new(BIO_s_mem());
BIO *bioOut = BIO_new(BIO_s_mem());

/* get new SSL state with context */
SSL *clientSSL = SSL_new(ctx);      
SSL_set_bio(clientSSL, bioIn , bioOut);

/* set connection socket to SSL state */        
SSL_set_fd(clientSSL, mClientSocket);      

/* serverNameCallBack will set ctx with certificate created for this domain */
SSL_CTX_set_tlsext_servername_callback(ctx, serverNameCallback);

/* accept ssl connection */
SSL_accept(clientSSL);

//Using WSARecv() here to get encrypted request to a buffer

//read from buffer
//bridge->getBuffer() returns the buffer with encrypted  data received
int retBio = BIO_write(bioIn, bridge->getBuffer(), bytesTransfered);

char *buffer = (char *)malloc(sizeof(char) * 1024);
ZeroMemory(buffer, sizeof(buffer));

int retSSL = SSL_read(clientSSL, (void*)buffer, 1023);

retSSL == -1 and SSL_get_error(clientSSL, retSSL) returns SSL_ERROR_SSL

Answer 1

I fixed this issue thanks to some blogs. The correct sequence to do this is, BIO should be created and associated with ssl object after calling ssl_accept. If you are associating it before ssl_accept then you have to handle it differently. And you should set SSL_set_accept_state before calling ssl_accept.

Here is the correct sequence of code

const SSL_METHOD *method;
SSL_CTX *ctx;

method = SSLv23_method();    /* create new server-method instance */
ctx = SSL_CTX_new(method);   /* create new context from method */
if ( ctx == NULL )
{
    printf("SSL Context Creation failed\n");
}

/* get new SSL state with context */
SSL *clientSSL = SSL_new(ctx);      

/* set connection socket to SSL state */        
SSL_set_fd(clientSSL, mClientSocket);      

/* serverNameCallBack will set ctx with certificate created for this domain */
SSL_CTX_set_tlsext_servername_callback(ctx, serverNameCallback);

/* set ssl handle to be used as a server */
SSL_set_accept_state(clientSSL);

/* accept ssl connection */
SSL_accept(clientSSL);

//Using WSARecv() here to get encrypted request to a buffer

//create bio
BIO *bioIn = BIO_new(BIO_s_mem());
BIO *bioOut = BIO_new(BIO_s_mem());
SSL_set_bio(clientSSL, bioIn , bioOut);

//read from buffer
//bridge->getBuffer() returns the buffer with encrypted  data received
int retBio = BIO_write(bioIn, bridge->getBuffer(), bytesTransfered);

char *buffer = (char *)malloc(sizeof(char) * 1024);
ZeroMemory(buffer, sizeof(buffer));

int retSSL = SSL_read(clientSSL, (void*)buffer, 1023);