Directly Read/Write Handshake data with Memory BIO

Question

I need to create an OpenSSL connection where I can directly read/write handshake data. The reason is the handshake data will be transported in a UDP connection (DTLS is not an option, because the data is not directly in the datagram, but inside another protocol packets, EAP if you're curious). So far, I've created an OpenSSL connection but I've not even been able to read the client's handshake to send to the server.

In my research I've found I need a Memory BIO to read/write to the connection, but cannot figure out how to extract the handshake data. Here's how I initialize the client connection:

SSL_library_init();
SSL_load_error_strings();
OpenSSL_add_all_algorithms();

ctx = SSL_CTX_new(SSLv3_client_method());
SSL_CTX_set_verify(ctx, SSL_VERIFY_NONE, NULL);

ssl = SSL_new(ctx);
rbio = BIO_new(BIO_s_mem());
wbio = BIO_new(BIO_s_mem());

SSL_set_bio(ssl, rbio, wbio);
SSL_set_connect_state(ssl);

I've tried doint SSL_connect, to initiate the handshake:

int ret = SSL_connect(ssl);

But returns -1, and doing SSL_get_error(ssl, res) I get an error code 2, then I execute ERR_error_string with that code and get:

error:00000002:lib(0):func(0):system lib

Also, if I use SSL_do_handshake instead of SSL_connect I get exactly the same error.

I've been able to set a OpenSSL connection over TCP, but have never done this with Memory BIOs, so any help with this would be very appreciated. Thanks!

Answer 1

Finally I get it to work, I was in the right way:

The function SSL_set_connect_state(ssl) is needed to tell the connection to be prepared for the handshake init. Then, we call SSL_do_handshake(ssl) to start the handshake. This function will return -1 because the handshake was not finished, but we can actually read from the client ssl connection BIO writer and send the data using the protocol we want (in my case, EAP RADIUS packets over UDP).

Client

ctx = SSL_CTX_new(SSLv3_client_method());
SSL_CTX_set_verify(ctx, SSL_VERIFY_NONE, NULL);

ssl = SSL_new(ctx);
rbio = BIO_new(BIO_s_mem());
wbio = BIO_new(BIO_s_mem());

SSL_set_bio(ssl, rbio, wbio);
SSL_set_connect_state(ssl); 

SSL_do_handshake(ssl); // This will return -1 (error) as the handshake is not finished, we can ignore it.

char buf[4096];
BIO_read(wbio, buf, 4096); // Read from BIO, put data in buffer

// Then use data in buffer to send to the server

The server, in the other hand, should be configured using the credential and private key. Also, instead of SSL_set_connect_state() we should use SSL_set_accept_state() as the server will wait for the client's handshake hello. Then, we simply write the client handshake hello data to the server BIO reader:

Server

ctx = SSL_CTX_new(SSLv3_server_method()); // This is the server!
SSL_CTX_set_verify(ctx, SSL_VERIFY_NONE, NULL);

ssl = SSL_new(ctx);
rbio = BIO_new(BIO_s_mem());
wbio = BIO_new(BIO_s_mem());

SSL_set_bio(ssl, rbio, wbio);
SSL_set_accept_state(ssl); // The server uses SSL_set_accept_state

// Here we get the data from the client suppose it's in the variable buf
// and write it to the connection reader BIO.
BIO_write(rbio, buf, strlen(buf));

if (!SSL_is_init_finished(ssl)) {
    SSL_do_handshake(ssl);
}

We can use the the SSL_is_init_finished(ssl) function to check if the handshake was done, and while it is not done we call SSL_do_handshake(ssl), and then read again from the BIO_writer to send data to the client.

This process between client and server should be done until the connection is done (i.e. SSL_is_init_finished(ssl) returns true).

Then, after the handshake is done, you can send secure data between client/server, by using the SSL_read and SSL_write functions. Hope this short explanation is useful for someone!