Writing bytes directly to OpenSSL Connection, without TCP

Question

I'm implementing a RADIUS server with support for EAP(PEAP) authentication. The RADIUS relies on UDP and each packet will contain an EAP message with the authentication data. To simplify, each EAP (PEAP) message contains TLS packets so the client first will send an EAP packet containing TLS Client Handshake, which the server will process, and then return the TLS Server Handshake, and so on with the following packages.

I've been able to implement a simple TLS server over TCP, but what I need is write TLS bytes directly in the SSL connection.

SSLConn_init_thread();
SSL_load_error_strings();
ERR_load_crypto_strings();
SSL_library_init();

SSL_CTX *ctx;
...
ctx = SSL_CTX_new(SSLv23_server_method());
SSL_CTX_use_PrivateKey(...);
SSL_CTX_use_certificate(...);

ssl = SSL_new(ctx); // I want to read/write TLS packets directly with this object

I've looked into OpenSSL docs but it's not very well documented. In this guide says one can make the handshake "by hand" using the SSL_set_accept_state() with SSL_read and SSL_write, but I don't understand exactly how to do that.

Any advice with this or about handling OpenSSL connections without relying on TCP would be very appreciated.

Answer 1

I ended up reading the book Network Security with OpenSSL and found Memory BIOs were what i needed.

Network Security With OpenSSL

A memory BIO treats a memory segment the same as a file or socket, and can be created by using BIO_s_mem to obtain a BIO_METHOD object suitable for use with BIO_new and BIO_set

BIO_s_mem()

BIO_s_mem() return the memory BIO method function.

A memory BIO is a source/sink BIO which uses memory for its I/O. Data written to a memory BIO is stored in a BUF_MEM structure which is extended as appropriate to accommodate the stored data.

Any data written to a memory BIO can be recalled by reading from it. Unless the memory BIO is read only any data read from it is deleted from the BIO.

Basically you can read/write encrypted data directly to the SSL connection using the memory BIOs:

// setup SSL_context...
ssl = SSL_new(ctx);

// Create read/write BIOs 
rbio = BIO_new(BIO_s_mem());
wbio = BIO_new(BIO_s_mem());

SSL_set_bio(ssl, rbio, wbio);

if (/* Server */) {
  SSL_set_accept_state(ssl); 
} else {
  SSL_set_connect_state(ssl);
}

To write/read from BIOs:

BIO_read(wbio, buffer, strlen(buffer));
BIO_write(wbio, buffer, strlen(buffer));

Answer 2

In ascii diagram terms, here's what the flow of bytes loops like. SSL_read/SSL_write are used to pass unencrypted bytes in-to/out-of the SSL object, and BIO_read/BIO_write are used to get corresponding encrypted bytes in-to/out-of the SSL object. It's the up to you to then transfer the encrypted bytes over a network connection.

Git hub example of non blocking sockets / SSL

(https://gist.github.com/darrenjs/4645f115d10aa4b5cebf57483ec82eca)

  +------+                                    +-----+
  |......|--> read(fd) --> BIO_write(rbio) -->|.....|--> SSL_read(ssl)  --> IN
  |......|                                    |.....|
  |.sock.|                                    |.SSL.|
  |......|                                    |.....|
  |......|<-- write(fd) <-- BIO_read(wbio) <--|.....|<-- SSL_write(ssl) <-- OUT
  +------+                                    +-----+
          |                                  |       |                     |
          |<-------------------------------->|       |<------------------->|
          |         encrypted bytes          |       |  unencrypted bytes  |