Multithreaded communication with OpenSSL

Question

I'm using OpenSSL to communicate with a server. I can send data to the server at any time and the server might or might not send back a response. The server can also send data to the client without a request.

I'm using SSL over a BIO made with BIO_new_ssl_connect and then using SSL_read and SSL_write.

My first approach was to use blocking sockets. I would start a thread and call SSL_read on it in a loop. Each call blocks and only returns when some data was read. After each call I can then package up the data and dispatch it somewhere. When I have to write, I just call SSL_write from another thread.

I cannot figure out if it is valid to call SSL_write while doing a SSL_read on the same connection from different threads. The SSL_read call crashes when I try to disconnect (SSL_free/BIO_free) the connection.

Are these calls from different threads advisable? If not, is there a better approach to this problem (which seems like a very common thing)?

Would non-blocking sockets work better maybe?

EDIT: Sorry, I should have added that I already implemented the thread safe locking as described in the OpenSSL documentation.

Answer 1

The OpenSSL library can be made thread-safe, but you must provide the locking primitives yourself. From the OpenSSL FAQ:

Multi-threaded applications must provide two callback functions to OpenSSL by calling CRYPTO_set_locking_callback() and CRYPTO_set_id_callback(), for all versions of OpenSSL up to and including 0.9.8[abc...]. As of version 1.0.0, CRYPTO_set_id_callback() and associated APIs are deprecated by CRYPTO_THREADID_set_callback() and friends. This is described in the threads(3) manpage.

In regards to calling SSL_free() while another thread is blocked in SSL_read(), don't do that. It doesn't matter if the library is thread safe, that is an API violation. Simultaneous SSL_read() and SSL_write() from separate threads is fine. If another thread is still using the SSL_CTX *, the threads will need to cooperate with each other to figure out when it is safe to call SSL_free(), because it is an error to allow another thread to read from or write to freed memory. OpenSSL is a library, after all, and SSL_CTX * is just a plain structure allocated from the heap.

You usually track when it is safe to really free an object by using reference counts. The reference counts can be hidden within a custom BIO if you don't want to manage them in the application code itself.

You mention the possibility of using non-blocking mode. That in of itself is insufficient to solve your memory management bug. You still need reference counts to decide if it is safe to call SSL_free().

As an aside, if you decide to use non-blocking mode, you probably should combine non-blocking mode with multi-threaded in order to maximize CPU utilization on a multi-core system. But, non-blocking OpenSSL is actually an order of magnitude more complex that a regular non-blocking BSD socket. This is because in addition to the regular "would block" on read or write, an OpenSSL read may report that it needs a write operation to complete, and an OpenSSL write may report that it needs a read operation to complete. Thus, your non-blocking code needs to remember what operation it was trying after handling the completion notification from your demultiplexer (e.g., select or poll). Moreover, OpenSSL dictates that you must pass in the exact same arguments that had been attempted when the "would block" notification was returned. So, as an example, any new data that you may actually want to send has to be buffered until the current OpenSSL write completes.