Confused about OpenSSL non-blocking I/O

Question

In general, the OpenSSL library (C API) seems to offer two ways to do everything: you can either use plain system sockets configured to your liking, or you can use OpenSSL BIO objects which are sort of like streams.

However, I'm often confused by some of the duplicated functionality. For example, how do you make an SSL connection non-blocking? One way seems to be to simply access the underlying file descriptor and set it to non-blocking using fcntl. But there is also an OpenSSL API function called BIO_set_nbio which takes in a BIO* object and sets it to non-blocking mode.

So what is the best way to set up a non-blocking SSL socket? What happens if you pass OpenSSL a native file descriptor which is already set to non-blocking mode via fnctl? Do you still need to specifically call BIO_set_nbio to make the BIO object non-blocking?

Answer 1

I think most people prefer the BIO interface, but the BIO routines just use whatever native non-blocking socket APIs that are available on the platform. I don't know what happens if you mix and match.

Note that non-blocking I/O for SSL is much trickier than for TCP in general. If you don't understand this going in you're going to be torturing yourself. There are books by John Viega and another by Eric Rescorla that go into this, and you can certainly read the OpenSSL mailing list to get a sense of the heartburn this has caused. Some good code examples showing non-blocking SSL programming with OpenSSL are contained in the software for the TOR project, and the curl utility.