Cannot finish handshake due to 'wrong version number' error

Question

I'm new to programming with OpenSSL library and having hard time implementing handshaking using memory BIO. (for use of Overlapped I/O later)

I don't quite understand examples that do not process handshake explicitly, so I made a simple program that only does handshake, but I got this error from server,

9332:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:ssl\record\ssl3_record.c:210:

and from client,

18792:error:141A10F4:SSL routines:ossl_statem_client_read_transition:unexpected message:ssl\statem\statem_clnt.c:269:

Wireshark says both server and client uses TLSv1.2 for record layer, but client is using TLS 1.0 (0x0301) while server is using TLS 1.2 (0x0303)

So I tried to limit TLS version with SSL_set_min_proto_version and SSL_set_max_proto_version (and CTX version of these functions) that TLS1_2_VERSION <= compatible <= TLS1_2_VERSION. But the error still occurs.

Here's snippet,

auto send = [&sock, &session]() -> void
{
    if (session.should_send()) // BIO_ctrl_pending(bio_write)
    {
        std::vector<char> buffer(session.read_from_write_bio());
        sock.send(buffer);
    }
};

auto receive = [&sock, &session]() -> void
{
    std::vector<char> buffer(sock.receive());
    auto status = session.write_to_read_bio(buffer);

    if (status != Status::DONE)
        while (session.should_write_bio_again())  // BIO_should_retry(bio_read)
            status = session.write_to_read_bio(buffer); // BIO_write(bio_read, ...)

    if (session.handshaking())  // SSL_is_init_finished(ssl);
        session.handshake();    // SSL_do_handshake(ssl);

    else // for later use
    {
        std::vector<char> buffer(GetMaxBufferSize(), '\0');
        session.read(buffer);
    }
};

from server,

receive();  // client hello
send();     // server hello
receive();  // client spec
send();     // server spec

from client,

send();     // client hello
receive();  // server hello
send();     // client spec
receive();  // server spec

I'm using OpenSSL 1.1.0h, may be concerned?

Edit:

I downgraded library version (1.0.2n) and it gives me same error so I guess library version does not affect. But I found out that DTLS protocol (specified with DTLS_method when creating SSL context) is only protocol that works with code above.

Other methods fails during handshaking generating these errors,

SSLv23_method
17484:error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number:.\ssl\s3_pkt.c:365:
2812:error:1408E0F4:SSL routines:ssl3_get_message:unexpected message:.\ssl\s3_both.c:408:

TLSv1_2_method
20472:error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number:.\ssl\s3_pkt.c:365:
8580:error:1408E0F4:SSL routines:ssl3_get_message:unexpected message:.\ssl\s3_both.c:408:

TLSv1_1_method
7868:error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number:.\ssl\s3_pkt.c:365:
3748:error:1408E0F4:SSL routines:ssl3_get_message:unexpected message:.\ssl\s3_both.c:408:

TLSv1_method
19008:error:1408E0F4:SSL routines:ssl3_get_message:unexpected message:.\ssl\s3_both.c:408:
9768:error:1408E0F4:SSL routines:ssl3_get_message:unexpected message:.\ssl\s3_both.c:408:

SSLv3_method
13948:error:1408E0F4:SSL routines:ssl3_get_message:unexpected message:.\ssl\s3_both.c:408:
14356:error:1408E0F4:SSL routines:ssl3_get_message:unexpected message:.\ssl\s3_both.c:408:

(SSL context fails to initialize with SSLv2_method)

Hope someone can help me ; )

Answer 1

After some days of digging, I found what I did wrong.

The problem happened when reading data from 'write BIO'.

int size_buffer = BIO_get_mem_data(bio_write, &p);
// copy contents from pointer 'p' to buffer

Code above caused data on 'write BIO' remain even after being read, which caused OpenSSL thought that IO has not processed.

I solved it by changing reading method to,

int size_buffer = BIO_read(bio_write, buffer, buffer.size());

BIO_read clears data on 'write BIO' after reading, so that 'write BIO' does not have any pending data.