DTLS using Schannel

Question

I am trying to create a DTLS "connection" using Schannel under Windows (I am testing under recent Windows 10 version, so all DTLS versions supported by Schannel should be available)

I tried starting from working code to establish a regular TLS connection by following the documentation:

1. InitializeSecurityContext with null input on the first pass, SECBUFFER_TOKEN & SECBUFFER_ALERT on output
2. AcceptSecurityContext with SECBUFFER_TOKEN & SECBUFFER_EMPTY on input, SECBUFFER_TOKEN & SECBUFFER_ALERT on output.
3. Repeat the two steps until they succeed, and then move on to using Encrypt/DecryptMessage

This works perfectly fine in stream mode (ISC_REQ_SEQUENCE_DETECT | ISC_REQ_REPLAY_DETECT | ISC_REQ_CONFIDENTIALITY | ISC_RET_EXTENDED_ERROR | ISC_REQ_ALLOCATE_MEMORY | ISC_REQ_STREAM)

If I try to substitute STREAM with ISC/ASC_REQ_DATAGRAM, my InitializeSecurityContext succeeds with SEC_I_CONTINUE_NEEDED as expected, but my very first AcceptSecurityContext then fails with SEC_E_INVALID_PARAMETER.

I have tried setting grbitEnabledProtocols of my SCHANNEL_CRED to 0 to use the defaults as documented on both sides, I also tried setting it to SP_PROT_DTLS1_X, and I still get the Invalid Parameter return from my first ASC. I have also tried the DTLS_1_0 constants just in case.

Also, all security protocols are enabled by default in my registry settings.

From my understanding of the DTLS RFC, my code is failing at the HelloVerifyRequest step, and, again from my understanding of the RFC, this part requires that the security provider generates a cookie from a few parts of the ClientHello message as well as the source's IP address. However, I could not find any documented way to pass this information to the ASC function.

(I think? :) ) I have searched the entire internet for any working example usage of DTLS under Schannel without any luck. On stackoverflow, I found this question that simply mentions that it is supported: Is DTLS supported by Schannel on Windows 7?, and the linked MSDN article is just a high level overview.

I searched for any usage of the constants that are related to this feature... I searched for any usage of the constants that are related to this (ISC_REQ_DATAGRAM, SP_PROT_DTLS*, SECBUFFER_DTLS_MTU, ...) and the only thing I could find on all search engines I could think of were either copies of sspi.h or sites that index the constants in the Windows API...

I know DTLS is well supported by other implementations (OpenSSL etc), but I would really prefer to stay with Schannel, as other parts of my code currently work just fine with Schannel in TLS mode.

Answer 1

From Microsoft: There is no documentation for implementing DTLS using Schannel. Known and persistent doc gap.

There are a few differences, but a TLS client or server can be adapted to DTLS fairly easily (a number of customers have done this successfully).

1. Set SCHANNEL_CRED.grbitEnabledProtocols to SP_PROT_DTLS1_X.
2. When calling AcceptSecurityContext, pass the client’s SOCKADDR via SECBUFFER_EXTRA.
3. MTU can be set via SetContextAttributes using constant SECPKG_ATTR_DTLS_MTU where the buffer is just an pointer to a ULONG. [Default is 1096 bytes.]
4. When ISC/ASC return SEC_I_MESSAGE_FRAGMENT, send this fragment and call ISC/ASC again, in a loop, to get the next fragment (without trying to read data from the network).
5. Implement timeout and retransmit logic in your application (since schannel does not own the socket).
6. When receiving fragments, schannel will attempt to eliminate duplicates, re-order and re-assemble, if possible.
7. SCHANNEL_SHUTDOWN does not apply to DTLS.