What is causing intermittent SEC_E_BUFFER_TOO_SMALL error coming from WinHttpSendRequest?

Question

I have a tool which executes an HTTPS POST command against the same URL with same headers, same post body, etc. for a number of iterations.

What I have run into is that for some testers, every so often the WinHttpSendRequest() function fails and the subsequent call to GetLastError() returns SEC_E_BUFFER_TOO_SMALL (0x80090321) documented here: COM Error Codes (Security and Setup).

This is not a documented error code for WinHttpSendRequest() and fairly extensive Googling has not turned up anything at all for this.

I have quadruple checked that the inputs I am providing WinHttpSendRequest() are correct and valid, and those inputs work tens of thousands of times in a row... until it doesn't.

I cannot provide a MVCE, but under the assumptions provided here, Im looking for any possible reason for the error code coming back.

Answer 1

I was provided an answer by someone offline and find it very interesting.

During the key exchange that occurs in TLS 1.2 with RSA+ECDHE, the 256-byte (2048-bit) public modulus integer of ECDHE is generated randomly, and as such it will occasionally have a high order byte of zero. In this situation, the server being used (some Linux box with OpenSSL, do not know distro or version of anything) sends the integer using 255 bytes instead of 256.

The WinHTTP code which receives the public modulus integer in its slightly shorter form apparently does not handle it correctly. It is worth noting that I have not yet seen this problem reproduced on Windows 7 with all software updates, but see it quite often on Windows 8 (Windows 10 not yet tested).

This bug report in Microsoft Edge confirms the same behavior, just with a 1024-bit modulus instead of 2048, but probably it is the same problem:

TLS ServerKeyExchange with 1024 DHE may encode dh_Y as 127 bytes, breaking Internet Explorer 11

However, it does make me wonder if OpenSSL should be padding the integer. I have not looked for the actual spec to see what is the allowed behavior(s) in this situation.