A websocket's ReceiveAsync method does not await the entire message

Question

I am receiving JSON through a websocket. At least: I am partially. Using an online websocket service I receive the full JSON response (all the HTML markup is ignored). When I look at the JSON that I receive in my console I can see the HTML markup (viewing it with the HTML viewer during debugging removes the HTML) but it ends abruptly (incomplete data).

My buffer has plenty of space and I am using async-await to (supposedly) wait for the entire response to come in before continuing.

private async Task Receive()
{
  var buffer = new byte[4096 * 20];

  while (_socket.State == WebSocketState.Open)
  {
      var response = await _socket.ReceiveAsync(new ArraySegment<byte>(buffer), CancellationToken.None);

      if (response.MessageType == WebSocketMessageType.Close)
      {
          await
              _socket.CloseAsync(WebSocketCloseStatus.NormalClosure, "Close response received",
                  CancellationToken.None);
      }
      else
      {
          var result = Encoding.UTF8.GetString(buffer);
          var a = buffer[1000];
          var b = buffer[10000];
          var c = buffer[50000];
          var d = buffer[81000];
          Console.WriteLine(result);
          var responseObject = JsonConvert.DeserializeObject<Response>(result, _requestParameters.ResponseDataType);

          OnSocketReceive.Invoke(this, new SocketEventArgs {Response = responseObject });
          buffer = new byte[4096 * 20];
      }
  }
}   

Things to note: The buffer is perfectly big enough and b, c and d are never filled. I should also note that this only happens for the 1-questions-newest-tag-java request, 155-questions-active works perfectly fine.

After doing some digging I have found that response.CloseStatus and response.CloseStatusDescription are always null, response.Count is always 1396 (copy-pasting the result in Word does show that there are always 1396 characters) and response.EndOfMessage is false.

Digging through some source code I have found that the DefaultReceiveBufferSize is 16 * 1024 (big enough) and the WebSocketGetDefaultKeepAliveInterval() refers to an external implementation (but the debugger shows 00:00:30).

It is not a matter of timeout since the debugger halts at the same moment the online service receives its response.

Why is my method continuing to execute when the socket has not yet received all data?

Answer 1

Just to complete @Noseratio response, the code would be something like this:

ArraySegment<Byte> buffer = new ArraySegment<byte>(new Byte[8192]);

WebSocketReceiveResult result= null;

using (var ms = new MemoryStream())
{
     do
     {
         result = await socket.ReceiveAsync(buffer, CancellationToken.None);
         ms.Write(buffer.Array, buffer.Offset, result.Count);
     }
     while (!result.EndOfMessage);

     ms.Seek(0, SeekOrigin.Begin);

     if (result.MessageType == WebSocketMessageType.Text)
     {
          using (var reader = new StreamReader(ms, Encoding.UTF8))
          {
               // do stuff
          }
     }
}

Cheers.