Why is HttpWebResponse losing data?

Question

In another question, people are getting incomplete data when reading from a HttpWebResponse via GetResponseStream().

I too encountered this problem when reading data from an embedded device which should send me the configuration of 1000 inputs, all in all 32 bytes header and 64 bytes * 1000 resulting in 64032 bytes of data.

Reading the response stream directly only gives me data for the first 61 and a half inputs, from there on only zeros.

Version a) Not working:

int headerSize = 32;
int inputSize = 64;
byte[] buffer = new byte[(inputSize*1000) + headerSize];

HttpWebResponse response = (HttpWebResponse)request.GetResponse();

using (Stream stream = response.GetResponseStream())
{
    if (stream != null)
    {
        stream.Seek(0, SeekOrigin.Begin);
        stream.Read(buffer, 0, buffer.Length);
    }
}

response.Close();
return buffer;

To visualize the problem, I printed the 64 bytes for each input configuration seperately. It consists basically of 40 ascii chars and a few bytes which represent boolean and integer values.

Version A) Output:

1/1000 | 46656E7374657220576F686E656E2020202020202020202020202020202020202020202020202020000000000F0EB0AA00008100000001800000100090010020
2/1000 | 42574D20576F686E656E202020202020202020202020202020202020202020202020202020202020000000000F0EB0AA00008100000001800000100091010080
…
61/1000 | 53656E736F72203631202020202020202020202020202020202020202020202020202020202020200000000000000000000010003300000000001000C3010000
62/1000 | 53656E736F7220363220202020202020202020202020202020202020202020200000000000000000000000000000000000000000000000000000000000000000
63/1000 | 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
…
999/1000 | 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
1000/1000 | 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000

When I copy the ResponseStream to a new MemoryStream, I can read all 1000 inputs completely without any corrupt bytes.

Version B) Working perfectly:

(See also https://stackoverflow.com/a/22354617/6290907 which fixed my problem in the first case)

int headerSize = 32;
int inputSize = 64;
byte[] buffer = new byte[(inputSize*1000) + headerSize];

HttpWebResponse response = (HttpWebResponse)request.GetResponse();

using (Stream stream = response.GetResponseStream())
{
    if (stream != null)
    {
        MemoryStream memStream = new MemoryStream();
        stream.CopyTo(memStream);
        memStream.Flush();
        stream.Close();

        memStream.Seek(0, SeekOrigin.Begin);
        memStream.Read(buffer, 0, buffer.Length);

        memStream.Close();
    }
}

response.Close();
return buffer;

Version B) Output

1/1000 | 46656E7374657220576F686E656E2020202020202020202020202020202020202020202020202020000000000F0EB0AA00008100000001800000100090010020
2/1000 | 42574D20576F686E656E202020202020202020202020202020202020202020202020202020202020000000000F0EB0AA00008100000001800000100091010080
…
61/1000 | 53656E736F72203631202020202020202020202020202020202020202020202020202020202020200000000000000000000010003300000000001000C3010000
62/1000 | 53656E736F72203632202020202020202020202020202020202020202020202020202020202020200000000000000000000010003300000000001000C3010000
63/1000 | 53656E736F72203633202020202020202020202020202020202020202020202020202020202020200000000000000000000010003300000000001000C3010000
…
999/1000 | 53656E736F7220393939202020202020202020202020202020202020202020202020202020202020000000000000000000001000DA030000000010006A050000
1000/1000 | 53656E736F7220313030302020202020202020202020202020202020202020202020202020202020000000000000000000001000DB030000000010006B050000

From a technical point of view: Why is the HttpWebResponse losing data when accessed directly? I don't just want it to work, but I want to understand why version a fails and version b is successfull while both depend on the same source of data (response.GetResponseStream()). What is happening under the hood in this case?

Thank you for your efforts!

Answer 1

Check the int returned by Stream.Read, as described by the docs:

This can be less than the number of bytes requested if that many bytes are not currently available, or zero (0) if the end of the stream has been reached.

I'm willing to bet that only part of the stream is returned in the first call.

If you called Stream.Read repeatedly, you'd get all the bytes in the end. The http stream is simply loading more slowly than your code is running - it doesn't have time to complete before you call Read.

By using CopyTo with a MemoryStream, the call blocks until the whole stream is read. Wrapping in a StreamReader, then calling ReadToEnd would get the same successful result.