Receiving Chunked HTTP Data With Winsock

Question

I'm having trouble reading in some chunked HTTP response data using winsock. I send a request fine and get the following back:

HTTP/1.1 200 OK
Server: LMAX/1.0
Content-Type: text/xml; charset=utf-8
Transfer-Encoding: chunked
Date: Mon, 29 Aug 2011 16:22:19 GMT

using winsock recv. At this point however it just hangs. I have the listener running in an infinite loop but nothing is ever picked up.

I think it's a C++ issue but it could also be related to the fact that I pushing the connection through stunnel to wrap it up inside HTTPS. I have a test application using some libs in C# which works perfectly through stunnel. I'm confused as to why my loop is not receiving the C++ chunked data after the initial recv.

This is the loop in question...it is called after the chunked ok response above...

while(true)
{
    recvBuf= (char*)calloc(DEFAULT_BUFLEN, sizeof(char)); 
    iRes = recv(ConnectSocket, recvBuf, DEFAULT_BUFLEN, 0);
    cout << WSAGetLastError() << endl;
    cout << "Recv: " << recvBuf << endl;
    if (iRes==SOCKET_ERROR)
    {
        cout << recvBuf << endl;
        err = WSAGetLastError();
        wprintf(L"WSARecv failed with error: %d\n", err);
        break;
    }     

}

Any ideas?

Answer 1

You need to change your reading code. You cannot read chunked data using a fixed-length buffer like you are trying to do. The data is sent in variable-length chunks, where each chunk has a header that specifies the actual length of the chunk in bytes, and the final chunk of the data has a length of 0. You need to read the chunked headers in order to process the chunks properly. Please read RFC 2616 Section 3.6.1. Your logic needs to be more like the following pseudo-code:

send request;

status = recv() a line of text until CRLF;
parse status as needed;
response-code = extract response-code from status;
response-version = extract response-version from status;

do
{
    line = recv() a line of text until CRLF;
    if (line is blank)
        break;
    store line in headers list;
}
while (true);

parse headers list as needed;

if ((response-code is not in [1xx, 204, 304]) and (request was not "HEAD"))
{
    if (Transfer-Encoding header is present and not "identity")
    {
        do
        {
            line = recv a line of text until CRLF;
            length = extract length from line;
            extensions = extract extensions from line;
            process extensions as needed; // optional
            if (length == 0)
                break;
            recv() length number of bytes into destination buffer;
            recv() and discard bytes until CRLF;
        }
        while (true);

        do
        {
            line = recv a line of text until CRLF;
            if (line is blank)
                break;
            store line in headers list as needed;
        }
        while (true);

        re-parse headers list as needed;
    }
    else if (Content-Length header is present)
    {
        recv() Content-Length number of bytes into destination buffer;
    }
    else if (Content-Type header starts with "multipart/")
    {
        boundary = extract boundary from Content-Type's "boundary" attribute;
        recv() data into destination buffer until MIME termination boundary is reached;
    }
    else
    {
        recv() data into destination buffer until disconnected;
    }
}

if (not disconnected)
{
    if (response-version is "HTTP/1.1")
    {
        if (Connection header is "close")
            close connection;
    }
    else
    {
        if (Connection header is not "keep-alive")
            close connection;
    }
}

check response-code for errors;
process destination buffer, per info in headers list;