How to receive HTTP messages using Socket

Question

I'm using Socket class for my web client. I can't use HttpWebRequest since it doesn't support socks proxies. So I have to parse headers and handle chunked encoding by myself. The most difficult thing for me is to determine length of content so I have to read it byte-by-byte. First I have to use ReadByte() to find last header ("\r\n\r\n" combination), then check whether body has transfer-encoding or not. If it does I have to read chunk's size etc:

public void ParseHeaders(Stream stream)
{
    while (true)
    {
        var lineBuffer = new List<byte>();
        while (true)
        {
            int b = stream.ReadByte();
            if (b == -1) return;
            if (b == 10) break;
            if (b != 13) lineBuffer.Add((byte)b);
        }
        string line = Encoding.ASCII.GetString(lineBuffer.ToArray());
        if (line.Length == 0) break;
        int pos = line.IndexOf(": ");
        if (pos == -1) throw  new VkException("Incorrect header format");
        string key = line.Substring(0, pos);
        string value = line.Substring(pos + 2);
        Headers[key] = value;
    }
}

But this approach has very poor performance. Can you suggest better solution? Maybe some open source examples or libraries that handle http request through sockets (not very big and complicated though, I'm a noob). The best would be to post link to example that reads message body and correctly handles the cases when: content has chunked-encoding, is gzip- or deflate-encoded, Content-Length header is omitted (message ends when connection is closed). Something like source code of HttpWebRequest class.

Upd: My new function looks like this:

int bytesRead = 0;
byte[] buffer = new byte[0x8000];
do
{
    try
    {
        bytesRead = this.socket.Receive(buffer);
        if (bytesRead <= 0) break;
        else
        {
            this.m_responseData.Write(buffer, 0, bytesRead);
            if (this.m_inHeaders == null) this.GetHeaders();
        }
    }
    catch (Exception exception)
    {
        throw new Exception("Read response failed", exception);
    }
}
while ((this.m_inHeaders == null) || !this.isResponseBodyComplete());

Where GetHeaders() and isResponseBodyComplete() use m_responseData (MemoryStream) with already received data.

Answer 1

I suggest that you don't implement this yourself - the HTTP 1.1 protocol is sufficiently complex to make this a project of several man-months.

The question is, is there a HTTP request protocol parser for .NET? This question has been asked on SO, and in the answers you'll see several suggestions, including source code for handling HTTP streams.

Converting Raw HTTP Request into HTTPWebRequest Object

EDIT: The rotor code is reasonably complex, and difficult to read/navigate as webpages. But still, the implementaiton effort to add SOCKS supports is much lower than implementing the entire HTTP protocol yourself. You will have something working within a few days at most that you can depend upon, that is based on a tried and tested implementation.

The request and response are read from/written to to a NetworkStream, m_Transport, in the Connection class. This is used in these methods:

internal int Read(byte[] buffer, int offset, int size) 
//and
private static void ReadCallback(IAsyncResult asyncResult)

both in http://www.123aspx.com/Rotor/RotorSrc.aspx?rot=42903

The socket is created in

private void StartConnectionCallback(object state, bool wasSignalled)

So you could modify this method to create a Socket to your socks server, and do the necessary handshake to obtain the external connection. The rest of the code can remain the same.

I gammered this info in about 30 mins looking on the pages on the web. This should go much faster if you load these files into an IDE. It may seem like a burden to have to read through this code - after all, reading code is far harder than writing it, but you are making just small changes to an already established, working system.

To be sure the changes work in all cases, it will be wise to also test when the connection is broken, to ensure that the client reconnects using the same method , and so re-establishes the SOCKS connection and sends the SOCKS request.