Unit testing HTTP requests in c#

Question

I'm writing some code that calls a web service, reads back the response and does something with it. My code looks nominally like this:

string body = CreateHttpBody(regularExpression, strategy);

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(_url);
request.Method = "POST";
request.ContentType = "text/plain; charset=utf-8";

using (Stream requestStream = request.GetRequestStream())
{
    requestStream.Write(Encoding.UTF8.GetBytes(body), 0, body.Length);
    requestStream.Flush();
}

using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
    byte[] data = new byte[response.ContentLength];

    using (Stream stream = response.GetResponseStream())
    {
        int bytesRead = 0;

        while (bytesRead < data.Length)
        {
            bytesRead += stream.Read(data, bytesRead, data.Length - bytesRead);
        }
    }

    return ExtractResponse(Encoding.UTF8.GetString(data));
}

The only parts where I am actually doing any custom manipulation is in the ExtractResponse and CreateHttpBody methods. However it feels wrong to just unit test those methods, and hope that the rest of the code comes together correctly. Is there any way I can intercept the HTTP request and feed it mock data instead?

EDIT This information is now out of date. It is much easier to construct this kind of code using the System.Net.Http.HttpClient libraries.

Answer 1

In your code you can not intercept the calls to HttpWebRequest because you create the object in the same method. If you let another object create the HttpWebRequest, you can pass in a mock object and use that to test.

So instead of this:

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(_url);

Use this:

IHttpWebRequest request = this.WebRequestFactory.Create(_url);

In your unit test, you can pass in a WebRequestFactory which creates a mock object.

Furthermore, you can split of your stream reading code in a separate function:

using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
    byte[] data = ReadStream(response.GetResponseStream());
    return ExtractResponse(Encoding.UTF8.GetString(data));
}

This makes it possible to test ReadStream() separately.

To do more of an integration test, you can set up your own HTTP server which returns test data, and pass the URL of that server to your method.

Answer 2

If mocking out the HttpWebRequest and HttpWebResponse becomes too cumbersome, or if you ever need to test code in an acceptance test, where you are calling your code from the "outside", then creating a fake service is probably the best way to go.

I actually wrote an open source library called MockHttpServer to assist with this, making it super simple to mock out any external services that communicate over HTTP.

Here is an example of using it with RestSharp to call an API endpoint with it, but HttpWebRequest would work just as well.

using (new MockServer(3333, "/api/customer", (req, rsp, prm) => "Result Body"))
{
    var client = new RestClient("http://localhost:3333/");
    var result = client.Execute(new RestRequest("/api/customer", Method.GET));
}

There is a fairly detailed readme on the GitHub page that goes through all the options available for using it, and the library itself is available through NuGet.

Answer 3

I would probably start by refactoring the code in order to make it more weakly coupled to an actual HTTP request. Right now this code seems to do quite a lot of things.

This could be done by introducing an abstraction:

public interface IDataRetriever
{
    public byte[] RetrieveData(byte[] request);
}

Now the class that you are trying to unit test could be decoupled from the actual HTTP request using the Inversion of Control design pattern:

public class ClassToTest
{
    private readonly IDataRetriever _dataRetriever;
    public Foo(IDataRetriever dataRetriever)
    {
        _dataRetriever = dataRetriever;
    }

    public string MethodToTest(string regularExpression, string strategy)
    {
        string body = CreateHttpBody(regularExpression, strategy);
        byte[] result = _dataRetriever.RetrieveData(Encoding.UTF8.GetBytes(body));
        return ExtractResponse(Encoding.UTF8.GetString(result));
    }
}

It is no longer the ClassToTest's responsibility to deal with an actual HTTP request. It is now decoupled. Testing the MethodToTest becomes a trivial task.

And the last part obviously is to have an implementation of the abstraction that we have introduced:

public class MyDataRetriever : IDataRetriever
{
    private readonly string _url;
    public MyDataRetriever(string url)
    {
        _url = url;
    }

    public byte[] RetrieveData(byte[] request)
    {
        using (var client = new WebClient())
        {
            client.Headers[HttpRequestHeader.ContentType] = "text/plain; charset=utf-8";
            return client.UploadData(_url, request);
        }
    }
}

You could then configure your favorite DI framework to inject a MyDataRetriever instance into the ClassToTest class constructor in your actual application.