How to make a C# application using HttpWebRequest behave like fiddler

Question

I have a console app that uses 20 or so threads to connect to a remote web server and send arbitrary http requests rather small in size, 100% over ssl. The remote web server is actually an entire load balanced data center full of high availability systems that can handle hundreds of thousands of request per second. This is not a server or bandwidth issue. With that being said, I don't run it, nor do i have any influence in how it is configured, so I couldn't make server side changes even if I wanted to.

When running the app with fiddler the app performs amazingly fast. When not running in fiddler its really much slower, to the point of being useless for the task at hand. It also seems to lock up at some point rather early in the process, but this could simply be a deadlock issue, im not sure yet.

Anyhow, fiddler being a proxy , is undoubtedly modifying my requests/connections in some way that ensures wonderful throughput, however I have no idea what its doing. I am trying to figure it out so that I can force my .net application to mimic fiddlers connection handling behavior without actually having to run it through fiddler

I've pasted the connection code below.

     using System;
     using System.Collections.Generic;
     using System.Linq;
     using System.Text;
     using System.Net;
     using System.IO;

     namespace Redacted
     {
        public class HiveCommunicator
        {

           public static IResponse SendRequest(IRequest request) {

              ServicePointManager.DefaultConnectionLimit = 60;
              ServicePointManager.Expect100Continue = false;


              string hostUrlString = string.Empty;
              if (request.SiteID <= 0)
                 hostUrlString = string.Format("{0}://{1}{2}", request.UseSSL ? "https" : "http", DataCenters.GetCenter(request.DataCenter), request.Path);
              else
                 hostUrlString = string.Format("{0}://{1}{2}", request.UseSSL ? "https" : "http", DataCenters.GetCenter(request.DataCenter), string.Format(request.Path, request.SiteID));

              HttpWebRequest webRequest = (HttpWebRequest)HttpWebRequest.Create(hostUrlString);

              switch (request.ContentType)
              {
                 default:
                 case ContentTypes.XML:
                    webRequest.ContentType = "application/xml";
                    break;
                 case ContentTypes.JSON:
                    webRequest.ContentType = "application/json";
                    break;
                 case ContentTypes.BINARY:
                    webRequest.ContentType = "application/octet-stream";
                    break;
              }

              if (request.RequiresAuthorizationToken)
              {
                 AuthorizationToken tok = HiveAuthentication.GetToken(request.SiteID);
                 if (tok == null)
                 {
                    return null;
                 }
                 webRequest.Headers.Add(HttpRequestHeader.Authorization, tok.Token);
              }

              bool UsesRequestBody = true;

              switch (request.HttpVerb)
              {
                 case HttpVerbs.POST:
                    webRequest.Method = "POST";
                    break;
                 case HttpVerbs.DELETE:
                    webRequest.Method = "DELETE";
                    UsesRequestBody = false;
                    break;
                 case HttpVerbs.PUT:
                    webRequest.Method = "PUT";
                    break;
                 default:
                 case HttpVerbs.GET:
                    webRequest.Method = "GET";
                    UsesRequestBody = false;
                    break;
              }

              HttpWebResponse webResponse = null;
              Stream webRequestStream = null;

              byte[] webRequestBytes = null;
              if (UsesRequestBody)
              {
                 webRequestBytes = request.RequestBytes;
                 webRequest.ContentLength = webRequestBytes.Length;
                 webRequestStream = webRequest.GetRequestStream();
                 for (int i = 0; i < webRequest.ContentLength; i++)
                 {
                    webRequestStream.WriteByte(webRequestBytes[i]);
                 }
              }

              try
              {
                 webResponse = (HttpWebResponse)webRequest.GetResponse();
              }
              catch (WebException ex)
              {

                 webResponse = (HttpWebResponse)ex.Response;
              }

              if (UsesRequestBody)
              {
                 webRequestStream.Close();
                 webRequestStream.Dispose();
              }

              IResponse respReturn = request.ParseResponse(webResponse);
              webResponse.Close();

              return respReturn;
           }
        }
     }

Answer 1

I thank the folks here who tried to help. Unfortunately this needed a call to Microsoft Profesional Support.

Even though I was using ServicePointManager.Expect100Continue = false; It was happening to late in the app life cycle. Looking at the System.Net.Trace logs we saw that the expect-100 continue header was still being used (except when using fiddler). The solution was to put this into the app startup (in Main())

I was also trying to read the response stream before closing the request stream.

After fixing that, everything sped up nicely. The app runs much faster without fiddler than with, which is what i would expect.

A couple people said to call dispose on on the HttpWebResponse. That class does not have a public Dispose method. I'm assuming .Close() calls .Dispose() internally though.

Answer 2

You can play around with Fiddler's "Connection Options" to see if the reason for Fiddler's powerfull throughput is reusing of client connections. If that's the case, you may want to consider implementing a shared secure http connection pool or just go watch a movie or something. ^^