System.Net (HttpWebRequest) tracing without using files or app.config?

Question

I want to capture certain, but not all, HttpWebRequest traffic in my application for debugging purposes. It's a web service hosted by IIS.

I have read How to: Configure Network Tracing. This works great, but I don't want to direct the trace to a file, due to possible permission problems on the file system, data sensitivity, etc. I would like to capture directly to a something in-memory that I can subsequently inspect or encrypt and email. Preferably, this would not involve any changes to the app.config file.

I tried the following, but obviously I am missing a step to tie the TextWriterTraceListener into System.Net. How can I capture the System.Net traffic into my StringWriter?

StringWriter sw = new StringWriter();
TextWriterTraceListener myListener = new TextWriterTraceListener(sw);
Trace.Listeners.Add(myListener);
HttpWebRequest req = (HttpWebRequest) WebRequest.Create("http://www.microsoft.com");
HttpWebResponse resp = (HttpWebResponse) req.GetResponse();
Stream s = resp.GetResponseStream();

byte[] buf = new byte[4096];
while (s.Read(buf, 0, buf.Length) > 0) ;
s.Close();

myListener.Flush();
sw.Flush();

Edit: Specifically, I want to do the equivalent of this at runtime, except that I don't want output to go to network.log, I want it to go to a string buffer I've set up for this purpose.

<configuration>

<system.diagnostics>

<sources>
  <source name="System.Net.Sockets" tracemode="includehex">
    <listeners>
      <add name="System.Net.Sockets" type="System.Diagnostics.TextWriterTraceListener" initializeData="network.log" />
    </listeners>
  </source>
</sources>

<switches>
  <add name="System.Net.Sockets" value="Verbose"/>
</switches>

<trace autoflush="true" />
</system.diagnostics>
</configuration>

Answer 1

Big thanks @LMK, that's nice. I had the same problem, because i want to log network traffic for error analysis in code.

With your VB-Code, adapted to C# i have wrote this method:

/// <summary>
/// Executes a action with enabled System.Net.Logging with listener(s) at the code-site
/// 
/// Message from Microsoft:
/// To configure you the listeners and level of logging for a listener you need a reference to the listener that is going to be doing the tracing. 
/// A call to create a new TraceSource object creates a trace source with the same name as the one used by the System.Net.Sockets classes, 
/// but it's not the same trace source object, so any changes do not have an effect on the actual TraceSource object that System.Net.Sockets is using.
/// </summary>
/// <param name="webTraceSourceLevel">The sourceLevel for the System.Net traceSource</param>
/// <param name="httpListenerTraceSourceLevel">The sourceLevel for the System.Net.HttpListener traceSource</param>
/// <param name="socketsTraceSourceLevel">The sourceLevel for the System.Net.Sockets traceSource</param>
/// <param name="cacheTraceSourceLevel">The sourceLevel for the System.Net.Cache traceSource</param>
/// <param name="actionToExecute">The action to execute</param>
/// <param name="listener">The listener(s) to use</param>
public static void ExecuteWithEnabledSystemNetLogging(SourceLevels webTraceSourceLevel, SourceLevels httpListenerTraceSourceLevel, SourceLevels socketsTraceSourceLevel, SourceLevels cacheTraceSourceLevel, Action actionToExecute, params TraceListener[] listener)
{
    if (listener == null)
    {
        throw new ArgumentNullException("listener");
    }

    if (actionToExecute == null)
    {
        throw new ArgumentNullException("actionToExecute");
    }

    var logging = typeof(WebRequest).Assembly.GetType("System.Net.Logging");
    var isInitializedField = logging.GetField("s_LoggingInitialized", BindingFlags.NonPublic | BindingFlags.Static);
    if (!(bool)isInitializedField.GetValue(null))
    {
        //// force initialization
        HttpWebRequest.Create("http://localhost");
        Thread waitForInitializationThread = new Thread(() =>
        {
            while (!(bool)isInitializedField.GetValue(null))
            {
                Thread.Sleep(100);
            }
        });

        waitForInitializationThread.Start();
        waitForInitializationThread.Join();
    }

    var isEnabledField = logging.GetField("s_LoggingEnabled", BindingFlags.NonPublic | BindingFlags.Static);
    var webTraceSource = (TraceSource)logging.GetField("s_WebTraceSource", BindingFlags.NonPublic | BindingFlags.Static).GetValue(null);
    var httpListenerTraceSource = (TraceSource)logging.GetField("s_HttpListenerTraceSource", BindingFlags.NonPublic | BindingFlags.Static).GetValue(null);
    var socketsTraceSource = (TraceSource)logging.GetField("s_SocketsTraceSource", BindingFlags.NonPublic | BindingFlags.Static).GetValue(null);
    var cacheTraceSource = (TraceSource)logging.GetField("s_CacheTraceSource", BindingFlags.NonPublic | BindingFlags.Static).GetValue(null);

    bool wasEnabled = (bool)isEnabledField.GetValue(null);
    Dictionary<TraceListener, TraceFilter> originalTraceSourceFilters = new Dictionary<TraceListener, TraceFilter>();

    //// save original Levels
    var originalWebTraceSourceLevel = webTraceSource.Switch.Level;
    var originalHttpListenerTraceSourceLevel = httpListenerTraceSource.Switch.Level;
    var originalSocketsTraceSourceLevel = socketsTraceSource.Switch.Level;
    var originalCacheTraceSourceLevel = cacheTraceSource.Switch.Level;

    //System.Net
    webTraceSource.Listeners.AddRange(listener);
    webTraceSource.Switch.Level = SourceLevels.All;
    foreach (TraceListener tl in webTraceSource.Listeners)
    {
        if (!originalTraceSourceFilters.ContainsKey(tl))
        {
            originalTraceSourceFilters.Add(tl, tl.Filter);
            tl.Filter = new ModifiedTraceFilter(tl, originalWebTraceSourceLevel, webTraceSourceLevel, originalHttpListenerTraceSourceLevel, httpListenerTraceSourceLevel, originalSocketsTraceSourceLevel, socketsTraceSourceLevel, originalCacheTraceSourceLevel, cacheTraceSourceLevel, listener.Contains(tl));
        }
    }

    //System.Net.HttpListener
    httpListenerTraceSource.Listeners.AddRange(listener);
    httpListenerTraceSource.Switch.Level = SourceLevels.All;
    foreach (TraceListener tl in httpListenerTraceSource.Listeners)
    {
        if (!originalTraceSourceFilters.ContainsKey(tl))
        {
            originalTraceSourceFilters.Add(tl, tl.Filter);
            tl.Filter = new ModifiedTraceFilter(tl, originalWebTraceSourceLevel, webTraceSourceLevel, originalHttpListenerTraceSourceLevel, httpListenerTraceSourceLevel, originalSocketsTraceSourceLevel, socketsTraceSourceLevel, originalCacheTraceSourceLevel, cacheTraceSourceLevel, listener.Contains(tl));
        }
    }

    //System.Net.Sockets
    socketsTraceSource.Listeners.AddRange(listener);
    socketsTraceSource.Switch.Level = SourceLevels.All;
    foreach (TraceListener tl in socketsTraceSource.Listeners)
    {
        if (!originalTraceSourceFilters.ContainsKey(tl))
        {
            originalTraceSourceFilters.Add(tl, tl.Filter);
            tl.Filter = new ModifiedTraceFilter(tl, originalWebTraceSourceLevel, webTraceSourceLevel, originalHttpListenerTraceSourceLevel, httpListenerTraceSourceLevel, originalSocketsTraceSourceLevel, socketsTraceSourceLevel, originalCacheTraceSourceLevel, cacheTraceSourceLevel, listener.Contains(tl));
        }
    }

    //System.Net.Cache
    cacheTraceSource.Listeners.AddRange(listener);
    cacheTraceSource.Switch.Level = SourceLevels.All;
    foreach (TraceListener tl in cacheTraceSource.Listeners)
    {
        if (!originalTraceSourceFilters.ContainsKey(tl))
        {
            originalTraceSourceFilters.Add(tl, tl.Filter);
            tl.Filter = new ModifiedTraceFilter(tl, originalWebTraceSourceLevel, webTraceSourceLevel, originalHttpListenerTraceSourceLevel, httpListenerTraceSourceLevel, originalSocketsTraceSourceLevel, socketsTraceSourceLevel, originalCacheTraceSourceLevel, cacheTraceSourceLevel, listener.Contains(tl));
        }
    }

    isEnabledField.SetValue(null, true);

    try
    {
        actionToExecute();
    }
    finally
    {
        //// restore Settings
        webTraceSource.Switch.Level = originalWebTraceSourceLevel;
        httpListenerTraceSource.Switch.Level = originalHttpListenerTraceSourceLevel;
        socketsTraceSource.Switch.Level = originalSocketsTraceSourceLevel;
        cacheTraceSource.Switch.Level = originalCacheTraceSourceLevel;

        foreach (var li in listener)
        {
            webTraceSource.Listeners.Remove(li);
            httpListenerTraceSource.Listeners.Remove(li);
            socketsTraceSource.Listeners.Remove(li);
            cacheTraceSource.Listeners.Remove(li);
        }

        //// restore filters
        foreach (var kvP in originalTraceSourceFilters)
        {
            kvP.Key.Filter = kvP.Value;
        }

        isEnabledField.SetValue(null, wasEnabled);
    }
}

The class ModifiedTraceFilter:

public class ModifiedTraceFilter : TraceFilter
{
    private readonly TraceListener _traceListener;

    private readonly SourceLevels _originalWebTraceSourceLevel;

    private readonly SourceLevels _originalHttpListenerTraceSourceLevel;

    private readonly SourceLevels _originalSocketsTraceSourceLevel;

    private readonly SourceLevels _originalCacheTraceSourceLevel;

    private readonly SourceLevels _modifiedWebTraceTraceSourceLevel;

    private readonly SourceLevels _modifiedHttpListenerTraceSourceLevel;

    private readonly SourceLevels _modifiedSocketsTraceSourceLevel;

    private readonly SourceLevels _modifiedCacheTraceSourceLevel;

    private readonly bool _ignoreOriginalSourceLevel;

    private readonly TraceFilter _filter = null;

    public ModifiedTraceFilter(TraceListener traceListener, SourceLevels originalWebTraceSourceLevel, SourceLevels modifiedWebTraceSourceLevel, SourceLevels originalHttpListenerTraceSourceLevel, SourceLevels modifiedHttpListenerTraceSourceLevel, SourceLevels originalSocketsTraceSourceLevel, SourceLevels modifiedSocketsTraceSourceLevel, SourceLevels originalCacheTraceSourceLevel, SourceLevels modifiedCacheTraceSourceLevel, bool ignoreOriginalSourceLevel)
    {
        _traceListener = traceListener;
        _filter = traceListener.Filter;
        _originalWebTraceSourceLevel = originalWebTraceSourceLevel;
        _modifiedWebTraceTraceSourceLevel = modifiedWebTraceSourceLevel;
        _originalHttpListenerTraceSourceLevel = originalHttpListenerTraceSourceLevel;
        _modifiedHttpListenerTraceSourceLevel = modifiedHttpListenerTraceSourceLevel;
        _originalSocketsTraceSourceLevel = originalSocketsTraceSourceLevel;
        _modifiedSocketsTraceSourceLevel = modifiedSocketsTraceSourceLevel;
        _originalCacheTraceSourceLevel = originalCacheTraceSourceLevel;
        _modifiedCacheTraceSourceLevel = modifiedCacheTraceSourceLevel;
        _ignoreOriginalSourceLevel = ignoreOriginalSourceLevel;
    }

    public override bool ShouldTrace(TraceEventCache cache, string source, TraceEventType eventType, int id, string formatOrMessage, object[] args, object data1, object[] data)
    {
        SourceLevels originalTraceSourceLevel = SourceLevels.Off;
        SourceLevels modifiedTraceSourceLevel = SourceLevels.Off;

        if (source == "System.Net")
        {
            originalTraceSourceLevel = _originalWebTraceSourceLevel;
            modifiedTraceSourceLevel = _modifiedWebTraceTraceSourceLevel;
        }
        else if (source == "System.Net.HttpListener")
        {
            originalTraceSourceLevel = _originalHttpListenerTraceSourceLevel;
            modifiedTraceSourceLevel = _modifiedHttpListenerTraceSourceLevel;
        }
        else if (source == "System.Net.Sockets")
        {
            originalTraceSourceLevel = _originalSocketsTraceSourceLevel;
            modifiedTraceSourceLevel = _modifiedSocketsTraceSourceLevel;
        }
        else if (source == "System.Net.Cache")
        {
            originalTraceSourceLevel = _originalCacheTraceSourceLevel;
            modifiedTraceSourceLevel = _modifiedCacheTraceSourceLevel;
        }

        var level = ConvertToSourceLevel(eventType);
        if (!_ignoreOriginalSourceLevel && (originalTraceSourceLevel & level) == level)
        {
            if (_filter == null)
            {
                return true;
            }
            else
            {
                return _filter.ShouldTrace(cache, source, eventType, id, formatOrMessage, args, data1, data);
            }
        }
        else if (_ignoreOriginalSourceLevel && (modifiedTraceSourceLevel & level) == level)
        {
            if (_filter == null)
            {
                return true;
            }
            else
            {
                return _filter.ShouldTrace(cache, source, eventType, id, formatOrMessage, args, data1, data);
            }
        }

        return false;
    }

    private static SourceLevels ConvertToSourceLevel(TraceEventType eventType)
    {
        switch (eventType)
        {
            case TraceEventType.Critical:
                return SourceLevels.Critical;
            case TraceEventType.Error:
                return SourceLevels.Error;
            case TraceEventType.Information:
                return SourceLevels.Information;
            case TraceEventType.Verbose:
                return SourceLevels.Verbose;
            case TraceEventType.Warning:
                return SourceLevels.Warning;
            default:
                return SourceLevels.ActivityTracing;
        }
    }
}

Have fun, Marko

Answer 2

Here's how to wireup System.Net logging in code via reflection. Code is in VB but is trivial to convert to c#...

Dim logging = GetType(Net.HttpWebRequest).Assembly.GetType("System.Net.Logging")
Dim enabled = logging.GetField("s_LoggingEnabled", Reflection.BindingFlags.NonPublic Or  Reflection.BindingFlags.Static)
enabled.SetValue(Nothing, True)
Dim webTr = logging.GetProperty("Web", Reflection.BindingFlags.NonPublic Or Reflection.BindingFlags.Static)
Dim tr as TraceSource = webTr.GetValue(Nothing, Nothing)
tr.Switch.Level = SourceLevels.Verbose
tr.Listeners.Add(New MyTraceListener())

Put this in Global.asax Application_Start() with whatever conditions you want to turn it on. You may need to Flush() tr before reading.

Answer 3

You can create your own TraceListener implementation. The example(s) I have found online that configure things at run-time do not show working with the system trace sources. If you don't mind getting your hands dirty you could try using reflection to toggle the private static bool System.Net.Logging.s_LoggingEnabled (.NET 2).

Take the example in the following article and switch it from sending e-mails to publishing a static event that you can subscribe to when you're interested in receiving trace messages:

Extending System.Diagnostics

This does cause a performance hit since logging is enabled all the time (all or nothing as configured in the web.config). (See this article and comments explaining the importance of removing the default trace to improve performance.)