System.Timers.Timer Elapsed intermittently not firing when using an Task.Run with async from Console App

Question

I am using a console application and I have batches of 20 URIs that I need to read from and I have found a massive speed boost by making all tasks and running them in parallel then sorting the results on completion in a different thread (allowing the next batch to be fetched).

In the calls I am currently using, each thread blocks when it gets the response stream, I also see there is a async version of the same method GetResponseAsync.

I understand there are benefits of freeing up the thread pool by using async Await and Async in same line instead of blocking:

Async version

return Task.Run(async () =>
{
    var uri = item.Links.Alternate();
    var request = (HttpWebRequest)WebRequest.Create(uri);

    var response = await request.GetResponseAsync();
    var stream = response.GetResponseStream();
    if (stream == null) return null;
    var reader = new StreamReader(stream);
    return new FetchItemTaskResult(reader.ReadToEnd(), index, uri);
});

Blocking version

return Task<FetchItemTaskResult>.Factory.StartNew(() =>
{
    var uri = item.Links.Alternate();
    var request = (HttpWebRequest)WebRequest.Create(uri);

    var response = request.GetResponse();
    var stream = response.GetResponseStream();
    if (stream == null) return null;
    var reader = new StreamReader(stream);
    return new FetchItemTaskResult(reader.ReadToEnd(), index, uri);
});

However I am seeing strange pauses on the console app with the async version where a System.Timers.Timer elapsed event stops being called for a many seconds (when it should go off every second).

The blocking one runs at around 3,500 items per second, CPU usage is at ~30% across all cores.

The async on runs at around 3,800 events per second, CPU usage is a little higher than the blocking but not by much (only 5%)... however my timer I am using seems to pause for around 10 to 15 seconds once every minute or so, in my Main() function:

private static void Main(string[] args)
{  
    // snip some code that runs the tasks

    var timer = new System.Timers.Timer(1000);

    timer.Elapsed += (source, e) =>
    {
        Console.WriteLine(DateTime.UtcNow);

        // snip non relevant code
        Console.WriteLine("Commands processed: " + commandsProcessed.Sum(s => s.Value) + " (" + logger.CommandsPerSecond() + " per second)");
    };
    timer.Start();
    Console.ReadKey();
}

So would seem the timer and thread pool are some how related when using async (and only async, no pauses when blocking), or perhaps not, either way any ideas what is going on please and how to diagnose further?

Answer 1

the timer and thread pool are some how related

Your suspicion is correct. What is happening is basically called the infamous thread starvation i.e. all your threads are busy so ThreadPool would not have enough threads to run the event delegate.

Once running in ASP.NET, autoConfig="True" makes sure you get enough threads (which is not always true in the case of spikes) and also make sure you are not bound by the connection limits. But in a console app, you have to do that yourself.

So simply add this snippet and I bet your problem will go away:

ThreadPool.SetMinThreads(100, 100);
ServicePointManager.DefaultConnectionLimit = 1000;

Answer 2

The System.Timers.Timer has a (not so) nice feature of posting the Elapsed event onto the thread pool. If all threads are busy or blocked, or the action queue is long, the Elapsed event will not be triggered until it gets the resources to do it.

This is the case even when you use Timer.SynchronizedObject property - see https://docs.microsoft.com/en-us/dotnet/api/system.timers.timer.elapsed?view=net-5.0.

The same goes for System.Threading.Timer. Looks like System.Windows.Threading.DispatcherTimer might be executed on the UI thread - but you can't use it in a console app.

So I guess the only way is to create a dedicated thread for the timer and have a while(true) loop there with Thread.Sleep().

Update

The "dedicated thread timer" class looks roughly like that. Note, I write it from memory, so it probably won't compile just like that.

public sealed class DedicatedTimer
{
  public DedicatedTimer(int interval, Action action)
  {
    mInterval = interval; // Create the field
    mAction = action; // Create the field
    var thr = new Thread(Runner, isBackground); // Check the docs on the background flag
    thr.Start();
  }

  private void Runner()
  {
    while (true)
    {
      Thread.Sleep(mInterval);
      action.Invoke();
    }
  }
}

Using a background thread allows you to forget about it, so it is killed when the program goes down. You might need your own Dispose depending on your case.

As an exercise to the reader, you can separate creation and starting of the timer, you can add a CancellationToken, you can employ a list of timers and calculate the delay to the next in line after every trigger.

Do remember that the action is executed on this thread. If you want to avoid it, use SynchronizationContext.