Task gets stuck in "[Scheduled and Waiting to Run]"

Question

I've run into an issue with tasks I can't seem to figure out. This application makes repeated HTTP calls via WebClient to several servers. It maintains a dictionary of tasks that are running the HTTP calls, and every five seconds it checks for results, then once the results are in it makes an HTTP call again. This goes on for the lifetime of the application.

Recently, it has started having a problem where the tasks are randomly getting stuck in WaitingForActivation. In the debugger the task shows as "[Scheduled and waiting to run]", but it never runs.

This is the function that it's running, when I click on the "Scheduled" task in the debugger, it points to the DownloadStringTaskAsync() line:

private static async Task<string> DownloadString(string url)
{
    using (var client = new WebClient()) {
        try {
            var result = await client.DownloadStringTaskAsync(url).ConfigureAwait(false);
            return result;
        } catch (WebException) {
            return null;
        }
    }
}

The code that is actually creating the task that runs the above function is this. It only hits this line once the existing task is completed, Task.IsCompleted never returns true since it's stuck in scheduled status. Task.Status gets stuck in WaitingForActivation.

tasks[resource] = Task.Run(() => DownloadString("http://" + resources[resource] + ":8181/busy"));

The odd thing about this is that, as far as I can tell, this code ran perfectly fine for two years, until we recently did a server migration which included an upgraded OS and spawned a network packet loss issue. That's when we started noticing this particular problem, though I don't see how either of those would be related.

Also, this tends to only happen after the application has been running for several thousand seconds. It runs perfectly fine for a while until tasks, one-by-one, start getting stuck. After about a day, there's usually four or five tasks stuck in scheduled. Since it usually takes time for the first task to get stuck, that seems to me like there would be a race condition of some sort, but I don't see how that could be the case.

Is there a reason a task would get stuck in scheduled and never actually run?

Answer 1

I'm not familiar with ancient WebClient (maybe it contains bugs) but can suggest the recommended by Microsoft way to get a response from a server using System.Net.Http.HttpClient. Also HttpClient is rather faster works with multiple requests per endpoint, especially in .NET Core/.NET 5.

// HttpClient is intended to be instantiated once per application, rather than per-use
private static readonly HttpClient client = new HttpClient();

private static async Task<string> DownloadString(string url)
{
    try
    {
        return await client.GetStringAsync(url).ConfigureAwait(false);
    }
    catch (HttpRequestException ex)
    {
        Debug.WriteLine(ex.Message);
        return null;
    }
}

Also remove Task.Run, it's a kind of redundancy.

tasks[resource] = DownloadString($"http://{resources[resource]}:8181/busy");

Asynchronous programming - read the article. You have to get a difference between I/O-bound and CPU-bound work, and don't spawn Threads without special need for concurrency. You need no Thread here.