HttpClient - Request times out but I don't understand why

Question

I have put together an example code that shows what I mean. I am trying to download two different datasheets, one from TI, the other from ST. Both exist and I can download them using any browser (even in incognito mode)

I am using the same code for both. One of them times out, the other doesn't and I have absolutely no clue why.

internal class Program
{
    static void Main(string[] args)
    {
        var task = Xx1();
        task.Wait();
    }
 
    private static async Task Xx1()
    {
        HttpClientHandler handler = new HttpClientHandler()
        {
            AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate,
            AllowAutoRedirect = true,
            CheckCertificateRevocationList = false,
            UseProxy = false
        };
 
        var httpClient = new HttpClient(handler);
 
        httpClient.DefaultRequestHeaders.UserAgent.ParseAdd("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36");
        httpClient.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("*/*"));
        httpClient.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("gzip"));
        httpClient.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("deflate"));
        httpClient.Timeout = TimeSpan.FromSeconds(10);
 
        try
        {
            var httpResult1 = await httpClient.GetAsync("https://www.ti.com/lit/gpn/TPS723").ConfigureAwait(false);
            httpResult1.EnsureSuccessStatusCode();
            var resultBytes1 = await httpResult1.Content.ReadAsByteArrayAsync().ConfigureAwait(false);
            Console.WriteLine(resultBytes1.Length);
        }
        catch (TaskCanceledException taskCanceledException)
        {
            Console.WriteLine("1 Request timed out: " + taskCanceledException.Message);
        }
        catch (Exception exception)
        {
            Console.WriteLine("1 An error occurred: " + exception.Message);
        }
 
        try
        {
            var httpResult2 = await httpClient.GetAsync("https://www.st.com/resource/en/datasheet/stlq015.pdf").ConfigureAwait(false);
            httpResult2.EnsureSuccessStatusCode();
            var resultBytes2 = await httpResult2.Content.ReadAsByteArrayAsync().ConfigureAwait(false);
            Console.WriteLine(resultBytes2.Length);
        }
        catch (TaskCanceledException taskCanceledException)
        {
            Console.WriteLine("2 Request timed out: " + taskCanceledException.Message);
        }
        catch (Exception exception)
        {
            Console.WriteLine("2 An error occurred: " + exception.Message);
        }
    }
}

I think it has to do with STs webserver but I don't know what the reason for it is and how I could work around it. Any help is highly appreciated.

Answer 1

Since a WHOIS lookup on the www.st.com domain showed that it uses Akamai Cloud hosting, then confirmed when checking the Response Headers: the Breadcrumb feature Header, akamai-request-bc, is set.
Well, this means some more strict RFC interpretation server-side.
Also distributed resources, so speculation rules are in place. The Access-Control-Allow-Origin Header is set to a fixed location, which is a different HTTP (not HTTPS) resource origin.

We are just pretending to be a Web Browser, e.g., we cannot send preflight requests. See the docs about Cross-Origin Resource Sharing (CORS).

First instruct to disable CORS, setting the Sec-Fetch-Mode Header to no-cors.

To emulate a request coming from a Web Browser, we set the User-Agent Header.
This Header may not be considered enough information for the server to provide accurate content in the response. Chrome (Chromium) browsers also add other headers to provide more specific information on their version, Platform, bitness etc.

We can set the same headers that the Edge Chrome Browser does:
The Sec-Ch-Ua Header (details on who we are) and the Sec-Fetch-Dest (specifies how the data that's being fetched will be used)

• To set the Sec-Ch-Ua header, issue a request (any) with the Web Browser set in the User-Agent Header - keeping the Dev Tools panel open in Network view, then copy the strings you find in the Request Headers.
  If the Web Browser doesn't provide any, pick another :) We cannot handle feature requests.
• The Sec-Fetch-Dest Header is also required and should be set to document, but also to empty (not sure it actually matters what it's set to).

The Accept-Language must also be set, the server may provide documents in different languages (this Header should always be set, IMO, and it doesn't hurt anybody)

Refactored code, targeting .NET 9:

using System.Net;
using System.Net.Http.Headers;

internal class Program {

    // The HttpMessageHandler handles cookies automatically
    private static readonly HttpClient client = new(
        new SocketsHttpHandler() {
            AllowAutoRedirect = true,
            AutomaticDecompression = DecompressionMethods.Brotli |
                                     DecompressionMethods.GZip |
                                     DecompressionMethods.Deflate,
            PooledConnectionLifetime = TimeSpan.FromMinutes(2),
        }
    );

    static async Task Main(string[] args) {
        client.DefaultRequestHeaders.Clear();
        // Edge Chrome User-Agent Header
        client.DefaultRequestHeaders.UserAgent.ParseAdd("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 Edg/137.0.0.0");

        client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("*/*"));
        client.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("br", .9));
        client.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("gzip", .7));
        client.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("deflate", .5));

        client.DefaultRequestHeaders.AcceptLanguage.Add(new StringWithQualityHeaderValue("en"));
        client.DefaultRequestHeaders.TryAddWithoutValidation("Sec-Ch-Ua", """"Microsoft Edge";v="137", "Chromium";v="137", "Not/A)Brand";v="24"""");
        client.DefaultRequestHeaders.TryAddWithoutValidation("Sec-Fetch-Mode", "no-cors");
        client.DefaultRequestHeaders.TryAddWithoutValidation("Sec-Fetch-Dest", "document");

        try {
            //var uri = new Uri("https://www.ti.com/lit/gpn/TPS723");
            var uri = new Uri("https://www.st.com/resource/en/datasheet/stlq015.pdf");
            using var response = await client.GetAsync(uri);
            var dataBytes = await response.Content.ReadAsByteArrayAsync();

            if (response.IsSuccessStatusCode) {
                await File.WriteAllBytesAsync("[Some path]", dataBytes);
            }
        }
        catch (Exception) {
            // Log StackTrace etc.
            throw;
        }
        Console.ReadKey();
    }
}