Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Getting HttpRequestExceptions: The response ended prematurely

For some reason, I'm getting a HttpRequestException with the message "The response ended prematurely. I'm creating about 500 tasks that use my RateLimitedHttpClient to make a request to a website so it can scrape it.

The exception is being thrown from the line return await response.Content.ReadAsStringAsync();.

Is it possible that with 500 tasks, each with ~20 pages to be downloaded and parsed (~11000 total), that I'm exceeding the capability of .Net's HttpClient?

public class SECScraper
{
    public event EventHandler<ProgressChangedEventArgs> ProgressChangedEvent;

    public SECScraper(EPSDownloader downloader, FinanceContext financeContext)
    {
        _downloader = downloader;
        _financeContext = financeContext;
    }

    public void Download()
    {
        _numDownloaded = 0;

        var companies = _financeContext.Companies.OrderBy(c => c.Name);
        _interval = companies.Count() / 100;

        var tasks = companies.Select(c => ScrapeSEC(c.CIK) ).ToList();
        Task.WhenAll(tasks);
    }
}

public class RateLimitedHttpClient : IHttpClient
{
    public RateLimitedHttpClient(System.Net.Http.HttpClient client)
    {
        _client = client;
        _client.Timeout = TimeSpan.FromMinutes(30);
        ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls;
    }
    public async Task<string> ReadAsync(string url)
    {
        if (!_sw.IsRunning)
            _sw.Start();

        await Delay();

        using var response = await _client.GetAsync(url);

        return await response.Content.ReadAsStringAsync();
    }

    private async Task Delay()
    {
        var totalElapsed = GetTimeElapsedSinceLastRequest();

        while (totalElapsed < MinTimeBetweenRequests)
        {
            await Task.Delay(MinTimeBetweenRequests - totalElapsed);
            totalElapsed = GetTimeElapsedSinceLastRequest();
        };

        _timeElapsedOfLastHttpRequest = (int)_sw.Elapsed.TotalMilliseconds;
    }

    private int GetTimeElapsedSinceLastRequest()
    {
        return (int)_sw.Elapsed.TotalMilliseconds - _timeElapsedOfLastHttpRequest;
    }

    private readonly System.Net.Http.HttpClient _client;
    private readonly Stopwatch _sw = new Stopwatch();
    private int _timeElapsedOfLastHttpRequest;
    private const int MinTimeBetweenRequests = 100;
}

It appears that I am getting a few HttpRequestExceptions here.

System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: The response ended prematurely.
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
   at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
   at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
   at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.. ---> System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine.
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
   at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
   at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
   at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
   at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.. ---> System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine.
   --- End of inner exception stack trace ---
   at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteAsync|210_1[TWriteAdapter](ValueTask writeTask, Byte[] bufferToReturn)
   at System.Net.Security.SslStream.WriteAsyncInternal[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
   at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
   at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
   at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: The response ended prematurely.
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
   at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
   at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
   at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: The response ended prematurely.
>    at System.Net.Http.HttpConnection.FillAsync()
>    at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
>    at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
>    --- End of inner exception stack trace ---
>    at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
>    at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
>    at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
>    at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
>    at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
>    at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
>    at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
>    at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
>    at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.. ---> System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine.
>    --- End of inner exception stack trace ---
>    at System.Net.Security.SslStream.<FillBufferAsync>g__InternalFillBufferAsync|215_0[TReadAdapter](TReadAdapter adap, ValueTask`1 task, Int32 min, Int32 initial)
>    at System.Net.Security.SslStream.ReadAsyncInternal[TReadAdapter](TReadAdapter adapter, Memory`1 buffer)
>    at System.Net.Http.HttpConnection.FillAsync()
>    at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
>    at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
>    --- End of inner exception stack trace ---
>    at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
>    at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
>    at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
>    at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
>    at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
>    at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
>    at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
>    at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
>    at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: Unable to read data from the transport connection: An established connection was aborted by the software in your host machine.. ---> System.Net.Sockets.SocketException (10053): An established connection was aborted by the software in your host machine.
>    --- End of inner exception stack trace ---
>    at System.Net.Security.SslStream.<WriteSingleChunk>g__CompleteAsync|210_1[TWriteAdapter](ValueTask writeTask, Byte[] bufferToReturn)
>    at System.Net.Security.SslStream.WriteAsyncInternal[TWriteAdapter](TWriteAdapter writeAdapter, ReadOnlyMemory`1 buffer)
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessAuthentication(LazyAsyncResult lazyResult, CancellationToken cancellationToken)
   at System.Net.Security.SslStream.BeginAuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken, AsyncCallback asyncCallback, Object asyncState)
   at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__65_0(SslClientAuthenticationOptions arg1, CancellationToken arg2, AsyncCallback callback, Object state)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl[TArg1,TArg2](Func`5 beginMethod, Func`2 endFunction, Action`1 endAction, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
   at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func`5 beginMethod, Action`1 endMethod, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
   at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func`5 beginMethod, Action`1 endMethod, TArg1 arg1, TArg2 arg2, Object state)
   at System.Net.Security.SslStream.AuthenticateAsClientAsync(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
   at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
   at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
   at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
>    at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessAuthentication(LazyAsyncResult lazyResult, CancellationToken cancellationToken)
   at System.Net.Security.SslStream.BeginAuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken, AsyncCallback asyncCallback, Object asyncState)
   at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__65_0(SslClientAuthenticationOptions arg1, CancellationToken arg2, AsyncCallback callback, Object state)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl[TArg1,TArg2](Func`5 beginMethod, Func`2 endFunction, Action`1 endAction, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
   at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func`5 beginMethod, Action`1 endMethod, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
   at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func`5 beginMethod, Action`1 endMethod, TArg1 arg1, TArg2 arg2, Object state)
   at System.Net.Security.SslStream.AuthenticateAsClientAsync(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
   at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
   at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
   at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
>    --- End of inner exception stack trace ---
System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception. ---> System.IO.IOException: Authentication failed because the remote party has closed the transport stream.
   at System.Net.Security.SslStream.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslStream.ProcessAuthentication(LazyAsyncResult lazyResult, CancellationToken cancellationToken)
   at System.Net.Security.SslStream.BeginAuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken, AsyncCallback asyncCallback, Object asyncState)
   at System.Net.Security.SslStream.<>c.<AuthenticateAsClientAsync>b__65_0(SslClientAuthenticationOptions arg1, CancellationToken arg2, AsyncCallback callback, Object state)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncImpl[TArg1,TArg2](Func`5 beginMethod, Func`2 endFunction, Action`1 endAction, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
   at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func`5 beginMethod, Action`1 endMethod, TArg1 arg1, TArg2 arg2, Object state, TaskCreationOptions creationOptions)
   at System.Threading.Tasks.TaskFactory.FromAsync[TArg1,TArg2](Func`5 beginMethod, Action`1 endMethod, TArg1 arg1, TArg2 arg2, Object state)
   at System.Net.Security.SslStream.AuthenticateAsClientAsync(SslClientAuthenticationOptions sslClientAuthenticationOptions, CancellationToken cancellationToken)
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at POLib.Http.RateLimitedHttpClient.ReadAsync(String url) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\Http\RateLimitedHttpClient.cs:line 23
   at POLib.SECScraper.EPS.EPSDownloader.GetReportLinks(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 65
   at POLib.SECScraper.EPS.EPSDownloader.GetEPSData(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\EPS\EPSDownloader.cs:line 19
   at POLib.SECScraper.SECScraper.ScrapeSEC(Int32 cik) in C:\Users\Joshua\source\repos\PortfolioOptimizer\POLib\SECScraper\SECScraper.cs:line 40
like image 559
keelerjr12 Avatar asked Jul 09 '19 21:07

keelerjr12


2 Answers

You just need to keep digging. The exception "The response ended prematurely" isn't the root cause. Keep digging into the inner exceptions until you find the last one. You'll find this:

System.IO.IOException: Authentication failed because the remote party has closed the transport stream.

So it's not about your code. It seems the server you're hitting either can't handle the load, or is intentionally dropping your requests because you're hitting it too hard.

like image 141
Gabriel Luci Avatar answered Sep 22 '22 07:09

Gabriel Luci


I added the host and agent to the header, the problem was fixed.

var request = (HttpWebRequest)WebRequest.Create(url);
request.Host = request.RequestUri.Host; //"example.com"
request.UserAgent = "Dotnet";
like image 23
Ugur BAS Avatar answered Sep 19 '22 07:09

Ugur BAS