Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Azure and local number of TCP connections are very different

I'm now trying to read data from a lot of Azure blobs in parallel using Azure Function and fail to do so, because my service plan does not allow more than ~4000 TCP connections (which I get an error in the portal about), however when I try to run it locally all of the following:

  • netstat with all possible flags
  • Wireshark
  • TCPView
  • network inspector in Windows task manager

just show a couple dozens of items. Is there a tool or maybe code snippet which will allow me to emulate locally the situation that I have once my app is deployed?

Even better would be knowing if it is possible to somehow limit the number of TCP connections that my Azure Function is trying to open (using .NET Azure SDK, or Azure portal, or some settings.json file or whatever)

Edit1: I've rewritten the whole thing to be sequential and also I've split blob reads into chunks of 100 items, this seemed to somewhat help the number of TCP connections (it's about 500 at peak now, so still a lot, but at least fitting the app service plan, the app, of course, became slow as hell as a result), but it still tries to allocate ~4000 of "Socket handles" and fails, still can't find a way to see locally the same amount of socket handles allocated - Handles column is Details tab of windows task manager shows roughly the same amount of handles during the whole process execution

like image 916
Oleg Golovkov Avatar asked May 10 '18 15:05

Oleg Golovkov


1 Answers

To answer the question itself: I wasn't able to find a way to see locally the TCP-related metrics that I get when actually running my functions in Azure. For now it feels like some important development tools and/or docs are missing. The "serverless" experience turned out be the deepest dive into Windows system programming I ever had as a .NET developer.

The solution for the problem itself was the following: I've rewritten the whole thing to be sequential and managed it to get establishing about a hundred simultaneous connections. Then I just used binary search playing with MaxDegreeOfParallelism until I found a value suitable for my plan.

like image 117
Oleg Golovkov Avatar answered Nov 04 '22 10:11

Oleg Golovkov