Debugging TCP/IP Port Exhaustion in an Azure WebApp

Question

I'm getting a TCP/IP Port exhaustion error on a web application deployed to Azure App Services. The issue is related to a RavenDB connection session eating up the ports. I think I have a fix and should be good but I feel outclassed as I really wasn't able to do much for debugging in the deployed environment.

In debugging the issue I tried to access netstat but I get access is denied message and can't seem to get around it. Any thoughts?

On a tangent: How are ports allocated in an Azure webapp? Options put forward (as outlined in https://msdn.microsoft.com/en-us/library/aa560610(v=bts.20).aspx) there are to reduce socket connection timeout value or increase the upper range of ports dynamically allocated to client TCP/IP socket connections. However, I don't think this is possible in Azure Web Apps.

Answer 1

This could possibly be solved by updating your Ravendb instance! Read more here: https://ayende.com/blog/173057/production-postmorterm-houston-we-have-a-problem.

From changelog, release 3.0.30100:

Fixed UDP port leak in licensing component

also this, from 3.0.30115:

[Major] [Prefetching] Fixed memory leak that could lead into slowness of the server under heavy load impacting indexing and replication mechanisms

And some useful knowledge on Azure services and availability of ports: http://www.freekpaans.nl/2015/08/starving-outgoing-connections-on-windows-azure-web-sites/

In short, the issues goes for async calls where the UDP reply packet for whatever reason is dropped/lost. Before the issue was solved, this would mean an async call that keeps listening for the response until e.g. a crash or iisreset.

We are rolling out the upgrade to try to fix our instance of the same issue within the next 24 hours, but I thought I would reply right away, so you might be able to test too if this is indeed your problem.

I will get back to this post to confirm that the issue will go away - only way to know for sure is to update Ravendb and wait and see for a week or two.