Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Azure website warmup on scale out

I am trying to make sure that my website is fully warmed up when I deploy or scale out. To do this I have taken advantage of the App Initialization config as discussed here: http://ruslany.net/2015/09/how-to-warm-up-azure-web-app-during-deployment-slots-swap/

I am trying to get it to work on scale out as it says it is supported here: https://feedback.azure.com/forums/169385-websites/suggestions/6972595-application-initialization-to-warm-up-specific-pag

My config itself is set as follows:

<applicationInitialization>
  <add initializationPage="api/status"/>
</applicationInitialization>

I have not specified a hostname on the Initialization element as all discussions seem to say it is not required. I also would be hosting this across multiple environments, and would not know the host names for the scaled out versions and so it would be difficult to have all the config transforms required.

When I deploy the web app the warm up works fine, I can see it be deployed into the staging slot, then go to my Status Page. However when I scale out I cannot see that it is warming up. In fact I can see that api calls made to this api, sometimes come back with a 503 response, which is coming from the newly scaled app.

How can I prevent the scaled out application from receiving requests until it has warmed up? How can I get it to actually do the warm up as discussed here: https://feedback.azure.com/forums/169385-websites/suggestions/6972595-application-initialization-to-warm-up-specific-pag

Thanks

like image 858
ObiEff Avatar asked Jun 20 '17 11:06

ObiEff


People also ask

How does Azure scale out?

You scale up by changing the pricing tier of the App Service plan that your app belongs to. Scale out: Increase the number of VM instances that run your app. You can scale out to as many as 30 instances, depending on your pricing tier.

Does scale out in Azure increases compute and memory?

Scale-up – Upgrade the capacity of the host where the app is hosted (PAAS environment). Ex: Increase the RAM size from 1 cores to 4 cores. Scale-out – Upgrade the capacity of the app by increasing the number of host instances (PAAS Environment).


1 Answers

We had a similar issue scaling-out app services. When we add new instances, request received 502 Bad Gateway for 3 minutes.

I will explain the steps we follow with a support engineer and we could find our problem.

  1. Go to KUDU https://{appservicename}.scm.azurewebsites.net/support
  2. On tab Mitigate -> Slow Request
  3. Add a rule to get a memory dump when request take longer. We add a rule for:

    Number Of Requests : 50

    Time Taken (Seconds) : 10

    Interval (Seconds) : 60

You should change this value for your current workload. Also you can add another rule for 5xx errors...

Configuration for memory dump

  1. On tab "action". Add a "custom action". Change CollectKillAnalyze for CollectLogs. Help

Action

  1. Reproduce the issue and download the .dump file. You can find the link to these files from the portal on "Diagnose and solve problems". Or from Kudu in "D:\home\data\DaaS\Logs" enter image description here
  2. Analyze this file with DebugDiag or WinDebug
  3. We did not initialize Redis connection on warm up. So basically RedisClient.get has a sync block in the code which will only allow one thread to access a time. Since the volumes is very high, so hundreds of worker threads are waiting on the sync block. – it needs to be passed from the owner to the next then the next one by one.

Stack trace

After change Redis connection to the warm up. We did not have more issue when scaling out our service.

I hope this can help you.

like image 103
Serafín Avatar answered Sep 24 '22 00:09

Serafín