Azure Web App Service "container is unhealthy" error forcing container restart every 15min

Question

I have a dot net core 3.1 app running in an Azure Web App environment (Linux). The app is running fine in local environment/debug. When publishing to Azure, the app runs, but in the Applictions logs, I see the container is restarting every ~15min due to unhealthy container message. The memory and processor load is well below 50% and there is no special load on the application.

I dont know why it shows "unhealthy" in the logs and restarts the container so frequently. Is there anything missing in the configuration settings?

Why does the container restart frequently and where to look for more error specific information to debug this issue?

Any hint is welcome.

Application Logs:

2020-10-06T08:08:57.693Z INFO  - Starting container for site
2020-10-06T08:08:57.693Z INFO  - docker run -d -p 5159:8080 --name wttcycleswebapp_0_8c7e9ba1 -e WEBSITE_SITE_NAME=WTTCyclesWebApp -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=wttcycleswebapp.azurewebsites.net -e WEBSITE_INSTANCE_ID=92c475eedd41771f9c6f8d7f8c52ad11b610d1d082a92269eb132b8b216a0d8d -e HTTP_LOGGING_ENABLED=1 appsvc/dotnetcore:3.1-latest_20200717.2 dotnet Cycles.dll 

2020-10-06T08:08:58.845Z INFO  - Initiating warmup request to container wttcycleswebapp_0_8c7e9ba1 for site wttcycleswebapp
2020-10-06T08:09:04.094Z INFO  - Container wttcycleswebapp_0_8c7e9ba1 for site wttcycleswebapp initialized successfully and is ready to serve requests.
2020-10-06T08:24:34.102Z ERROR - Container for wttcycleswebapp_0_8c7e9ba1 site wttcycleswebapp is unhealthy, Stopping site.
2020-10-06T08:24:34.102Z INFO  - Stopping site wttcycleswebapp because it is not healthy.
2020-10-06T08:27:15.247Z INFO  - Starting container for site
2020-10-06T08:27:15.248Z INFO  - docker run -d -p 6236:8080 --name wttcycleswebapp_0_6391dc4a -e WEBSITE_SITE_NAME=WTTCyclesWebApp -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=wttcycleswebapp.azurewebsites.net -e WEBSITE_INSTANCE_ID=92c475eedd41771f9c6f8d7f8c52ad11b610d1d082a92269eb132b8b216a0d8d -e HTTP_LOGGING_ENABLED=1 appsvc/dotnetcore:3.1-latest_20200717.2 dotnet Cycles.dll 

2020-10-06T08:27:16.384Z INFO  - Initiating warmup request to container wttcycleswebapp_0_6391dc4a for site wttcycleswebapp
2020-10-06T08:27:20.657Z INFO  - Container wttcycleswebapp_0_6391dc4a for site wttcycleswebapp initialized successfully and is ready to serve requests.
2020-10-06T08:36:35.799Z ERROR - Container for wttcycleswebapp_0_6391dc4a site wttcycleswebapp is unhealthy, Stopping site.
2020-10-06T08:36:35.799Z INFO  - Stopping site wttcycleswebapp because it is not healthy.
2020-10-06T08:36:38.668Z INFO  - Starting container for site
2020-10-06T08:36:38.669Z INFO  - docker run -d -p 3648:8080 --name wttcycleswebapp_0_6ad6dbd7 -e WEBSITE_SITE_NAME=WTTCyclesWebApp -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=wttcycleswebapp.azurewebsites.net -e WEBSITE_INSTANCE_ID=92c475eedd41771f9c6f8d7f8c52ad11b610d1d082a92269eb132b8b216a0d8d -e HTTP_LOGGING_ENABLED=1 appsvc/dotnetcore:3.1-latest_20200717.2 dotnet Cycles.dll 

2020-10-06T08:36:39.633Z INFO  - Initiating warmup request to container wttcycleswebapp_0_6ad6dbd7 for site wttcycleswebapp
2020-10-06T08:36:43.884Z INFO  - Container wttcycleswebapp_0_6ad6dbd7 for site wttcycleswebapp initialized successfully and is ready to serve requests.
2020-10-06T08:51:38.900Z ERROR - Container for wttcycleswebapp_0_6ad6dbd7 site wttcycleswebapp is unhealthy, Stopping site.
2020-10-06T08:51:38.901Z INFO  - Stopping site wttcycleswebapp because it is not healthy.
2020-10-06T08:51:39.775Z INFO  - Starting container for site
2020-10-06T08:51:39.776Z INFO  - docker run -d -p 3002:8080 --name wttcycleswebapp_0_f59f6020 -e WEBSITE_SITE_NAME=WTTCyclesWebApp -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=wttcycleswebapp.azurewebsites.net -e WEBSITE_INSTANCE_ID=92c475eedd41771f9c6f8d7f8c52ad11b610d1d082a92269eb132b8b216a0d8d -e HTTP_LOGGING_ENABLED=1 appsvc/dotnetcore:3.1-latest_20200717.2 dotnet Cycles.dll 

2020-10-06T08:51:40.854Z INFO  - Initiating warmup request to container wttcycleswebapp_0_f59f6020 for site wttcycleswebapp
2020-10-06T08:51:45.135Z INFO  - Container wttcycleswebapp_0_f59f6020 for site wttcycleswebapp initialized successfully and is ready to serve requests.
2020-10-06T09:23:50.153Z ERROR - Container for wttcycleswebapp_0_f59f6020 site wttcycleswebapp is unhealthy, Stopping site.
2020-10-06T09:23:50.154Z INFO  - Stopping site wttcycleswebapp because it is not healthy.
2020-10-06T09:23:50.958Z INFO  - Starting container for site
2020-10-06T09:23:50.959Z INFO  - docker run -d -p 3270:8080 --name wttcycleswebapp_0_0c8ce05e -e WEBSITE_SITE_NAME=WTTCyclesWebApp -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=wttcycleswebapp.azurewebsites.net -e WEBSITE_INSTANCE_ID=92c475eedd41771f9c6f8d7f8c52ad11b610d1d082a92269eb132b8b216a0d8d -e HTTP_LOGGING_ENABLED=1 appsvc/dotnetcore:3.1-latest_20200717.2 dotnet Cycles.dll 

2020-10-06T09:24:01.308Z INFO  - Initiating warmup request to container wttcycleswebapp_0_0c8ce05e for site wttcycleswebapp
2020-10-06T09:24:01.600Z INFO  - Container wttcycleswebapp_0_0c8ce05e for site wttcycleswebapp initialized successfully and is ready to serve requests.
2020-10-06T11:00:33.648Z INFO  - Starting container for site
2020-10-06T11:00:33.649Z INFO  - docker run -d -p 4757:8080 --name wttcycleswebapp_2_de12fb53 -e WEBSITE_SITE_NAME=WTTCyclesWebApp -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=wttcycleswebapp.azurewebsites.net -e WEBSITE_INSTANCE_ID=92c475eedd41771f9c6f8d7f8c52ad11b610d1d082a92269eb132b8b216a0d8d -e HTTP_LOGGING_ENABLED=1 appsvc/dotnetcore:3.1-latest_20200717.2 dotnet Cycles.dll 

2020-10-06T11:00:35.792Z INFO  - Initiating warmup request to container wttcycleswebapp_2_de12fb53 for site wttcycleswebapp
2020-10-06T11:00:42.307Z INFO  - Container wttcycleswebapp_2_de12fb53 for site wttcycleswebapp initialized successfully and is ready to serve requests.
2020-10-06T11:16:17.316Z ERROR - Container for wttcycleswebapp_2_de12fb53 site wttcycleswebapp is unhealthy, Stopping site.
2020-10-06T11:16:17.316Z INFO  - Stopping site wttcycleswebapp because it is not healthy.
2020-10-06T11:16:24.769Z INFO  - Starting container for site
2020-10-06T11:16:24.769Z INFO  - docker run -d -p 1185:8080 --name wttcycleswebapp_0_3904d3bf -e WEBSITE_SITE_NAME=WTTCyclesWebApp -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=wttcycleswebapp.azurewebsites.net -e WEBSITE_INSTANCE_ID=92c475eedd41771f9c6f8d7f8c52ad11b610d1d082a92269eb132b8b216a0d8d -e HTTP_LOGGING_ENABLED=1 appsvc/dotnetcore:3.1-latest_20200717.2 dotnet Cycles.dll 

2020-10-06T11:16:25.662Z INFO  - Initiating warmup request to container wttcycleswebapp_0_3904d3bf for site wttcycleswebapp
2020-10-06T11:16:36.943Z INFO  - Container wttcycleswebapp_0_3904d3bf for site wttcycleswebapp initialized successfully and is ready to serve requests.
2020-10-06T11:31:56.959Z ERROR - Container for wttcycleswebapp_0_3904d3bf site wttcycleswebapp is unhealthy, Stopping site.
2020-10-06T11:31:56.959Z INFO  - Stopping site wttcycleswebapp because it is not healthy.
2020-10-06T11:31:58.658Z INFO  - Starting container for site
2020-10-06T11:31:58.659Z INFO  - docker run -d -p 2576:8080 --name wttcycleswebapp_0_590a51f3 -e WEBSITE_SITE_NAME=WTTCyclesWebApp -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=wttcycleswebapp.azurewebsites.net -e WEBSITE_INSTANCE_ID=92c475eedd41771f9c6f8d7f8c52ad11b610d1d082a92269eb132b8b216a0d8d -e HTTP_LOGGING_ENABLED=1 appsvc/dotnetcore:3.1-latest_20200717.2 dotnet Cycles.dll 

2020-10-06T11:31:59.684Z INFO  - Initiating warmup request to container wttcycleswebapp_0_590a51f3 for site wttcycleswebapp
2020-10-06T11:32:03.912Z INFO  - Container wttcycleswebapp_0_590a51f3 for site wttcycleswebapp initialized successfully and is ready to serve requests.

In addition, I can see in the "container crash" log:

Container exited unexpectedly: last 10 seconds logs [
2020-10-06T11:16:14.928728537Z Could not open output /home/logs/dumps/coredump.f7d267b6a38a.28.1601982035: No such file or directory
2020-10-06T11:16:14.931442136Z Writing minidump with heap to file /home/logs/dumps/coredump.f7d267b6a38a.28.1601982035
2020-10-06T11:16:15.118711399Z /opt/startup/startup.sh: line 10:    28 Aborted                 (core dumped) dotnet Cycles.dll
]

Answer 1

It looks like your application is crashing when trying to write logs on the file system. This could be the cause of failure if your logs are written to the disk or rotated every 15 minutes. As a good practise, it's probably better to redirect the logs to standard console output instead of writing them to the file system.

Another potential explanation could be a permission issue on the log location

Answer 2

Agree with totor that it is related to the log files.

That it works locally but not in Azure gives a clue to what may be the root cause.

It may be that you need to enable container logs.

App Service -> Monitoring -> App Service logs : Then turn on file system logging