Menu
Symptoms: ASP.NET Web Forms website on Azure sporadically crashes and all .aspx page requests fail with this error. The problem seems random and only happens once in a great while. The site may run for months with no issues then out of the blue it will stop serving any .aspx pages and gives the error on every .aspx page request. A restart of the website is the only solution (or redeploy, which causes the same thing).
This was a very difficult problem to debug since it was so sporadic and none of the other answers I found helped, they did not address the problem where the site would deploy and run for long periods of time then crash with this error seemingly randomly. In the end I got some help from Microsoft.
594
According to Microsoft this is caused by a known bug with the .Net Framework versions 4.7.x triggered by memory pressure in the web server worker process. It is fixed in .Net framework 4.8 (not yet released at this time). My solution was simply to check "allowed precompiled site to be updatable" and check the "Do not merge" option.
The root cause of the issue is relating to the memory pressure on the worker process (w3wp.exe) address space. In ASP.net, when the .Net Framework detects that we have exceeded a certain memory pressure threshold, it will proceed to try and eject items from the internal caching structures in order to relieve space. Upon such a cache trim, assemblies belonging to your application are removed from the memory cache. This triggers a callback delegate that will try and reflect that changes in memory cache onto the cache of assemblies on disk – and will try and delete the .dll assembly ejected from the in-memory cache. However, the worker process is still keeping a reference to this file (an open handle) and as such, the delete fails. When this occurs, the .Net Framework creates a .delete file next to the assembly to mark it as stale. This will prevent the worker process from loading the file back into memory cache and causes the compilation error you are seeing.
The cause of all of this cache processing is an incorrect insertion into the ASP.net memory cache of dynamic assemblies issued from the pre-compilation process with non-updatable UI. In this process of compilation, the names of the dynamic assemblies cannot be changed, as the .compiled resource files which indicate to the ASP.net Runtime where a compiled resource is located (in what binary) do not change if changes are made to the site – contrary to the pre-compilation with updatable UI, where the markup for pages can be subject to change, and this generates changes to the .compiled files and then to the names of the dynamic assemblies, thus preventing the re-usage of the old names.
For assemblies that have come from the pre-compilation of an ASP.net application with non-updatable UI, the standard was to insert them into the ASP.net memory cache in the worker process with a special attribute that would indicate to the cache manager that these entries cannot be removed. This attribute is missing in the .Net Framework 4.7.x release builds and is the cause of the error. Without it, when the cache is trimmed, the assemblies can be removed, and since they are still in usage, they cannot be deleted from the Temporary ASP.net Files folder (the shadow copy folder). As such, this results in the creation of the .delete files. This was not an issue in builds of the .Net Framework prior to 4.7. The product group has also confirmed that they have resolved this issue in the upcoming .Net Framework 4.8 release.
There are a couple of solutions around this issue:
The caching error is only present in builds of the .Net Framework of 4.7 and above, thereby reverting to a 4.6.x version would allow you to continue working without facing the problem. They can remain in the 4.6.x distribution until the .Net Framework 4.8 is released later this year. I do not have a timeline from the product group for this release as of now.
The cache is trimmed in ASP.net when we see that the memory pressure is too great inside the worker process. This memory pressure is evaluated as follows: if you do not have a recycling condition set in the application’s application pool in IIS, ASP.net will consider the threshold to be 60 % of the total RAM available to the machine. When the private bytes of the worker process exceed this threshold, the internal cache will be trimmed, and the assemblies will be ejected, causing the appearance of the .delete files. I advise you work with the customer to increase the RAM memory available to their most impacted servers if possible, and also set a memory recycling based on private bytes on the app pool hosting the impacted application. We can set this recycling limitation to a higher value than the RAM so that we can ensure the threshold is high enough not to be reached by the worker process.
To set the private bytes recycling on the application pool:
You can try to install the preview version of the .Net Framework 4.8 which is available for download online now. However, this is a preview version and you may run into other issues which will not be supported until the Framework is released. You can also opt to install a private fix from Microsoft (reach out to me if you want this nand I will request it), which will change the malfunctioning assemblies from the .Net Framework 4.7.x to allow correct insertion of assemblies into the cache. However, this private fix is not signed, and will hence require us to disable assembly sign checking on the server which may expose the customer to other security risks.
The bug only affects non-updatable precompiled binaries, so this avoids the bug, but startup times are obviously greatly affected. For me, this solution was simple and works fine until 4.8 rolls out for Azure web services.
147
I get the same problem from time to time, on aleatory pages that where working and suddenly stop.
Like this
But i know for a fact that the page is pre-complied The "WORKAROUND" for me is very strange.
Eric Sassaman explains, in his answer, why this happens. Will try to upgrade to 4.8 and see if it solves the issue.
*I've been having this issue, and applying my workaround, since 4.2, i think. I find it strange that microsoft states that is was introduced with 4.7. Today i gave another shot at finding the root cause of the problem and found this thread.
Thank you Eric for your efforts, for contacting Microsoft and posting their response.
39
In case it was working before, and suddenly you get this error message, try setting the 'Enable 32-Bit Applications' to True in the advanced options of the ApplicationPool.
1
Microsoft's gave some advice here... Microsoft article
...to remove just the files in the "root" folder...
C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET\root
For us, this didn't work (neither did restarting IIS, our server, enabling 32-bit, etc).
We needed to delete all of folders and files under this folder: C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Temporary ASP.NET
Oh, and to check whether you do have .Net Framework 4.7 installed (which is the cause of this issue), you can check the registry here:
My head hurts...
1
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
