IIS 7 Error: Cannot execute a program. The command being executed was csc.exe (ApplicationPoolIdentity default settings)

Question

This is my first question in SO, please apologize me if I have some grammatical mistakes.

Problem Description:

I have 32-bit MVC 5-based CRUD systems running on IIS 7 (Windows Server 2008) & SQL Server 2008 R2 installed into several servers, providing company intranet management service since August 2015. For months, all of them running smoothly without any problems. However, yesterday I found that I cannot access site root page on a 172.16.1.101 server with this message:

> Server Error in '/' Application.

> [Win32Exception (0x80004005): Access is denied]

>[ExternalException (0x80004005): Cannot execute a program. The command being executed was "C:\Windows\Microsoft.NET\Framework\v4.0.30319\csc.exe" /noconfig /fullpaths @"C:\Windows\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files\root\-(random 8-digit hex)-\-(random 8-digit hex)-\-(random 8-char base64).cmdline".]
System.CodeDom.Compiler.Executor.ExecWaitWithCaptureUnimpersonated(SafeUserTokenHandle userToken, String cmd, String currentDir, TempFileCollection tempFiles, String& outputName, String& errorName, String trueCmdLine) +3365
System.CodeDom.Compiler.Executor.ExecWaitWithCapture(SafeUserTokenHandle userToken, String cmd, String currentDir, TempFileCollection tempFiles, String& outputName, String& errorName, String trueCmdLine) +8591876
Microsoft.CSharp.CSharpCodeGenerator.Compile(CompilerParameters options, String compilerDirectory, String compilerExe, String arguments, String& outputFile, Int32& nativeReturnValue, String trueArgs) +331
Microsoft.CSharp.CSharpCodeGenerator.FromFileBatch(CompilerParameters options, String[] fileNames) +784
Microsoft.CSharp.CSharpCodeGenerator.System.CodeDom.Compiler.ICodeCompiler.CompileAssemblyFromFileBatch(CompilerParameters options, String[] fileNames) +170
System.Web.Compilation.AssemblyBuilder.Compile() +12440424
System.Web.Compilation.BuildProvidersCompiler.PerformBuild() +303
System.Web.Compilation.ApplicationBuildProvider.GetGlobalAsaxBuildResult(Boolean isPrecompiledApp) +12978145
System.Web.Compilation.BuildManager.CompileGlobalAsax() +50
System.Web.Compilation.BuildManager.EnsureTopLevelFilesCompiled() +665

>[HttpException (0x80004005): Cannot execute a program. The command being executed was "C:\Windows\Microsoft.NET\Framework\v4.0.30319\csc.exe" /noconfig /fullpaths @"C:\Windows\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files\root\-(random 8-digit hex)-\-(random 8-digit hex)-\-(random 8-char base64)-.cmdline".]
System.Web.Compilation.BuildManager.ReportTopLevelCompilationException() +80
System.Web.Compilation.BuildManager.EnsureTopLevelFilesCompiled() +1142
System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException) +1193

>[HttpException (0x80004005): Cannot execute a program. The command being executed was "C:\Windows\Microsoft.NET\Framework\v4.0.30319\csc.exe" /noconfig /fullpaths @"C:\Windows\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files\root\-(random 8-digit hex)-\-(random 8-digit hex)-\-(random 8-char base64)-.cmdline".]
System.Web.HttpRuntime.FirstRequestInit(HttpContext context) +12980692
System.Web.HttpRuntime.EnsureFirstRequestInit(HttpContext context) +159
System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context) +12820285

Here is an event log when the exception occured:

>IIS 7 Event Log

>Event code: 3005 

>Event message: An unhandled exception has occurred. 

>Event time: 5/24/2016 1:42:08 PM 

>Event time (UTC): 5/24/2016 6:42:08 AM 

>Event ID: 062751a2dc724be9af5de99014e58a47 

>Event sequence: 2 

>Event occurrence: 1 

>Event detail code: 0 

>Application information: 

>Application domain: /LM/W3SVC/3/ROOT-1-131085457263004464 

>Trust level: Full 

>Application Virtual Path: / 

>Application Path: E:\Test\ 

>Machine name: -- edited --

>Process information: 

>Process ID: 1172

>Process name: w3wp.exe 

>Account name: IIS APPPOOL\ASP.NET v4.0 

>Exception information:

>Exception type: HttpException

>Exception message: Cannot execute a program. The command being executed was "C:\Windows\Microsoft.NET\Framework\v4.0.30319\csc.exe" /noconfig /fullpaths @"C:\Windows\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files\root\-(random 8-digit hex)-\-(random 8-digit hex)-\-(random 8-char base64)-.cmdline".
   at System.Web.Compilation.BuildManager.ReportTopLevelCompilationException()
   at System.Web.Compilation.BuildManager.EnsureTopLevelFilesCompiled()
   at System.Web.Compilation.BuildManager.CallAppInitializeMethod()
   at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException)

>Cannot execute a program. The command being executed was "C:\Windows\Microsoft.NET\Framework\v4.0.30319\csc.exe" /noconfig /fullpaths @"C:\Windows\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files\root\-(random 8-digit hex)-\-(random 8-digit hex)-\-(random 8-char base64).cmdline".
   at System.CodeDom.Compiler.Executor.ExecWaitWithCaptureUnimpersonated(SafeUserTokenHandle userToken, String cmd, String currentDir, TempFileCollection tempFiles, String& outputName, String& errorName, String trueCmdLine)
   at System.CodeDom.Compiler.Executor.ExecWaitWithCapture(SafeUserTokenHandle userToken, String cmd, String currentDir, TempFileCollection tempFiles, String& outputName, String& errorName, String trueCmdLine)
   at Microsoft.CSharp.CSharpCodeGenerator.Compile(CompilerParameters options, String compilerDirectory, String compilerExe, String arguments, String& outputFile, Int32& nativeReturnValue, String trueArgs)
   at Microsoft.CSharp.CSharpCodeGenerator.FromFileBatch(CompilerParameters options, String[] fileNames)
   at Microsoft.CSharp.CSharpCodeGenerator.System.CodeDom.Compiler.ICodeCompiler.CompileAssemblyFromFileBatch(CompilerParameters options, String[] fileNames)
   at System.CodeDom.Compiler.CodeDomProvider.CompileAssemblyFromFile(CompilerParameters options, String[] fileNames)
   at System.Web.Compilation.AssemblyBuilder.Compile()
   at System.Web.Compilation.BuildProvidersCompiler.<PerformBuild>b__1(AssemblyBuilder assemblyBuilder)
   at System.Threading.Tasks.Parallel.<>c__DisplayClass32`2.<PartitionerForEachWorker>b__30()
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
   at System.Threading.Tasks.Task.<>c__DisplayClass11.<ExecuteSelfReplicating>b__10(Object param0)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Web.Compilation.BuildProvidersCompiler.PerformBuild()
   at System.Web.Compilation.CodeDirectoryCompiler.GetCodeDirectoryAssembly(VirtualPath virtualDir, CodeDirectoryType dirType, String assemblyName, StringSet excludedSubdirectories, Boolean isDirectoryAllowed)
   at System.Web.Compilation.BuildManager.CompileCodeDirectory(VirtualPath virtualDir, CodeDirectoryType dirType, String assemblyName, StringSet excludedSubdirectories)
   at System.Web.Compilation.BuildManager.CompileResourcesDirectory()
   at System.Web.Compilation.BuildManager.EnsureTopLevelFilesCompiled()

>Access is denied

>Request information:

>Request URL: //172.16.1.101/

>Request path: /

>User host address: //172.16.1.101

>User:

>Is authenticated: False

>Authentication Type:

>Thread account name: IIS APPPOOL\ASP.NET v4.0

>Thread information:

>Thread ID: 5

>Thread account name: IIS APPPOOL\ASP.NET v4.0

>Is impersonating: True

>Stack trace:    at >System.Web.Compilation.BuildManager.ReportTopLevelCompilationException()
   at System.Web.Compilation.BuildManager.EnsureTopLevelFilesCompiled()
   at System.Web.Compilation.BuildManager.CallAppInitializeMethod()
   at System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException)


>Custom event details: 

Sometimes the message shows \Temporary ASP.NET Files\tls directory, rather than \Temporary ASP.NET Files\root directory.

The IIS 7 server configured in its default state and single pool without any running web services by default (only basic configurations), including XML (de-)serializer.

Primarily thought I have permission error on IIS system account, I had done setting permission to "Full Control" without success.

Thus, expected missing file compilation on server-side according to the message, trying to compile the cmdline files manually on CMD by this command

csc.exe /noconfig /fullpaths @"C:\Windows\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files\root-(random 8-digit hex)--(random 8-digit hex)--(random 8-char base64).cmdline"

provides error:

fatal error CS2021: File name '@C:\Windows\Microsoft.NET\Framework\v4.0.30319\Temporary ASP.NET Files\root-(random 8-digit hex)--(random 8-digit hex)--(random 8-char base64).cmdline' is too long or invalid

error CS2011: Error opening response file 'c:\Windows\Microsoft.NET\Framework\v4.0.30319\root-(random 8-digit hex)--(random 8-digit hex)--(random 8-char base64).cmdline' -- 'The system cannot find the file specified. '

Won't giving up, I conducted a google search about this problem and found related similar problems on SO:

Problems publishing a website on smarterasp.net with csc.exe file included?

https://stackoverflow.com/questions/37146958/an-unhandled-exception-was-generated-during-the-execution-of-the-current-web-re

However, I have neither installed "Microsoft.CodeDom.Providers.DotNetCompilerPlatform" package nor using SOAP when compiling the project for live server implementations, and already have 'identity impersonate="true"' attribute on web.config with no avail.

Next, I configured IIS application pool corresponding to the site with "Advanced Settings" option, changing Identity section from "ApplicationPoolIdentity" to "LocalSystem" and interestingly YSOD on accessing root page went away. I tried to change it back on "ApplicationPoolIdentity" and that YSOD turned back.

Even the problem has temporarily solved by LocalSystem identity, I still curious why the error occurred without any signs of system malfunction. 2 questions I want to ask here:

1. Can anyone explain why ApplicationPoolIdentity option didn't work on this case?

2. Which steps I can take to mitigate this issue for other servers, even with their default settings and ApplicationPoolIdentity is set?

tl;dr: ApplicationPoolIdentity with default configuration on IIS 7 servers worked until yesterday I had encountered YSOD described above on a server with neither XML serializers nor web services applied, please help me either explain or describe what problem already happened on the server in detail and ways to mitigate it.

PS: The "-" and "--" sign should be replaced with "\" on written directory path.

Answer 1

uninstall DotNetCompilerPlatform from nuget packages

Answer 2

In my case adding permissions to the local account "IIS_IUSRS" to read & execute on the directory the web app is running under fixed the problem.

Answer 3

On my webhosting, it helped me to delete this section from web.config:

  <system.codedom>
    <compilers>
      <compiler language="c#;cs;csharp" extension=".cs"
        type="Microsoft.CodeDom.Providers.DotNetCompilerPlatform.CSharpCodeProvider, Microsoft.CodeDom.Providers.DotNetCompilerPlatform, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
        warningLevel="4" compilerOptions="/langversion:default /nowarn:1659;1699;1701"/>
      <compiler language="vb;vbs;visualbasic;vbscript" extension=".vb"
        type="Microsoft.CodeDom.Providers.DotNetCompilerPlatform.VBCodeProvider, Microsoft.CodeDom.Providers.DotNetCompilerPlatform, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"
        warningLevel="4" compilerOptions="/langversion:default /nowarn:41008 /define:_MYTYPE=\&quot;Web\&quot; /optionInfer+"/>
    </compilers>
  </system.codedom>