Local ASP.NET MVC Suddenly Very Slow; Load times > 1 minute

Question

Over the last few weeks I've been subject to a sudden and significant performance deterioration when browsing locally hosted ASP.NET 3.5 MVC web applications (C#). Load times for a given page are on average 20 seconds (regardless of content); start up is usually over a minute. These applications run fast on production and even test systems (Test system is comparable to my development environment).

I am running IIS 6.0, VS2008, Vista Ultimate, SQL2005, .NET 3.5, MVC 1.0, and we use VisualSVN 1.7.

My SQL DB is local and IPv6 does not seem to be the cause. I browse in Firefox and IE8 outside of Debug mode using loopback, machine name, and 'localhost' and get the exact same results every time (hence DNS doesn't seem to be the issue either).

Below are screen shots of my dotTrace output.

http://www.glowfoto.com/static_image/28-100108L/3123/jpg/06/2010/img4/glowfoto

This issue has made it near impossible to debug/test any web app. Any suggestions very much appreciated!

SOLUTION: Complete re-installation of Windows, IIS, Visual Studio, etc. It wasn't the preferred solution, but it worked.

Answer 1

Surely the big red flag on that profiler output is the fact that AddDirectory is called 408 times and AddExistingFile is called 66,914 times?

Can you just confirm that there's not just a shed load of directories and files underneath your MVC app's root folder? Because it looks like the framework is busying itself trying to work out what files it needs to build (or add watches to) on startup.

[I am not au fait with MVC and so maybe this is not what is happening but 67k calls to a function with a name like "AddExistingFile" does smell wrong].

Answer 2

I've learnt that it's usually a "smell" when things fail near a power of two ...

Given

Over the last few weeks I've been subject to a sudden and significant performance deterioration

and

AddExistingFile is called 66,914 times

I'm wondering if the poor performance hit at about the time as the number of files exceeded 65,535 ...

Other possibilities to consider ...

• Are all 66,914 files in the same directory? If so, that's a lot of directory blocks to access ... try a hard drive defrag. In fact, it's even more directory blocks if they're distributed across a bunch of directories.

• Are you storing all the files in the same list? Are you preseting the capacity of that list, or allowing it to "grow" naturally and slowly?

• Are you scanning for files depth first or breadth first? Caching by the OS will favor the performance of depth first.

Update 14/7

Clarification of Are you storing all the files in the same list?

Naive code like this first example doesn't perform ideally well because it needs to reallocate storage space as the list grows.

var myList = new List<int>();
for (int i=0; i<10000; i++)
{
    myList.Add(i);
}

It's more efficient, if you know it, to initialize the list with a specific capacity to avoid the reallocation overhead:

var myList = new List<int>(10000);  // Capacity is 10000
for (int i=0; i<10000; i++)
{
    myList.Add(i);
}

Update 15/7

Comment by OP:

These web apps are not programmatically probing files on my hard disk, at least not by my hand. If there is any recursive file scanning, its by VS 2008.

It's not Visual Studio that's doing the file scanning - it is your web application. This can clearly be seen in the first profiler trace you posted - the call to System.Web.Hosting.HostingEnvironment.Initialize() is taking 49 seconds, largely because of 66,914 calls to AddExistingFile(). In particular, the read of the property CreationTimeUTC is taking almost all the time.

This scanning won't be random - it's either the result of your configuration of the application, or the files are in your web applications file tree. Find those files and you'll know the reason for your performance problems.