When scaling asp.net-mvc website, what are the top things to focus on in terms of priority?

Question

I asked this question on server fault around web site scalability but this was more focused around hardware configuration, increasing memory, etc. So i thought I would ask it here as well as there is the more development side of my question.

As per the question i have a perfect good working asp.net-mvc site with SQL server backend and using nhibernate and syscache 2nd level cache and i have a request to increase the user base from about 1000 to 7000 and i am trying to figure out where I should focus my development energies in terms of things that work perfectly fine now but are going to cause issues at scale. I have been doing a lot of reading and so far the things that seems to be of interest from a coding point of view are;

1. Asyncronous Controllers
2. Output caching (not really relevent for me as most of my pages are dynamic and user entitlement specific
3. Other?

My SQL server today is 4GB and in terms of data, I would expect a few of the table to grow in size linearly (such as a person table that would grow from 1000 rows to 7000 rows) with this increase of users but most of the other tables (reference data, etc) should only have marginal growth (a table like location maybe would double)

Answer 1

The architecture you describe is not scalable. But based on the numbers you provided, maybe scalability is not a necessity for you at all? Be pragmatic before you design for scalability. Don't get involved into it unless you need it.

Anyway, if you want go for it, you need to scale as follows.

First, distinguish between commands and queries. Commands modify data, queries retrieve data.

For commands, you can use a message broker (e.g. Rabbit MQ) or a service bus (e.g. NServiceBus). Idea is that web server can quickly place a command on the queue, and return the response to the user. Scalability is achieved by scaling out the number of command handlers without touching the web server. Obviously, if you want to inform the user, you need to use some technology such as SignalR.

For queries, you need to understand that they are not so good at scaling as commands are. So, you need to be creative with these.

1. Cache data. This comes with considerations for stale data, as well as data refresh strategies.
2. Distribute data (sharding), when you have many database servers, each holding only subset of the data. Web server issues parallel requests to servers, each server queries the subset (which is small and so quick) and returns back, then web server puts together all the data chunks for the user. Most of the NoSQL databases of today come with sharding and distributed queries built-in.
3. Redesign user flow to treat query as command. e.g. take user's request and let the user continue browsing. Collect all the required data in the background (typically using the same technique as for commands), and inform the user when done. When user goes to the results page, retrieve the pre-generated data (typically from local database which is faster than retrieval).
4. Async and await keywords in C#. Not only for controllers, but also for other methods that your controllers call, as long as they/you would block the thread while waiting for completion. Usage of async and await will free up the threads of requests so that other parallel requests can use those threads meanwhile. Remember that this does not shorten the actual request of the user, it only frees up the web server's resources for optimal use.

Answer 2

Backups

You really need to have a solid system of backing up data. With the chance of critical error increasing with the amount of users a good backup system is very important. Also you may go offline and loose data.

Data storing for offline

Data needs a home. Creating a robust system of storing data is very important. This is for when you go offline for whatever reason.

Structural testing

Stress testing will help you know what to fix. Fill up the database with 10,000 random items and test all the functions you can. Try searching for specific id numbers.

Make sure the bandwidth for the server will be able to handle it. Increasing the userbase will increasingly load down the server. The more users the more on at one time.

As the amount of data being requested increasing so do the deadlock chances increase. You may want to read this article

Answer 3

I would approach this problem in multiple steps to achieve best results.

Application Level

Generally in my projects some of the biggest performance pitfalls come from multiple DB queries per page load. Try loading pages and viewing the database query logs. If there are extra queries, try to consolidate the requests to lighten the load on the DB.

Also, ensure all your stylesheets and javascript assets are compiled into minified single files in your production environment. This will reduce load on your web server.

Backend Level

View your database logs and see which transactions are causing the most latency or triggering full table scans. Add good indexes to these problematic areas and watch your application performance take off.

Testing

In a test environment (!!!) use a database entry faker tool such as Faker (It's Ruby but you get the idea). Test out common transactions with a much larger table size than usual and the performance bottlenecks will begin to show themselves more prominently.