Intermittent "Failed to load viewstate" error

Question

Yes I know this problem has been reported many times, but my case doesn't seem to fit any proposed solutions (unless I miss something).

The ASP.NET page setup (simplified) is: user clicks a link (technically a node of Infragistics WebDataTree) and this action binds a grid (Infragistics WebHierarchicalDataGrid) located in an UpdatePanel.

Now this works fine in a local test system. When deployed to a production server - it also works - most of the times, but sometimes this throws error:

Failed to load viewstate. The control tree into which viewstate is being loaded must match the control tree that was used to save viewstate during the previous request

This seems to happen more if the production system under heavier load (more users accessing it). Again, I am not building any dynamic controls, just a button and a grid that is populated on button click.

What could be causing this? Why this doesn't happen always, but sometimes? Any idea how to fix it?

UPDATE

Here's confirmed scenario

1. If one user accesses and uses the page - error doesn't happen
2. If two user doing the same (can even be 2 browser sessions from the same client) - error happens.

How could one session affect viewstate of another?

UPDATE 2

Application is deployed to a single Windows 2008/IIS7 server (no web farms/web gardens, no load balancers). No updates rolls out during runtime.

Application works fine if a single users accesses it, but when multiple users hit the server - eventually some of them are getting ViewState error (while others still work fine). They're all doing the same thing - clicking tree nodes that trigger grid rebind inside of UpdatePanel. They use different browser - IE(9-11), FF, Chrome - error could happen to a random user/browser

Oh and we are not encrypting ViewState either.

UPDATE 3

Stack Trace:

at System.Web.UI.Control.LoadViewStateRecursive(Object savedState) at System.Web.UI.Control.LoadChildViewStateByIndex(ArrayList childState) at System.Web.UI.Control.LoadViewStateRecursive(Object savedState) at System.Web.UI.Control.LoadChildViewStateByIndex(ArrayList childState) at System.Web.UI.Control.LoadViewStateRecursive(Object savedState at System.Web.UI.Control.LoadChildViewStateByID(ArrayList childState) at System.Web.UI.Control.LoadViewStateRecursive(Object savedState) at System.Web.UI.Control.LoadChildViewStateByIndex(ArrayList childState) at System.Web.UI.Control.LoadViewStateRecursive(Object savedState) at System.Web.UI.Control.LoadChildViewStateByIndex(ArrayList childState) at System.Web.UI.Control.LoadViewStateRecursive(Object savedState) at System.Web.UI.Control.LoadChildViewStateByIndex(ArrayList childState) at System.Web.UI.Control.LoadViewStateRecursive(Object savedState) at System.Web.UI.Control.LoadChildViewStateByIndex(ArrayList childState) at System.Web.UI.Control.LoadViewStateRecursive(Object savedState) at System.Web.UI.Control.LoadChildViewStateByIndex(ArrayList childState) at System.Web.UI.Control.LoadViewStateRecursive(Object savedState) at System.Web.UI.Page.LoadAllState() at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

UPDATE 4

It seems if I reduce size of the data returned with each click (e.g. 20 rows instead of 100) the issue either disappears or appears a lot less often.

I've tried manipulating ViewState - e.g. splitting it into chunks, using session as a ViewState storage to reduce actual posted page size - nothing has effect.

Answer 1

After following this advice on how to debug things outside of my code and gaining access to debug symbols for the Infragistics WebHierarchicalDataGrid DLL as well as its source code - turned out exception is thrown by the control - and its not related to the viewstate. Internally in a 1-element array code tries to access second element, throwing "IndexOutOfBound" exception which undergoes magical transformation to "Failed to load viewstate" thrown to the user.

Infragistics is looking into the error and ways to fix it, but since we're really pressed for time we used following workaround: If "view state" error is detected at the end of async postback - we simple reload the page:

Sys.Application.add_load(appLoaded);

function appLoaded(sender, eventArgs) {
   Sys.WebForms.PageRequestManager.getInstance().add_endRequest(EndRequest)
}

function EndRequest(sender, args) {
    if (args.get_error() != undefined) {
        if (args.get_error().message.indexOf('Failed to load viewstate') != -1) {
            setTimeout(function () { location.reload(true) }, 100)
        }
    }
} 

Yes I do realize this is as elegant as hammering a nail with a screwdriver, but since the error is sporadic and refresh gets rid of it - this will have to do.

Answer 2

How many servers do you have in production? If there is more than one are you setting the machineKey in your web.config so that all servers will be using the same key to sign/validate the ViewState value?

As Michael Liu pointed out in a comment, it's unlikely you would get the control tree error for the machineKey problem. I would still like to know if you have multiple servers in production though because it could still be possible that you are running into an issue where ServerA served up newer version of the code and then the POST back occurred to ServerB which doesn't yet have the new code, so the control trees wouldn't match. This could technically happen even with a single server if you rolled out changes while there are active users on your site.