Possible Solutions to Poor Serialization Performance

Question

I recently did some performance testing and analysis of an ASP.NET application using out-of-process session state - this is necessary when using session state on a web farm so that state can be retrieved on any of the web servers, e.g. if subsequent HTTP requests arrive at a different server because the sessions aren't 'sticky' or the original server is down, etc.

What surprised me was that when I ran the web servers at full load and profiled the CPU usage something like 99% of CPU time was spent serializing and deserializing session state. Subsequently we implemented a customised 'caching' state server; this always serializes state but also keeps state in-memory so that if you use sticky sessions the state doesn't have to be deserialized most of the time. This improved server throughput by a factor of 2; However, serialization still makes up 98% or more of CPU time.

We obtained some further improvements in speed by 'trimming' unnecessary object references between objects in the session state prior to serialization - fixing up the references manually upon desrialization. This improved speed by another 10-20% or so. The reasoning here is that some of the performance loss is due to the built in serialization having to walk the graph of object pointers, which becomes a more complex task with more pointers.

Continuining the investigation we wrote customized serialization routines for some of our classes rather than relying on .Net's built-in serialization. What we found was that performance was greatly improved, by a factor of about 50x. It seems the bulk of the CPU load is caused by built in .Net serialization, which in turn is slow due to reliance on using Reflection to walk the object pointers/graph and extract field data.

It's very tempting to boost our performance by 50x, thus reducing the web server hardware requirements by a large factor (and power requirements by a lesser but still significant factor). The options currently are:

1) Write customized serialization. This is an issue due to the complexity of the task and the maintenance overhead it generates, that is, any change to class state requires a change to the serialization/deserialization routines.

2) Some third party solution. Perhaps some product that automatically generates state save/load code at build time, thus eliminating the need to use Reflection.

I'd be very interested to know if anyone knows of a third party solution, or has encountered this issue as I haven't found any mention of it from internet searches.

UPDATE: Some have suggested a sort of halfway solution between the default built-in serialization and pure customized serialization routines. The idea is that you implement customized serialization for the classes that affect performance the most by, e.g. overiding ISerializable. This is an interesting and promising approach; However I still think there's scope for a complete replacement for built-in serialization without having to write and maintain any custom code - this can't be done at runtime because Reflection is needed to query objects and access private data. But it is theoretically possible to post-process already built assemblies and inject new methods as an additional build step. Some profilers use this approach to inject profiling code into assemblies after they have been built by the C# compiler. Also I /think/ I read somewhere that the .Net framework supports injecting methods into classes - thus all the messing around with IL is potentially taken care of.

Answer 1

Unfortunately I am only aware of option one and tbh that can start to get very painful to work on.

But it does only what you want so it's as quick as it gets.

Good luck.

Answer 2

One other option is to aggressively disable ViewState on all controls that are not touched on server side postbacks.