Slow serialization of WCF response using DataContractSerializer

Question

I have a WCF service that performs a search and returns a list of reasonably complex objects to the client. This is an EAV system, so each entity that is returned has a list of values attached that varies in size depending on the entity blueprint.

In my test search, I have confirmed through logging that the actual search takes under a second to complete in almost every case. The very last thing I do before returning the response to the client is log that processing has finished.

Unfortunately the client does not receive the response until 15-20 seconds after I have finished with it. The total response size is about 250kb, so pretty small. This is being transmitted via LAN at the minute and I have tried disabling the firewall and antivirus to make sure neither is interfering.

I noticed, however, that if the response was considerably smaller, for instance by removing the fields attached to each entity, that the response came through a lot faster. I also tried stepping through a locally hosted (IIS) copy of the service, and after passing the final return statement it still took another 15 seconds to reach the local client application.

I am using basicHttpBinding as the service will be consumed by both .Net and PHP clients.

Now then, can anyone suggest a way I can confirm that this is indeed the case? And how might I fix the cripplingly slow serialization times?

Edit:

To clarify, I have marked each class with the [DataContract] attribute and each property with [DataMember] - WCF is handling the serialization when I return the data. In this case it is a List of type Entity (a custom class that contains a list of values.

Edit 2:

I've tested the speed of DataContractSerializer and it took about 15 seconds to write a list of 65 returned entities to a simple memory stream. This seems ridiculous, and I'm not sure what has changed to make it so painfully slow.

Answer 1

I've figured it out, and it is quite embarrassing.

Whilst testing the speed of the DataContractSerializer, I was using a list of 65 products that were being returned from my search. I decided to load in all products (about 600) in the database and then serialize them into memory, but got a few out of memory exceptions, so started writing the results to a text file instead.

It turns out the text file was 1.5GB, which is about 3 times the size of the entire database. It took 17 seconds to be serialized. So actually, it was doing a bloody good job. What was happening is that each product could have a list of attached entities, and these were being loaded in as well. Because they are serialized, those entities were duplicated, where they exist in the database only once.

Along with stripping out a ton of meta data that the client does not need, I managed to get a list of 50 products (which had been 194MB) down to a mere 3MB. (update: this weekend I got the 1.5GB list down to 66MB).

The moral of the story? Listen to the community. Everyone else told me how fast DataContractSerializer is, so when it seemed slow to me, I should have blamed myself instead of DataContractSerializer.

UPDATE:

I spent some time trying to figure out why this was a sudden problem. The answer lies in the nature of EAV systems - the list of data attached to any entity is dynamic. I had originally only been loading field data when an idividual entity was requested - multiple entities only loaded the bare minimum for the sake of speed. After I implemented caching I changed this to load all field data regardless, but hadn't anticipated how much of a difference this would make to the volume of data, due to the complex data models and large number of entities. Really I should not have assumed anything about the amount of data I was requesting.