Difficulty using WCF for uploading large files

Question

There are a number of other similar questions on SO about this. Unfortunately, many seem to be dupes of one another in some respect. I hope that this one will help others and put to rest other questions.

My project requirement is to upload 250MB files through IIS into a backend WCF service hosted in IIS. I created some unit tests for the backend WCF service hosted in IIS. They are:

1) Upload 1MB File
2) Upload 5MB File
3) Upload 10MB file
4) Upload 20MB File
5) Upload 200MB File

Right off the bat, it's probably clear that we need to be using some kind of streaming or chunking file transfer. I used this sample.

The sample describes a method that uses the .NET Stream object. A side effect of using the stream object, is that you must use Message contracts. It's not enough to put the Stream in your function's parameter list. So we do that.

By default, the web.config for this WCF service is pretty lean. And nothing works:

System.ServiceModel.ProtocolException: The remote server returned an unexpected response: (400) Bad Request. ---> System.Net.WebException: The remote server returned an error: (400) Bad Request.

After much searching and experimenting, it's clear that BasicHttpBinding is incompatible with this combination of the Stream object and the MessageContract. We must switch to WSHttpBinding.

To do this, the server's web.config gets slightly more complex under the section:

<system.serviceModel>
        <behaviors>
            <serviceBehaviors>
                <behavior>
                    <!-- To avoid disclosing metadata information, set the value below to false and remove the metadata endpoint above before deployment -->
                    <serviceMetadata httpGetEnabled="true"/>
                    <!-- To receive exception details in faults for debugging purposes, set the value below to true.  Set to false before deployment to avoid disclosing exception information -->
                    <serviceDebug includeExceptionDetailInFaults="true" httpHelpPageEnabled="true"/>
                </behavior>
                <behavior name="FileServiceBehavior">
                    <serviceMetadata httpGetEnabled="true"/>
                    <dataContractSerializer maxItemsInObjectGraph="2147483647"/>
                    <serviceDebug includeExceptionDetailInFaults="true"/>
                    <serviceThrottling maxConcurrentCalls="500" maxConcurrentSessions="500" maxConcurrentInstances="500"/>
                </behavior>
            </serviceBehaviors>
        </behaviors>
        <serviceHostingEnvironment multipleSiteBindingsEnabled="true" />
        <bindings>
            <wsHttpBinding>
                <binding name="FileServiceBinding" closeTimeout="10:01:00"
                  maxBufferPoolSize="104857600"
                  maxReceivedMessageSize="104857600" openTimeout="10:01:00"
                  receiveTimeout="10:10:00" sendTimeout="10:01:00"
                  messageEncoding="Mtom">
                    <readerQuotas maxDepth="104857600" maxStringContentLength="104857600"
                                  maxArrayLength="104857600" maxBytesPerRead="104857600"
                                  maxNameTableCharCount="104857600" />
                </binding>
            </wsHttpBinding>
        </bindings>
        <services>
            <service behaviorConfiguration="FileServiceBehavior" name="OMS.Service.FileService">
                <endpoint address="" binding="wsHttpBinding" bindingConfiguration="FileServiceBinding" contract="OMS.Service.IFileService"></endpoint>
            </service>
        </services>
    </system.serviceModel>

Without any more work to speak of, 1MB file are now passing with no problem.

In order to get files larger than 4MB to pass, you have to adjust a setting in the web.config in IIS (server side of your WCF service) This article from Microsoft explains what that setting is. For instance, if you set it to 8192, then you'll be able to upload the 5MB file, but not anything larger.

<httpRuntime maxRequestLength="8192" />

I set mine to something obscene for testing - 2147483647. The first 4 files pass this gate.

The 200MB didn't get a chance to make it to this gate for the next reason:

System.InsufficientMemoryException: Failed to allocate a managed memory buffer of 279620368 bytes. The amount of available memory may be low. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.

The explanation for this problem is described very well, by this poster.

Think of it like this. The 200MB file never made it out of the client. It has to be fully loaded up by the client, encrypted and then transmitted to the server.

When you use Visual Studio 2010 to generate the proxy classes for the service, it puts some stuff into your app.config. For me, it looks like this:

<binding 
 name="Binding_IFileService" closeTimeout="00:01:00"
 openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"
 bypassProxyOnLocal="false" transactionFlow="false" 
 hostNameComparisonMode="StrongWildcard"
 maxBufferPoolSize="524288" maxReceivedMessageSize="65536" messageEncoding="Mtom"
 textEncoding="utf-8" useDefaultWebProxy="true" allowCookies="false">
    <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
     maxBytesPerRead="4096" maxNameTableCharCount="16384" />
    <reliableSession ordered="true" inactivityTimeout="00:10:00"
         enabled="false" />
    <security mode="Message">
        <transport clientCredentialType="Windows" proxyCredentialType="None" realm="" />
        <message clientCredentialType="Windows" negotiateServiceCredential="true" />
    </security>
</binding>

The key is the security mode. It's set to "message" by default. That value is picked up by whatever is set on the server. By default, your server is using Message level security.

If you try to force it on the server to be like this:

 <security mode="None">

You get this error:

System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at http://localhost:8080/oms/FileService.svc that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. ---> System.Net.WebException: The remote server returned an error: (404) Not Found.

(I did remember to update the client proxy)

And so, that's where it stands for me.... Help!

Answer 1

WCF typically doesn't handle large file transfers well unless you implement streaming, which can actually be accomplished with BasicHttpBindings.

For my project, I have a custom host factory that creates service hosts:

protected override System.ServiceModel.ServiceHost CreateServiceHost(Type serviceType, Uri[] baseAddresses)
{
    ServiceHost host = new ServiceHost(serviceType, baseAddresses);

    ContractDescription contract = ContractDescription.GetContract(serviceType);

    BasicHttpBinding binding = new BasicHttpBinding();
    binding.OpenTimeout = TimeSpan.FromMinutes(1);
    binding.ReceiveTimeout = TimeSpan.FromMinutes(1);
    binding.SendTimeout = TimeSpan.FromHours(1);
    binding.TransferMode = TransferMode.StreamedResponse;
    binding.MessageEncoding = WSMessageEncoding.Mtom;

    ServiceEndpoint streaming = new ServiceEndpoint(contract, binding, new EndpointAddress(baseAddresses[0] + "/STREAMING"));

    host.AddServiceEndpoint(streaming);

    return host;
}

You will want to use StreamedRequest in your case.

And the implementation of the StreamingService:

[ServiceBehavior(ConcurrencyMode = ConcurrencyMode.Multiple, InstanceContextMode = InstanceContextMode.Single, AddressFilterMode = AddressFilterMode.Any)]
public class FileStreamingService : IFileStreamingV1
{
    Stream IFileStreamingV1.GetFileStream(string downloadFileLocation)
    {
        if (!File.Exists(downloadFileLocation))
        {
            throw new FaultException("The file could not be found");
        }

        FileStream stream = File.OpenRead(downloadFileLocation);
        return stream;
    }
}

I did not specify a maximum buffer size in the service itself but the client application in my case throttled the download to something like 5mb a second. In your case, the service itself will need to set the throttle behavior. This doesn't solve the issue of how you'll tell the service how many bytes are in the file in order to stream it properly, but it should give you a start.

You should also note the use of MTOM in the host configuration. MTOM was designed to help with the transfer of large files (not so great for small transfers).

I do not have an example of the client behavior, but your service should read the number of bytes from the upload stream's buffer and save them to file until nothing is left to stream. Although memory is cheap I would not recommend storing a complete in memory copy of the file, especially at 200mb.

You should also be aware that depending on your web hosting platform (IIS, Apache etc) you may also be limited to the amount of data that can be transfered at a given time. However, a change in configuration can usually resolve any hosting issues.

I hope this helps.