MemoryFailPoint throws InsufficientMemoryException unconditional

Question

I have a very strange issue; in our REST application we have introduced an option for upload of a WebPackage. Since these can be rather large when unpacked, we wanted to assure a successful experience with memory check by MemoryFailPoint.

On my local IIS 7.5 this works flawlessly - even up to a max. expected value of 2GB. On our virtual Windows Server 2008R2 x64 with IIS 7.5 this fails unconditionally - even if only trying 1MB.

The virtual server is hosted on VMWare ESXi 4.1.0, 348481.

Here is the sample code that fails on the virtual machine:

using (MemoryFailPoint failPoint = new MemoryFailPoint(1))
{
   ... // deliberately excluded webpackage code
}

And the exception (in XML):

<InsufficientMemoryException>
    <Message>Insufficient memory to meet the expected demands of an operation, and this system is likely to never satisfy this request.  If this is a 32 bit system, consider booting in 3 GB mode.</Message>
    <StackTrace>
        <Line>at System.Runtime.MemoryFailPoint..ctor(Int32 sizeInMegabytes)</Line>
        <Line>at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]&amp; outputs)</Line>
        <Line>at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc&amp; rpc)</Line>
        <Line>at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc&amp; rpc)</Line>
        <Line>at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc&amp; rpc)</Line>
        <Line>at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)</Line>
    </StackTrace>
    <UserDefinedInformation>
        <failpointSizeInMegaBytes>1.00</failpointSizeInMegaBytes>
        <gcTotalMemoryBeforeInMegaBytes>153.34</gcTotalMemoryBeforeInMegaBytes>
    </UserDefinedInformation>
</InsufficientMemoryException>

And a similiar exception from my machine - take note that i needed to allocate 64GB BEFORE i could trigger the exception:

<InsufficientMemoryException>
    <Message>Insufficient available memory to meet the expected demands of an operation at this time.  Please try again later.</Message>
    <StackTrace>
        <Line>at System.Runtime.MemoryFailPoint..ctor(Int32 sizeInMegabytes)</Line>
        <Line>at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]&amp; outputs)</Line>
        <Line>at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc&amp; rpc)</Line>
        <Line>at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc&amp; rpc)</Line>
        <Line>at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc&amp; rpc)</Line>
        <Line>at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)</Line>
    </StackTrace>
    <UserDefinedInformation>
        <failpointSizeInMegaBytes>65536.00</failpointSizeInMegaBytes>
        <gcTotalMemoryBeforeInMegaBytes>150.63</gcTotalMemoryBeforeInMegaBytes>
    </UserDefinedInformation>
</InsufficientMemoryException>

This is driving me nuts .. for real .. but hey; take note of the two (very) different exceptions:

Server: Insufficient memory to meet the expected demands of an operation, and this system is likely to never satisfy this request. If this is a 32 bit system, consider booting in 3 GB mode.

Using dotPeek from Jetbrains and this website, http://reflector.webtropy.com/default.aspx/4@0/4@0/untmp/DEVDIV_TFS/Dev10/Releases/RTMRel/ndp/clr/src/BCL/System/Runtime/MemoryFailPoint@cs/1305376/MemoryFailPoint@cs, we can see what might be the issue (although I am still clueless):

        // Check to see that we both have enough memory on the system
        // and that we have enough room within the user section of the
        // process's address space.  Also, we need to use the GC segment
        // size, not the amount of memory the user wants to allocate.
        // Consider correcting this to reflect free memory within the GC
        // heap, and to check both the normal & large object heaps.
        ulong num1 = (ulong) sizeInMegabytes << 20;
        this._reservedMemory = num1;
        ulong size = (ulong)(Math.Ceiling((double)num1 / (double)MemoryFailPoint.GCSegmentSize) * (double)MemoryFailPoint.GCSegmentSize);
        if (size >= MemoryFailPoint.TopOfMemory)
            throw new InsufficientMemoryException(Environment.GetResourceString("InsufficientMemory_MemFailPoint_TooBig"));

Local: Insufficient available memory to meet the expected demands of an operation at this time. Please try again later.

EDIT

Hans and I had a discussion about VM Size but since I have now concluded that it is more or less equal on the development server and my machine, I will rule this part out for now. Any hints, suggestions etc. is more than welcome.

Process Info

Process Name: w3wp
PID: 5716
User Name: NT AUTHORITY\NETWORK SERVICE
Working Set: 376.456 K
Peak Working Set: 432.400 K
Private Working Set: 320.684 K
Commit Size: 538.552 K
Handles: 919
Threads: 39

Operating System (WMI)

BootDevice: \Device\HarddiskVolume1
BuildNumber: 7601
BuildType: Multiprocessor Free
Caption: Microsoft Windows Server 2008 R2 Standard 
CodeSet: 1252
CountryCode: 45
CreationClassName: Win32_OperatingSystem
CSCreationClassName: Win32_ComputerSystem
CSDVersion: Service Pack 1
CSName: SOME_NAME [MODIFIED]
CurrentTimeZone: 120
DataExecutionPrevention_32BitApplications: True
DataExecutionPrevention_Available: True
DataExecutionPrevention_Drivers: True
DataExecutionPrevention_SupportPolicy: 3
Debug: False
Description: Development Server
Distributed: False
EncryptionLevel: 256
ForegroundApplicationBoost: 2
FreePhysicalMemory: 5.366.516 K
FreeSpaceInPagingFiles: 8.368.456 K
FreeVirtualMemory: 12.985.412 K
InstallDate: 17-01-2011 15:01:55
LargeSystemCache: null
LastBootUpTime: 11-09-2012 14:44:33
LocalDateTime: 03-10-2012 14:13:47
Locale: 0406
Manufacturer: Microsoft Corporation
MaxNumberOfProcesses: 4294967295
MaxProcessMemorySize: 8.589.934.464 K
MUILanguages: System.Object[]
Name: Microsoft Windows Server 2008 R2 Standard |C:\Windows|\Device\Harddisk0\Partition2
NumberOfLicensedUsers: null
NumberOfProcesses: 70
NumberOfUsers: 7
OperatingSystemSKU: 7
Organization: 
OSArchitecture: 64-bit
OSLanguage: 1033
OSProductSuite: 272
OSType: 18
OtherTypeDescription: null
PAEEnabled: null
PlusProductID: null
PlusVersionNumber: null
Primary: True
ProductType: 3
RegisteredUser: Windows User
SerialNumber: 11111-111-1111111-11111 [MODIFIED]
ServicePackMajorVersion: 1
ServicePackMinorVersion: 0
SizeStoredInPagingFiles: 8.388.152 K
Status: OK
SuiteMask: 272
SystemDevice: \Device\HarddiskVolume2
SystemDirectory: C:\Windows\system32
SystemDrive: C:
TotalSwapSpaceSize: null
TotalVirtualMemorySize: 16.774.452 K
TotalVisibleMemorySize: 8.388.152 K
Version: 6.1.7601
WindowsDirectory: C:\Windows

VMMap Screenshot from Development Server

Being curious about the Virtual Size, I download VMMap from Sysinternals: http://technet.microsoft.com/en-us/sysinternals/dd535533

VMMap Screenshot from My Local Machine (where MemoryFailPoint works as expected)

It is still not clear to me why the MemoryFailPoint fails - and it is important (especially when talking Windows Azure or other cloud providers) to make sure there is enough memory for the operation. To take an example, then an ExtraSmall instance will fail (even it's x64 architecture) somewhere in the process because of the very limited resources - this could be prevented by the MemoryFailPoint, hence assuring a valid state of data.

Any help is most appreciated.

Answer 1

There is a bug (or feature) in .net 4.5 and IIS.

The MemoryFailPoint class depends on an internal GC related variable, but under x64 this variable can be 0 and dividing by zero can generate positive infinity (double) and some operations (even multiply with zero) with this extreme value can generate a large double value (depends on the actual FPU settings).

You can check this bug very easy. Create a simple web application and add an page then paste this code:

<%
    var memoryFailPointType = typeof(System.Runtime.MemoryFailPoint);
    var staticPrivateFieldFlags = 
        System.Reflection.BindingFlags.NonPublic |
        System.Reflection.BindingFlags.Static;
    var gCSegmentSizeField = memoryFailPointType
        .GetField("GCSegmentSize", staticPrivateFieldFlags);
    var gCSegmentSize = (uint)gCSegmentSizeField.GetValue(null);
    string allocationResult;
    try
    {
        using (var mfp = new System.Runtime.MemoryFailPoint(1))
        {
            allocationResult = "Ok";
        }
    }
    catch (Exception exception)
    {
        allocationResult = exception.Message;
    }
%>
<%= gCSegmentSize %><br /> <!-- it can be 0 -->
<%= allocationResult %><br /> <!-- Insufficient memory to meet the expected demands of an operation, and this system is likely to never satisfy this request. If this is a 32 bit system, consider booting in 3 GB mode. -->

So if gCSegmentSize is zero and (ulong)(double.PositiveInfinite * (uint)0) doesn't equal 0, you shouldn't use MemoryFailPoint class.

In a console application the FPU control word is 0x09001F, but under IIS the FPU control word is 0x08001F.

Answer 2

There is a fix available for this issue in Hotfix Rollup 2828842 for .NET 4.5 http://support.microsoft.com/kb/2828842/en-us