What is the cause of this OutOfMemoryException on Mutex constructor?

Question

I am getting an System.OutOfMemoryException on this line of code:

mutex2 = new Mutex(true, "Name2");

Here is the stacktrace:

{"Exception of type 'System.OutOfMemoryException' was thrown."}
   at Microsoft.Win32.Win32Native.CreateMutex(SECURITY_ATTRIBUTES lpSecurityAttributes, Boolean initialOwner, String name)
   at System.Threading.Mutex.CreateMutexHandle(Boolean initiallyOwned, String name, SECURITY_ATTRIBUTES securityAttribute, SafeWaitHandle& mutexHandle)
   at System.Threading.Mutex.MutexTryCodeHelper.MutexTryCode(Object userData)
   at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
   at System.Threading.Mutex.CreateMutexWithGuaranteedCleanup(Boolean initiallyOwned, String name, Boolean& createdNew, SECURITY_ATTRIBUTES secAttrs)
   at System.Threading.Mutex..ctor(Boolean initiallyOwned, String name, Boolean& createdNew, MutexSecurity mutexSecurity)
   at System.Threading.Mutex..ctor(Boolean initiallyOwned, String name)
   at Foo.FooDefinitions.FooManager.FooForm.FooForm_Load(Object sender, EventArgs e) in c:\tfs\DWS\TRUNK\DEV\FooDefinitions\FooManager\FooForm.cs:line 92

It will only occur when I use impersonation. Without impersonation (running on my normal Windows-account) it will run fine. The impersonation is something like this:

    if (!NativeMethods.LogonUser(userName, domainName, password, 2, 0, ref this._tokenHandle)) // [DllImport("advapi32.dll", SetLastError = true, CharSet = CharSet.Auto)]
    {
        throw new Win32Exception(Marshal.GetLastWin32Error());
    }
    this._impersonatedUser = new WindowsIdentity(this._tokenHandle).Impersonate();

EDIT: Just to eloborate, I am creating automated tests on legacy code. I would have removed the use of mutexes if I could. I am currently investigating the SecurityCriticalAttribute on the Mutex constructor.

EDIT2: Here is a full example of the code:

using Microsoft.VisualStudio.TestTools.UnitTesting;
using System;
using System.ComponentModel;
using System.Net;
using System.Runtime.InteropServices;
using System.Security.Principal;
using System.Threading;

namespace ReinierDG.MutexTesting
{
    [TestClass]
    public class MutexTest
    {
        [TestMethod]
        public void CreateMutexUnderImpersonation()
        {
            var credentials = new NetworkCredential("testagent", "secretpassword");
            var tokenHandle = new IntPtr();
            if (!NativeMethods.LogonUser(credentials.UserName, credentials.Domain, credentials.Password, 2, 0, ref tokenHandle))
            {
                throw new Win32Exception(Marshal.GetLastWin32Error());
            }
            var impersonatedUser = new WindowsIdentity(tokenHandle).Impersonate();
            // this will run indefinately or untill memory is full with 1 cpu core at 100%
            var mutex = new Mutex(true, "test");
        }

        internal static class NativeMethods
        {
            [DllImport("advapi32.dll", SetLastError = true, CharSet = CharSet.Auto)]
            internal static extern bool LogonUser([MarshalAs(UnmanagedType.LPWStr)]string lpszUsername, [MarshalAs(UnmanagedType.LPWStr)]string lpszDomain, [MarshalAs(UnmanagedType.LPWStr)]string lpszPassword, int dwLogonType, int dwLogonProvider, ref IntPtr phToken);
        }
    }
}

Answer 1

    // this will run indefinately or untill memory is full

Well, that would be one explanation. We'll have to assume that the comment just doesn't match the code. Most obvious issue here is that you did not post a stack trace that is relevant enough to the error and will not help you to diagnose the underlying problem. I can only post hints to get you to the next stage.

It is too easy to assume that it is CreateMutex() that failed. That however is not the case, failure of that winapi function is reported differently, you'd see __Error.WinIOError() back in the stack trace. And the error code would be different, you'd get error 1450, ERROR_NO_SYSTEM_RESOURCES, "Insufficient system resources exist to complete the requested service".

It is in fact the CLR that threw the exception. Or in other words, it is the pinvoke marshaller that failed. That considerably complicates the diagnostic, there are a very large number of places in the very large amount of code where it can throw OOM. It often needs to allocate unmanaged memory to get the pinvoke job done, if that fails then you get the OOM-kaboom. Many ways that can happen, internal unmanaged heap corruption is certainly enough. Your LogonUser() pinvoke declaration is technically wrong (CharSet.Auto != UnmanagedType.LPWStr), but not wrong enough to explain this problem.

You'll need to get closer to the root of the exception and that requires enabling the unmanaged debugger. For VS2015, use Project > Properties > Debugging tab > tick the "Enable native code debugging" checkbox. You'll need debugging symbols for the CLR to make sense of the stack trace. Use Tools > Options > Debugging > Symbols > tick "Microsoft Symbol Server". You need to make the debugger stop on the first-chance exception, use Debug > Windows > Exception Settings > tick "Win32 Exceptions".

You'll know a lot more now, you can post a much better stack trace in your question. Still, the odds that this is going to give SO users or you a crystal-clear diagnostic that shows how this mishap could be explained by impersonation is remote. Calling in the help of Microsoft Support would be wise, they however are going to need to know a lot more about exactly how that "testagent" account is configured. Do keep in mind that such accounts are often intentionally crippled to ensure that unit tests can't demand too many system resources.