Out of memory exception occurs when enqueuing and processing background jobs

Question

I am able to cause a reproducible out of memory exception to occur when enqueuing and processing background jobs with Hangfire.

The jobs are simple Console.WriteLine invocations so I wouldn't expect heap memory to increase the way that it does.

Have I configured incorrectly or should I think about filing an issue?

Results (VMMap)

Using Redis as backing storage for Jobs:

• At start, total heap = 29,088K;
• after 5,000 jobs, 938,672K;
• 6,000 jobs, 1,056,004K;
• 7,000 jobs, 1,219,296K;
• 8,000 jobs, heap value is not present;
• within 100 more jobs, the iisexpress.exe instance crashes.

With SQL storage the limit is much higher ~= 15,000 jobs.

Setup

• empty ASP.NET project;
• install Owin packages for IIS hosting and Hangfire;
• startup class and controller.

Packages

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="Hangfire.Core" version="1.6.6" targetFramework="net452" />
  <package id="Hangfire.Pro" version="1.4.7" targetFramework="net452" />
  <package id="Hangfire.Pro.PerformanceCounters" version="1.4.7" targetFramework="net452" />
  <package id="Hangfire.Pro.Redis" version="2.0.2" targetFramework="net452" />
  <package id="Hangfire.SqlServer" version="1.6.6" targetFramework="net452" />
  <package id="Microsoft.AspNet.WebApi.Client" version="5.2.3" targetFramework="net452" />
  <package id="Microsoft.AspNet.WebApi.Core" version="5.2.3" targetFramework="net452" />
  <package id="Microsoft.AspNet.WebApi.Owin" version="5.2.3" targetFramework="net452" />
  <package id="Microsoft.CodeDom.Providers.DotNetCompilerPlatform" version="1.0.0" targetFramework="net452" />
  <package id="Microsoft.Net.Compilers" version="1.0.0" targetFramework="net452" developmentDependency="true" />
  <package id="Microsoft.Owin" version="3.0.1" targetFramework="net452" />
  <package id="Microsoft.Owin.Host.SystemWeb" version="3.0.1" targetFramework="net452" />
  <package id="Newtonsoft.Json" version="9.0.1" targetFramework="net452" />
  <package id="Owin" version="1.0" targetFramework="net452" />
  <package id="StackExchange.Redis" version="1.1.606" targetFramework="net452" />
</packages>

Controller

public class DefaultController : ApiController
{
    static int _;

    [HttpPost]
    public void Post(int count = 1000)
    {
        for (var i = 0; i < count; ++i)
        {
            BackgroundJob.Enqueue(() => Console.WriteLine(_));

            ++_;
        }
    }
}

Startup

static class AppSettings
{
    internal static bool   HangfireUseRedis => true;
    internal static int    RedisDatabase    => 0;
    internal static string RedisConnection  => "localhost:6379";

    internal static string SqlConnection    => "Data Source=(localdb)\\v11.0;Initial Catalog=Hangfire";
}

public class Startup
{
    public void Configuration(IAppBuilder app)
    {
        var config = new HttpConfiguration();

        config.Routes.MapHttpRoute(
            name: "Default",
            routeTemplate: "{controller}/{id}",
            defaults: new { id = RouteParameter.Optional }
        );

        if (AppSettings.HangfireUseRedis)
        {
            var redisOptions = new RedisStorageOptions
            {
                Database = AppSettings.RedisDatabase,
                Prefix   = "Foobar:"
            };

            GlobalConfiguration.Configuration.UseRedisStorage(AppSettings.RedisConnection, redisOptions);
        }
        else
        {
            GlobalConfiguration.Configuration.UseSqlServerStorage(AppSettings.SqlConnection);
        }

        JobHelper.SetSerializerSettings(new JsonSerializerSettings { TypeNameHandling = TypeNameHandling.All });

        app.UseHangfireServer();
        app.UseHangfireDashboard();

        app.UseWebApi(config);
    }
}

Answer 1

After receiving your mini dump file (1.2 GB), I was able to get information about heaps of your process. Most of them don't contain anything interesting, and their size is relatively small, here is the excerpt for the most important onces:

GC Heap Size:    Size: 0x9f7eb8 (10452664) bytes.
Jit code heap:   Size: 0x1000 (4096) bytes total, 0x905a4d00 (2421837056) bytes wasted.

As we can see, GC Heap Size is about 10 MB, so there are no leaks in .NET code itself, since its size is relatively small. But Jit code heap looks very strange, so I decided to see what modules are used by the process, and found Stackify Profiler's one:

6b0d0000 6b23a000   StackifyProfiler_x86   (deferred)

PEB shows the environment variable StackifyIsPrefix=1 that tells us Stackify Prefix is used. Profilers may modify the JIT code for instrumentation puproses, so I've decided to install Stackify Prefix to try to reproduce the issue.

I've created a simple MVC application, modified the Home/Index action to enqueue 10000 of background jobs, and enabled the profiler. After doing this step, I've found that it takes too long to get that page – 1.5 minutes, and profiler didn't show any data. It was too long. So I decided to compare the timing with profiler turned off – it took only 5 seconds. That's a huge difference, but I was unable to reproduce memory problems.

I've decreased the number of jobs to 100, turned on the profiler and realized that every call to Redis is counted, there are hundreds of records for calls to Redis. Storing all of them may introduce memory issues, but I don't know exactly how they are stored in Stackify Prefix.

I was unable to reproduce the original memory issue. However, Stackify Prefix does significantly affect the execution by increasing its duration. Have you tried to disable the Stackify Prefix profiler and re-run your tests? Updated version may also fix the memory issue.

Answer 2

I can concur with the above comment from odinserj, because I wrote the Prefix profiler.

We've made some design changes to help account for background threads that run in libraries like Hangfire. The issue is that we keep shadow stacks in memory per thread - in a normal web app, we flush this stack when the request ends. But the threads that Hangfire spins up will exist for the lifetime of the app domain.

You'll find that in the lastest version, the impact should be much smaller, as we have accounted for some specific hangfire methods, and we then release some of that shadow stack.