ASP.NET HttpContext Cache removes right after insertion

Question

I have an ASP.NET 4 web-service.
It has an ImportModule action in a ModuleController controller.

That's how it works:

1. The user uploads a module as a CSV-file.
2. This file is being read using HttpPostedFileBase.InputStream and custom CSV-reading class.
3. This file is being transformed to a C# object according to some rules and validations. If a file is valid, then it transforms to C# object, stores in a Cache with unique GUID name and a user is redirected to CompleteImportModule action.
4. User checks if data is correct and he confirms uploading.

Long story short, there is a code which tells you more:
ImportModule action.

public ActionResult ImportModule(HttpPostedFileBase file)
{
    if (!ModelState.IsValid) 
    {
        return RedirectToAction("Index");
    }

    ModuleQuestion[] questions;
    ModuleInfo moduleInfo;
    string uploadId = Guid.NewGuid().ToString();

    // It is my custom CSV-reader and it works. Values are assigned
    FormDataCsvReader csvReader = new FormDataCsvReader(file.InputStream);
    if (!csvReader.Process(out questions, out moduleInfo))
    {
        // File is invalid
        return RedirectToAction("Index");
    }

    ViewBag.UploadId = uploadId;
    ViewBag.ModuleInfo = moduleInfo;
    ViewBag.Questions = questions;

    HttpContext.Cache.Add("UploadModule_" + uploadId,
        new Tuple<ModuleInfo, ModuleQuestion[]>(moduleInfo, questions),
        null,
        Cache.NoAbsoluteExpiration,
        TimeSpan.FromMinutes(30),
        CacheItemPriority.NotRemovable,
        (k, v, r) => 
        {
            LoggingFactory.GetLogger().Debug("Removed from cache: {0}. Reason: {1}", k, r);
        });     

    return View();   
}

In View ImportModule:

// Output data from ViewBag.ModuleInfo and ViewBag.Questions

<form method="POST" action="@Url.Action("CompleteImportModule")">
    <input type="hidden" name="uploadId" value="@ViewBag.UploadId"/>
    <input type="submit" value="Send">
</form>

CompleteImportModule action:

[HttpPost]
public ActionResult CompleteImportModule(string uploadId)
{
    var item = HttpContext.Cache["UploadModule_" + uploadId];
    if (item == null) RedirectToAction("Index");
    // upload module 
    HttpContext.Cache.Remove("UploadModule_" + uploadId);
    return RedirectToAction("Index");
}

However, I met some problems. I cannot upload the module because the value is removed from a Cache right after being inserted. It is stored only for a second:

DEBUG 2015-06-22 15:00:18,696 thread 85: Added to cache:
UploadModule_c843077d-21d0-4e9f-9e5e-3df82da4bac8

DEBUG 2015-06-22 15:00:19,935 thread 48: Removed from cache:
UploadModule_c843077d-21d0-4e9f-9e5e-3df82da4bac8. Reason: Removed

The reason is "Removed" meaning that it is not expired and IIS hasn't removed it due to optimization but it looks like I removed is myself. I am pretty sure that I am even not accessing this cache record before CompleteImportModule.

I have tried putting new StackTrace().ToString() in a CacheItemRemovedCallback. That's it, if it can help somehow:

at CPMAdministrator.Controllers.ModulesReferenceController.<ImportModule>b__9(String key, Object value, CacheItemRemovedReason reason)
   at System.Web.Caching.CacheEntry.CallCacheItemRemovedCallback(CacheItemRemovedCallback callback, CacheItemRemovedReason reason)
   at System.Web.Caching.CacheEntry.Close(CacheItemRemovedReason reason)
   at System.Web.Caching.CacheSingle.UpdateCache(CacheKey cacheKey, CacheEntry newEntry, Boolean replace, CacheItemRemovedReason removedReason, Object& valueOld)
   at System.Web.Caching.CacheSingle.Dispose(Boolean disposing)
   at System.Web.Caching.CacheMultiple.Dispose(Boolean disposing)
   at System.Web.HttpRuntime.Dispose()
   at System.Web.HttpRuntime.ReleaseResourcesAndUnloadAppDomain(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()

Why is it happening? Is it sort of IIS pool recycling? How can I ensure that the file is not being removed? Or how can I effictively store this data another way?

Answer 1

I have spent several hours finding an answer and found it right after posting a question! Let me share my experience with you.

According to my StackTrace, I understood that the cache has been cleared because the application has been ended and disposed:

at System.Web.HttpRuntime.Dispose()
at System.Web.HttpRuntime.ReleaseResourcesAndUnloadAppDomain(Object state)

All I needed is to find a reason of it.

I have opened my Global.asax file and added a Application_End method.

public class MvcApplication : HttpApplication
{
    protected void Application_End()
    {
    }
}

It has been triggered right after view rendering success and right before cache clearing. Success! Now I needed to know the reason of application ending.

This post helped me:

There's System.Web.Hosting.HostingEnvironment.ShutdownReason property that indicates why the application is being terminated. Its value can be retrieved from inside Application_End().

I added a breakpoint in a beginning of Application_End and added System.Web.Hosting.HostingEnvironment.ShutdownReason to watch.

That's what it stored: BinDirChangeOrDirectoryRename.

After that, I have understood that the reason is that my log4net is writing logs right in BinDirectory. I just never knew that IIS is finishing web applications if bin directory has been changed.

I have moved my logs to a parent (application itself) folder and now it works.
It looks like I need to read more about ASP.NET.
I hope that it will help someone. Thanks to everybody who tried to help.