MVC - System.Threading.ThreadAbortException in a parallel task

Question

In my MVC application, a superadministrator can set a queue of tasks such as updating the database. So, when an admin adds an update to the queue, the controller starts a new task that works in the background. However, when you add a few tasks, the application throws System.Threading.ThreadAbortException: Thread was being aborted. Moreover, the stack trace suggests that it happens on different lines in code.

I should also add that the tasks use EF6 entities to work with SQL-server, and according to the stack trace, it happens after or while performing operations on the database. Since updates are usually large, I use db.Configuration.AutoDetectChangesEnabled = false and manually save changes every 20k rows, disposing and recreating the database.

Example of a stack trace:

5:18:36 PM Wednesday, July 15, 2015: [REPORT] Exception(Line:456667;Section6): System.Threading.ThreadAbortException: Thread was being aborted. at System.Array.Copy(Array sourceArray, Int32 sourceIndex, Array destinationArray, Int32 destinationIndex, Int32 length, Boolean reliable) at System.Collections.Generic.List`1.set_Capacity(Int32 value) at System.Data.Entity.Core.Metadata.Edm.MetadataCollection`1.SetReadOnly() at System.Data.Entity.Core.Metadata.Edm.TypeUsage..ctor(EdmType edmType, IEnumerable`1 facets) at System.Data.Entity.Core.Common.CommandTrees.DbExpression..ctor(DbExpressionKind kind, TypeUsage type, Boolean forceNullable) at System.Data.Entity.Core.Common.CommandTrees.ExpressionBuilder.DbExpressionBuilder.PropertyFromMember(DbExpression instance, EdmMember property, String propertyArgumentName) at System.Data.Entity.Core.Mapping.Update.Internal.UpdateCompiler.GenerateEqualityExpression(DbExpressionBinding target, EdmProperty property, PropagatorResult value) at System.Data.Entity.Core.Mapping.Update.Internal.UpdateCompiler.BuildPredicate(DbExpressionBinding target, PropagatorResult referenceRow, PropagatorResult current, TableChangeProcessor processor, Boolean& rowMustBeTouched) at System.Data.Entity.Core.Mapping.Update.Internal.UpdateCompiler.BuildUpdateCommand(PropagatorResult oldRow, PropagatorResult newRow, TableChangeProcessor processor) at System.Data.Entity.Core.Mapping.Update.Internal.TableChangeProcessor.CompileCommands(ChangeNode changeNode, UpdateCompiler compiler) at System.Data.Entity.Core.Mapping.Update.Internal.UpdateTranslator.d__a.MoveNext() at System.Linq.Enumerable.d__71`1.MoveNext() at System.Data.Entity.Core.Mapping.Update.Internal.UpdateCommandOrderer..ctor(IEnumerable`1 commands, UpdateTranslator translator) at System.Data.Entity.Core.Mapping.Update.Internal.UpdateTranslator.ProduceCommands() at System.Data.Entity.Core.Mapping.Update.Internal.UpdateTranslator.Update() at System.Data.Entity.Core.Objects.ObjectContext.ExecuteInTransaction[T](Func`1 func, IDbExecutionStrategy executionStrategy, Boolean startLocalTransaction, Boolean releaseConnectionOnSuccess) at System.Data.Entity.Core.Objects.ObjectContext.SaveChangesToStore(SaveOptions options, IDbExecutionStrategy executionStrategy, Boolean startLocalTransaction) at System.Data.Entity.SqlServer.DefaultSqlExecutionStrategy.Execute[TResult](Func`1 operation) at System.Data.Entity.Core.Objects.ObjectContext.SaveChangesInternal(SaveOptions options, Boolean executeInExistingTransaction) at System.Data.Entity.Internal.InternalContext.SaveChanges() at MyWebsite.Controllers.AdminPanelController.ApplyUpdate(String filePath, HttpApplicationStateBase context, Int32 saveInterval, Boolean checkRepetitions, String onCollision)

Is there anything I can be doing wrong?

Answer 1

I guess it may be related to IIS idle state. You could try keep IIS running background threads using self-based requests each few minutes from another idle thread. It will keep IIS from aborting your another background long running threads. Method example to do self request each 10 minutes:

public void KeepIisBackgroundThreadsAlive(Object stateInfo) {
    while (true) {
        var serverOwnIp = Dns.GetHostEntry(Dns.GetHostName()).AddressList.First(o => o.AddressFamily == AddressFamily.InterNetwork).ToString();
        var req = (HttpWebRequest) WebRequest.Create(new Uri("http://" + serverOwnIp));
        req.Method = "GET";
        var response = (HttpWebResponse) req.GetResponse();
        var respStream = response.GetResponseStream();
        var delay = new TimeSpan(0, 0, 10, 0);
        Thread.Sleep(delay);
    }
}

Then you could start this method using ThreadPool.QueueUserWorkItem method in Global.asax.cs OnApplicationStarted method:

protected override void OnApplicationStarted(){
    ThreadPool.QueueUserWorkItem(KeepIisBackgroundThreadsAlive)
}

IMPORTANT UPDATE

Found that this approach is suitable only for disabled IIS Application Pool recycling. Read carefully this article in case you are not familiar with this IIS feature. Be aware that you should be 100% sure what you are doing.

If you use this approach without turned off recycling you may face website won't startup critical bug after 1-st recycling occurrence. For these websites that need Application Pool recycling I'm strongly recommending to move long running jobs into separate service not dependent on IIS(In case you use Azure Web Role - at least into separate Worker Role instance).

Answer 2

I think you'll have more positive results in the long run by properly abstracting this functionality out to another application, more specifically a Windows Service. At my company I wrote a Daemon that has long running/polling workers and it's been an essential part of our technology stack for over 4 years now. It may seem like a log of work, but it'll will pay you back dividends.

BTW as for the actual problem you're facing; I agree with @Vova in that your application is being hosted in IIS, and IIS will do many things to make sure your application doesn't bring down the rest of the server. Some of those things may include the termination of threads.

Here are a couple of links where people are talking about your issue (Google a bit and you'll find heaps more):

• How to know who kills my threads
• https://channel9.msdn.com/Forums/TechOff/63934-IIS-killing-my-thread

EDIT: I thought that I should probably elaborate a little more on the architecture of this Daemon service for those interested.

Basically it's not too complicated, yet very effective. It consists of "worker" classes that do stuff. A load balancing class manages all instances of workers and calls on them to do a slice of work, therefore the load balancer can keep track of when a worker last did something and tell them to do another chunk while roughly making sure the server doesn't come under too much load. This may sound quite tricky, but I keep it fairly simple. The cool part is that each worker defines a processing style which can be:

• Precalculate: Calculates the amount of work to do ahead of time. This allows x of y% ui updates. This is sometimes difficult (or impossible) which leads to the next two processing types...
• Eager: Keep going until there is nothing left, don't try to evaluate the amount of to do ahead of time
• CallOnce: The process function will be called once and then terminate without checking or calculating if any more work needs to be done

The load balancer stores the state of each worker in a MongoDB record so the process is tolerant to failures... having said that though each worker base class ensures that exceptions are handled, logged and emailed (although that doesn't mean that things like memory leaks couldn't bring down the process/machine).

The final aspect to consider is how the worker statuses are fed back to humans and manual intervention (pausing, force running etc) is achieved. I do this by exposing very lightweight WCF services, only 2 of them as well:

1. A control service for getting worker statuses and controlling them
2. An imperative messaging service that queues items for immediate processing. This actually appends to an MSMQ that the load balancer polls and passes onto the workers automatically.

Our admin portal makes use of the first service to create a page that automatically polls (using KnockoutJS) back to the admin website, then on to the Daemon.

We've found this to be a very nice (yet lightweight and simple) backend processing suite that handles a wide variety of tasks Solr record updates, reporting, data mining, filestore cleanup and a ton more.