EF6 OutOfMemoryException Evaluating Entity with varbinary(max) Property

Question

In my WebAPI web app, I've added a varbinary(max) field to a table, and a byte[] field to the POCO (BatchCharge). This entity has a child entity (Charge).

Visual Studio 2013, Entity Framework 6, SQL Server 2014.

The data model is as follows:

public class BatchCharge
{
    public int ID { get; set; }
    public byte[] FileData { get; set; }
    public virtual ICollection<Charge> Charges { get; set; } 
}

public class Charge
{
    public int ID { get; set; }
    public DateTime CreatedUTC { get; set; }
    public decimal Amount { get; set; }
    public virtual BatchCharge BatchCharge { get; set; }
}

The mapping is done on the Charge entity (child) as follows:

public class ChargeMap : EntityTypeConfiguration<Charge>
{
    public ChargeMap()
    {
        // Primary Key
        HasKey(t => t.ID);

        // Table and Column Mappings
        ToTable("Charge");
        Property(t => t.ID).HasColumnName("ID");
        Property(t => t.ID).IsRequired();

        Property(t => t.CreatedUTC).HasColumnName("CreatedUTC");
        Property(t => t.CreatedUTC).IsRequired();

        Property(t => t.Amount).HasColumnName("Amount");
        Property(t => t.Amount).IsRequired();

        HasRequired(t => t.BatchCharge)
            .WithMany(t => t.Charges)
            .HasForeignKey(d => d.BatchChargeID);
    }
}

When retrieving a list of BatchCharges using the method below:

    [ActionName("GetBatchCharges")]
    [HttpGet]
    [Route("api/charges/batches")]
    [Authorize(Roles = "Administrator")]
    public HttpResponseMessage GetBatchCharges(int skip = 0,
        int take = 25,
        int statusFilter = 0)
    {
        try
        {
            var batchCharges = _centralDb.BatchCharges.AsQueryable();
            if (statusFilter > 0)
            {
                batchCharges = batchCharges.Where(bc => bc.StatusID == statusFilter);
            }

            // Page and list.
            var allBatchCharges = batchCharges.OrderByDescending(c => c.CreatedUTC);
            var totalCount = allBatchCharges.Count();
            var thePage = allBatchCharges.Skip(take * skip).Take(take).ToList();

            // Transform and return.
            var result = new
            {
                TotalCount = totalCount,
                CurrentPage = skip,
                BatchCharges = thePage.Select(c => MapperFactory.Mapper.Map<BatchCharge, BatchChargeDTO>(c)).ToList()
            };
            return Request.CreateResponse(HttpStatusCode.OK, result);
        }
        catch (Exception ex)
        {
            const string message = "Exception getting batch charges.";
            Logger.Error(ex, message);
            return Request.CreateErrorResponse(HttpStatusCode.InternalServerError, new HttpError(message));
        }
    }

I get an OutOfMemoryException. When I break on the line above and try to view the query results, I see instead the error "The function evaluation was disabled because of an out of memory exception.". See screenshot below.

This tells me the out of memory exception is occurring inside EF6.

I've seen several related answers about how to increase process memory. I don't believe this is the problem. The test data includes six rows in the BatchCharge (parent) table, and a max file size of 27 KB!

The file is a CSV file with the child entity (Charges) information. I have no problem uploading and saving BatchCharges with file data using this model. I've also used it successfully on small numbers of Charges (child entities) per BatchCharge.

The problem started when I uploaded a BatchCharge with 600 Charges as children. As I mentioned, it's not a file size issue, because the file size is 27KB.

If there is a circular reference here, it should happen in many other similarly configured relationships I have. Or is the byte array field creating the problem in conjunction with the parent/child relationship? If so, how?

EDIT: When I surround the code with a try/catch block, and run it step by step in a debugger, it doesn't throw an exception anywhere in my code! However, the response that reaches the browser includes a standard IIS unhandled exception page, with a server error as follows (run the snippet to see output):

         body {font-family:"Verdana";font-weight:normal;font-size: .7em;color:black;} 
         p {font-family:"Verdana";font-weight:normal;color:black;margin-top: -5px}
         b {font-family:"Verdana";font-weight:bold;color:black;margin-top: -5px}
         H1 { font-family:"Verdana";font-weight:normal;font-size:18pt;color:red }
         H2 { font-family:"Verdana";font-weight:normal;font-size:14pt;color:maroon }
         pre {font-family:"Consolas","Lucida Console",Monospace;font-size:11pt;margin:0;padding:0.5em;line-height:14pt}
         .marker {font-weight: bold; color: black;text-decoration: none;}
         .version {color: gray;}
         .error {margin-bottom: 10px;}
         .expandable { text-decoration:underline; font-weight:bold; color:navy; cursor:hand; }
         @media screen and (max-width: 639px) {
          pre { width: 440px; overflow: auto; white-space: pre-wrap; word-wrap: break-word; }
         }
         @media screen and (max-width: 479px) {
          pre { width: 280px; }
         }

    <body bgcolor="white">

            <span><H1>Server Error in '/' Application.<hr width=100% size=1 color=silver></H1>

            <h2> <i>Exception of type 'System.OutOfMemoryException' was thrown.</i> </h2></span>

            <font face="Arial, Helvetica, Geneva, SunSans-Regular, sans-serif ">

            <b> Description: </b>An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

            <br><br>

            <b> Exception Details: </b>System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.<br><br>

            <b>Source Error:</b> <br><br>

            <table width=100% bgcolor="#ffffcc">
               <tr>
                  <td>
                      <code>

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.</code>

                  </td>
               </tr>
            </table>

            <br>

            <b>Stack Trace:</b> <br><br>

            <table width=100% bgcolor="#ffffcc">
               <tr>
                  <td>
                      <code><pre>

[OutOfMemoryException: Exception of type &#39;System.OutOfMemoryException&#39; was thrown.]
   System.IO.MemoryStream.set_Capacity(Int32 value) +89
   System.IO.MemoryStream.EnsureCapacity(Int32 value) +90
   System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count) +326
   Microsoft.VisualStudio.Web.PageInspector.Runtime.Tracing.ArteryFilter.Write(Byte[] buffer, Int32 offset, Int32 count) +106
   System.Web.HttpWriter.FilterIntegrated(Boolean finalFiltering, IIS7WorkerRequest wr) +475
   System.Web.HttpResponse.FilterOutput() +154
   System.Web.CallFilterExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +80
   System.Web.HttpApplication.ExecuteStepImpl(IExecutionStep step) +247
   System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean&amp; completedSynchronously) +117
</pre></code>

                  </td>
               </tr>
            </table>

            <br>

            <hr width=100% size=1 color=silver>

            <b>Version Information:</b>&nbsp;Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.7.3056.0

            </font>

    </body>

Answer 1

Thanks to @IvanStoev, found the problem in the DTOs: they were referencing EF properties (highlighted below), which caused out of memory exceptions with larger data sets.

public class BatchChargeDTO
{
    public int ID { get; set; }
    public byte[] FileData { get; set; }
    // Problem is here: type should be ChargeDTO!!
    public ICollection<Charge> Charges { get; set; }
}

public class ChargeDTO
{
    public int ID { get; set; }
    public DateTime CreatedUTC { get; set; }
    public decimal Amount { get; set; }
    public int? BatchChargeID { get; set; }
    // Problem is here: type should be BatchChargeDTO!!
    public BatchCharge BatchCharge { get; set; }
}