Out of memory error when publishing large result from GetListItems

Question

I'm using SDL Tridion 2009 SP1 on a 64-bit server and trying to publish a massive XML of all the Multimedia Components in the system (190K+). I'm using the folder.GetListItems(filter) method with the filter set to Recursive="true".

The template runs for several seconds and then blows up with an out of memory error:

<?xml version="1.0"?>
<tcm:Error xmlns:tcm="http://www.tridion.com/ContentManager/5.0" ErrorCode="7" Category="7" Source="Kernel" Severity="1"><tcm:Line ErrorCode="7" Cause="true"><![CDATA[Out of memory]]></tcm:Line><tcm:Details><tcm:CallStack><tcm:Location>FolderBLST.GetListData</tcm:Location></tcm:CallStack></tcm:Details></tcm:Error>
   at Tridion.ContentManager.Interop.TDSBL._IBLOrganizationalItemST.GetListData(UserContext userContext, String URI, EnumListKind listKind, ListColumnFilter columnFilter, String rowFilter)
   at Tridion.ContentManager.ContentManagement.OrganizationalItem.GetListItems(Filter filter)
   at myNS.myTbb.Transform(Engine engine, Package package)
   at Tridion.ContentManager.Templating.Assembly.AssemblyMediator.Transform(Engine engine, Template template, Package package)
   at Tridion.ContentManager.Templating.Assembly.CSharpSourceCodeMediator.RunTemplate(Engine engine, Package package, String templateUri, String className)
   at Tridion.Templating.CSharpTemplate.CSharpSourceTemplate.Transform(Engine __engine, Package __package)
   at Tridion.ContentManager.Templating.Assembly.CSharpSourceCodeMediator.Transform(Engine engine, Template template, Package package)
   at Tridion.ContentManager.Templating.Engine.ExecuteTemplate(Template template, Package package)
   at Tridion.ContentManager.Templating.Engine.InvokeTemplate(Package package, TemplateInvocation templateInvocation, Template template)
   at Tridion.ContentManager.Templating.Compound.CompoundTemplateMediator.Transform(Engine engine, Template templateToTransform, Package package)
   at Tridion.ContentManager.Templating.Engine.ExecuteTemplate(Template template, Package package)
   at Tridion.ContentManager.Templating.Engine.InvokeTemplate(Package package, TemplateInvocation templateInvocation, Template template)
   at Tridion.ContentManager.Templating.Engine.TransformPackage(Template template, Package package)
   at Tridion.ContentManager.Templating.Engine.TransformItem(Template template, IdentifiableObject itemToRender)
   at Tridion.ContentManager.Templating.Debugging.DebuggingEngine.Run()
   at Tridion.ContentManager.Templating.Debugging.DebugSession.Run()

From the stack trace it looks as if the error is happening in the Business Layer of the CM server. Is there a memory setting that I can increase for this, and if so, which is it?

Answer 1

folder.GetListItems(filter) recursive is going to consume a huge amount of resources in your scenario.

If you have a big amount of multimedia items, it is adding a huge overload to the system. Even if you try to scale the server, you will face the same issue at some point.

In general, you will face this issue as you are trying to perform a huge operation for data retrieval.

Maybe you can use different techniques for achieve the same scenario (The following are samples)

Scenario 1

Using the Event System, you can add information of the binary (when is created, etc...) to a common repository (example an XML stored in a field of a System Component) and publish that XML once in a while.

If you need just a list of ids for instance, use a Component created for store that information. You can also define a range of ids and create new Components, if required, for not having too many entries in only one (example: ids from 0 to 10000 will be stored in a component named References_0_10000, ids from 10001 to 20000 in a component named References_10001_20000).

Scenario 2

Split the initial load in Sub-Loads (still using the recursive=true) when processing subfolders within the main Folder for instance, and assemble the results. In this case you minimize the folder.GetListItems(filter) load.

Scenario 3

Use still the folder.GetListItems(filter) multiple times but implement the recursive logic in your code, instead to use that in the filter options and assemble the results returned per each call.

Note: Check the TimeOut settings of the SDL Tridion Content Manager configuration MMC Snap-in and increase those in case that helps.