VM Tracker shows large Dirty Size

Question

There is a part of my app where I perform operations concurrently. They consist of initializing many CALayers and rendering them to bitmaps.

Unfortunately, during these operations (each takes about 2 seconds to complete on an iphone 4), the Dirty Size indicated by VM Tracker spikes to ~120MB. Allocations spike to ~12MB(does not accumulate) From my understanding, the Dirty size is memory that cannot be freed. so often, my app and all other apps in the background gets killed.

Incident Identifier: 7E6CBE04-D965-470D-A532-ADBA007F3433
CrashReporter Key:   bf1c73769925cbff86345a576ae1e576728e5a11
Hardware Model:      iPhone3,1
OS Version:          iPhone OS 5.1.1 (9B206)
Kernel Version:      Darwin Kernel Version 11.0.0: Sun Apr  8 21:51:26 PDT 2012; root:xnu-

1878.11.10~1/RELEASE_ARM_S5L8930X
Date:                2013-03-18 19:44:51 +0800
Time since snapshot: 38 ms

Free pages:        1209
Active pages:      3216
Inactive pages:    1766
Throttled pages:   106500
Purgeable pages:   0
Wired pages:       16245
Largest process:   Deja Dev

Processes
         Name                 UUID                    Count resident pages
            geod <976e1080853233b1856b13cbd81fdcc3>     338
        LinkedIn <24325ddfeed33d4fb643030edcb12548>    6666 (jettisoned)
    Music~iphone <a3a7a86202c93a6ebc65b8e149324218>     935
        WhatsApp <a24567991f613aaebf6837379bbf3904>    2509
      MobileMail <eed7992f4c1d3050a7fb5d04f1534030>     945
         Console <9925a5bd367a7697038ca5a581d6ebdf>     926 (jettisoned)
        Test Dev <c9b1db19bcf63a71a048031ed3e9a3f8>   81683 (active)
     MobilePhone <8f3f3e982d9235acbff1e33881b0eb13>     867
     debugserver <2408bf4540f63c55b656243d522df7b2>      92
        networkd <80ba40030462385085b5b7e47601d48d>     158
         notifyd <f6a9aa19d33c3962aad3a77571017958>     234
      aosnotifyd <8cf4ef51f0c635dc920be1d4ad81b322>     438
        BTServer <31e82dfa7ccd364fb8fcc650f6194790>     275
CommCenterClassi <041d4491826e3c6b911943eddf6aaac9>     722
     SpringBoard <c74dc89dec1c3392b3f7ac891869644a>    5062 (active)
      aggregated <a12fa71e6997362c83e0c23d8b4eb5b7>     383
            apsd <e7a29f2034083510b5439c0fb5de7ef1>     530
         configd <ee72b01d85c33a24b3548fa40fbe519c>     465
     dataaccessd <473ff40f3bfd3f71b5e3b4335b2011ee>     871
   fairplayd.N90 <ba38f6bb2c993377a221350ad32a419b>     169
       fseventsd <914b28fa8f8a362fabcc47294380c81c>     331
            iapd <0a747292a113307abb17216274976be5>     323
         imagent <9c3a4f75d1303349a53fc6555ea25cd7>     536
       locationd <cf31b0cddd2d3791a2bfcd6033c99045>    1197
   mDNSResponder <86ccd4633a6c3c7caf44f51ce4aca96d>     201
    mediaremoted <327f00bfc10b3820b4a74b9666b0c758>     257
    mediaserverd <f03b746f09293fd39a6079c135e7ed00>    1351
       lockdownd <b06de06b9f6939d3afc607b968841ab9>     279
          powerd <133b7397f5603cf8bef209d4172d6c39>     173
         syslogd <7153b590e0353520a19b74a14654eaaa>     178
           wifid <3001cd0a61fe357d95f170247e5458f5>     319
  UserEventAgent <dc32e6824fd33bf189b266102751314f>     409
         launchd <5fec01c378a030a8bd23062689abb07f>     126

**End**

On closer inspection, the dirty memory consists mostly of Image IO and Core Animation pages. multiple entries consisting of hundreds to thousands of pages. What does Image IO and Core Animation do exactly? and how can I reduce the Dirty Memory?

edit: tried doing this on a serial queue and no improvement on size of Dirty memory

another question. how large is too large for Dirty Memory and allocations?

Updated:

- (void) render
{
    for (id thing in mylist) {
        @autorelease {
            CALayer *layer = createLayerFromThing(thing);
            UIImage *img = [self renderLayer:layer];
            [self writeToDisk:img];
        }
    }
}

in createLayerFromThing(thing); I actually creating a layer with a huge amount of sub layers

UPDATED

first screenshot for maxConcurrentOperationCount = 4

second for maxConcurrentOperationCount = 1

============================================================================================================================================================

and since it bringing down the number of concurrent operations barely made a dent, I decided to try maxConcurrentOperationCount = 10

Answer 1

It's difficult to say what's going wrong without any details but here are a few ideas.

A. Use @autorelease. CALayers generate bitmaps in the backgound, which in aggregate can take-up lots of space if they are not freed in time. If you are creating and rendering many layers I suggest adding an autorelease block inside your rendering loop. This won't help if ALL your layers are nested and needed at the same time for rendering.

- (void) render
{
    for (id thing in mylist) {
        @autorelease {
            CALayer *layer = createLayerFromThing(thing);
            [self renderLayer:layer];
        }
    }
}

B. Also if you are using CGBitmapCreateContext for rendering are you calling the matching CGContextRelease? This goes also for CGColorRef.

C. If you are allocating memory with malloc or calloc are you freeing it when done? One way to ensure that this happens

Post the code for the rendering loop to provide more context.

Answer 2

I believe there are two possibilities here:

1. The items you create are not autoreleased.
2. The memory you are taking is what it is, due to the number of concurrent operations you are doing.

In the first case the solution is simple. Send an autorelease message to the layers and images upon their creation.

In the second case, you could limit the number of concurrent operations by using an NSOperationQueue. Operation queues have a property called maxConcurrentOperationCount. I would try with a value of 4 and see how the memory behavior changes from what you currently have. Of course, you might have to try different values to get the right memory vs performance balance.