How to handle memory constraints in iOS 8 Photo Extensions?

Question

I added a new iOS 8 Photo Extension to my existing photo editing app. My app has quite a complex filter pipeline and needs to keep multiple textures in memory at a time. However, on devices with 1 GB RAM I'm easily able to process 8 MP images.

In the extension, however, there are much higher memory constraints. I had to scale down the image to under 2 MP in order to get it processed without crashing the extension. I also figured that the memory problems only occurred when not having a debugger attached to the extension. With it, everything works fine.

I did some experiments. I modified a memory budget test app to work within an extension and came up with the following results (showing the amount of RAM in MB that can be allocated before crashing):

╔═══════════════════════╦═════╦═══════════╦══════════════════╗
║        Device         ║ App ║ Extension ║ Ext. (+Debugger) ║
╠═══════════════════════╬═════╬═══════════╬══════════════════╣
║ iPhone 6 Plus (8.0.2) ║ 646 ║       115 ║              645 ║
║ iPhone 5 (8.1 beta 2) ║ 647 ║        97 ║              646 ║
║ iPhone 4s (8.0.2)     ║ 305 ║        97 ║              246 ║
╚═══════════════════════╩═════╩═══════════╩══════════════════╝

A few observations:

• With the debugger attached the extension behaves like the "normal" app
• Even though the 4s has only half the total amount of memory (512 MB) compared to the other devices it gets the same ~100 MB from the system for the extension.

Now my question: How am I supposed to work with this small amount of memory in a Photo Editing extension? One texture containing an 8 MP (camera resolution) RGBA image eats ~31 MB alone. What is the point of this extension mechanism if I have to tell the user that full size editing is only possible when using the main app?

Did one of you also reach that barrier? Did you find a solution to circumvent this constraint?

Answer 1

I am developing a Photo Editing extension for my company, and we are facing the same issue. Our internal image processing engine needs more than 150mb to apply certain effects to an image. And this is not even counting panorama images which will take around ~100mb of memory per copy.

We found only two workarounds, but not an actual solution.

1. Scaling down the image, and applying the filter. This will require way less memory, but the image result is terrible. At least the extension will not crash.

or

2. Use CoreImage or Metal for image processing. As we analyzed the Sample Photo Editing Extension from Apple, which uses CoreImage, can handle very large image and even panoramas without quality or resolution loss. Actually, we were not able to crash the extension by loading very large images. The sample code can handle panoramas with a memory peek of 40mb, which is pretty impressive.

According to the Apple's App Extension Programming Guide, page 55, chapter "Handling Memory Constraints", the solution for memory pressure in extensions is to review your image-processing code. So far we are porting our image processing engine to CoreImage, and the results are far better than our previous engine.

I hope I could help a bit. Marco Paiva