How do I upscale an image in ImageIO iOS framework?

Question

I went through the usual process of progressively discovering the best performing Image resizing solution in iOS and I ended up with ImageIO.

The following works well

if let currentImage = CGImageSourceCreateWithData(imageData, nil)

   //...some code that extracts the existing size, and based on my maximum expected horizontal size..calculates a new horizontal/vertical maximum size..

let options: CFDictionary = [kCGImageSourceThumbnailMaxPixelSize as NSString            : maximumImageWidthOrHeight,
                             kCGImageSourceCreateThumbnailFromImageAlways as NSString   : true]

CGImageSourceCreateThumbnailAtIndex(image, options)

but only for down-sampling.

If I have an image that is full HD ie. 1920 x 1080, it is resized (with aspect ratio kept) so that it fits into 320 px horizontaly, because this is what I set to be the maximum horizontal size.

However if I pass in an image that is 16 pixels x 9 pixels.. a really small one (and having a 16:9 aspect ratio obviously). This image is not resized.

I want to it to be resized to 320 x 180 pixels.

Is there a combination of options/ a missed method call / anything that will give me that in the imageIO framework?

Answer 1

UpSampling Images, a few notes:

(unfortunatley, this is going to be written in Objective-C, in part due to the complexity of ImageMagick itself, it's written in C code, and that's that, there's no ObjC or Swift)

ImageMagick

Possibly one of the greatest image mod libraries available is ImageMagick, this library has been build by quite a few folks, one in particular of note is Nicolas Robidoux, who currently works as a Senior Research Scientist at Phase One. Phase One is:

Phase One is the world’s leader in open-platform, high-end medium format camera systems and solutions. Our products are known for their quality, flexibility and speed enabling pro photographers to shoot in a wide range of formats.

Nicholas has a PHD in applied mathematics from The University of New Mexico, and has worked on ImageMagick for quite some time. Here's what Nicholas has to say about upsampling images:

http://www.imagemagick.org/Usage/filter/nicolas/#upsampling

Recommended Upsampling Methods (Enlarging) Results of resizing using the LanczosSharp EWA Filter through a Sigmoidized Colorspace are fairly artefact free, if a bit blurry. The built-in LanczosSharp, discussed in the Short Answer, works well, but I prefer a version with slightly less de-blur, obtained with the "-filter Lanczos -define filter:blur=.9891028367558475".

convert {input} -colorspace RGB +sigmoidal-contrast 7.5 \ -filter Lanczos -define filter:blur=.9891028367558475 \ -distort Resize 500% \ -sigmoidal-contrast 7.5 -colorspace sRGB {output}

You would get nearly identical results with the built-in LanczosSharp filter, which is obtained by replacing "-filter Lanczos -define filter:blur=.9891028367558475" by "-filter LanczosSharp".

Wow, that's some great stuff, what else does Nicholas tell us:

http://www.imagemagick.org/Usage/filter/nicolas/

When resizing, I (Nicolas Robidoux) generally stick to either a tensor (orthogonal, or 2-pass) Resize (-resize) Operator filter, or an EWA (Elliptical Weighted Averaging) Distort Resize (-distort Resize) operator. Although the filter kernels used with tensor and EWA operations are built using the same families of mathematical functions, they give very different results when used with -resize and when used with -distort Resize.

If you want to use one single filter for everything and would rather avoid complications like color space manipulations and fancy parameters, use the LanczosSharp EWA (Elliptical Weighted Averaging) Filter. It produces images that are slightly blurry, with no jaggies to speak of and reasonably mild halos.

That sounds semi complex , so let's see what we can ACTUALLY do EASILY in IOS given some already available tools that are kept up-to-date and are super IOS friendly.

How do we do this in IOS?, well, there's an ImageMagick POD if you want to give it a try, the unfortunate part is that it's sort of difficult to work with. And, there's also Brad Larson's GPU Image, (thanks Brad!, great library). So, for the sake of time, I'm going to stick with Brad's GPUImage processing library to give you a quick taste of "upsampling" and what you can expect. I'm going to use the Shapren and Lanczos filters as Nicholas suggests, unfortunately, to my knowledge it looks like GPUImage doesn't have an "tensor" or "LanczosSharp EWA" filter specifically, feel free to correct me if I'm wrong. Well, with that said, here we go:

I'm going to be working with this image, it's a simple "book" icon:

Placing this image in code reveals the following, this is the image, untouched, extracted by XCode using a vectorized PDF of the "book icon" above, here's the results:

NO UPSAMPLING:

UIImage *inputImage = [UIImage imageNamed:@"book"];
cell.imageView.image = inputImage;

LANCZOS UPSAMPLING:

UIImage *inputImage = [UIImage imageNamed:@"book"];
GPUImagePicture *stillImageSource = [[GPUImagePicture alloc] initWithImage:inputImage];
GPUImageLanczosResamplingFilter *stillImageFilter = [[GPUImageLanczosResamplingFilter alloc] init];
[stillImageSource addTarget:stillImageFilter];
[stillImageFilter useNextFrameForImageCapture];
[stillImageSource processImage];
[stillImageSource forceProcessingAtSizeRespectingAspectRatio:CGSizeMake(200, 200)];
UIImage *currentFilteredVideoFrame = [stillImageFilter imageFromCurrentFramebuffer];
cell.imageView.image = currentFilteredVideoFrame;

SHARPEN UPSAMPLING:

UIImage *inputImage = [UIImage imageNamed:@"book"];
GPUImagePicture *stillImageSource = [[GPUImagePicture alloc] initWithImage:inputImage];
GPUImageSharpenFilter *stillImageFilter = [[GPUImageSharpenFilter alloc] init];
[stillImageSource addTarget:stillImageFilter];
[stillImageFilter useNextFrameForImageCapture];
[stillImageSource processImage];
UIImage *currentFilteredVideoFrame = [stillImageFilter imageFromCurrentFramebuffer];

So, this gives you a quick showdown of what's what concerning upsampling, obviously, using the ImageMagick library may or may not produce better results, unfortunatley, I don't have the time to dive head first into getting the ImageMagick library up and running, but the stuff I've included in the post will get you well on your way to some sort of solution, albeit more than likely a semi-blurry solution, but nonetheless, the best solution that humankind can put together given our computing technology and current mathematics. Perhaps there's more info that is more recent that what I've listed here, but for all intents and purposes, I feel that the info I've linked is going to give you your best shot at getting to where you want to be the quickest, unless of course you have time to write your own image library.

Anyway, if you'd like to try to implement Nicholas' ImageMagick techniques in your app, feel free to give it a shot by installing the following POD:

pod 'ImageMagick', '6.8.8-9'

For Brad Larson's GPUImage the POD is:

pod 'GPUImage'

Should you have questions on how to make the IOS ImageMagick library work:

http://blog.splitwise.com/2012/12/18/imagemagick-for-ios-armv7-and-armv7s/

https://github.com/marforic/imagemagick_lib_iphone

http://www.cloudgoessocial.net/2011/03/21/im-xcode4-ios4-3/

If you want to take a look at what Brad Larson has put together, here's the github repo:

https://github.com/BradLarson/GPUImage

To expand on my answer, I decided to jump into ImageMagick head first, here's the results, these results are based on a PNG file this time, not a vectorized PDF file:

first and foremost, make sure you import this header:

#import <wand/MagickWand.h>

and here's your property declaration

@property (nonatomic) MagickWand *wand;

And, you have to do the old, "viewDidUnlaod" stuffs:

-(void)viewDidUnload
{
    if (self.wand)
        self.wand = DestroyMagickWand(self.wand);
    MagickWandTerminus();
}

Lanczos2SharpFilter

MagickWandGenesis();
self.wand = NewMagickWand();
NSString *filePath = [[NSBundle mainBundle] pathForResource:@"book" ofType:@"png"];
MagickReadImage(self.wand, [filePath cStringUsingEncoding:NSASCIIStringEncoding]);
MagickResampleImage(self.wand,100,100,Lanczos2SharpFilter, 2),
MagickWriteImage(self.wand, [filePath cStringUsingEncoding:NSASCIIStringEncoding]);
UIImage *imgObj = [UIImage imageWithContentsOfFile:filePath];
cell.imageView.image = imgObj;

RobidouxSharpFilter this is Nicholas' filter

MagickWandGenesis();
self.wand = NewMagickWand();
NSString *filePath = [[NSBundle mainBundle] pathForResource:@"book" ofType:@"png"];
MagickReadImage(self.wand, [filePath cStringUsingEncoding:NSASCIIStringEncoding]);
MagickResampleImage(self.wand,75,75,RobidouxSharpFilter, 1),
MagickWriteImage(self.wand, [filePath cStringUsingEncoding:NSASCIIStringEncoding]);
UIImage *imgObj = [UIImage imageWithContentsOfFile:filePath];
cell.imageView.image = imgObj;

LanczosSharpFilter

MagickWandGenesis();
self.wand = NewMagickWand();
NSString *filePath = [[NSBundle mainBundle] pathForResource:@"book" ofType:@"png"];
MagickReadImage(self.wand, [filePath cStringUsingEncoding:NSASCIIStringEncoding]);
MagickResampleImage(self.wand,75,75,LanczosSharpFilter, 1),
MagickWriteImage(self.wand, [filePath cStringUsingEncoding:NSASCIIStringEncoding]);
UIImage *imgObj = [UIImage imageWithContentsOfFile:filePath];
cell.imageView.image = imgObj;

I could obviously keep going down this road, but I'll now leave it up to you to decide. Good luck, here's a list of all the filters that you can "EASILY" use with ImageMagick;

UndefinedFilter, PointFilter, BoxFilter, TriangleFilter,
HermiteFilter, HanningFilter, HammingFilter, BlackmanFilter,
GaussianFilter, QuadraticFilter, CubicFilter, CatromFilter,
MitchellFilter, JincFilter, SincFilter, SincFastFilter,
KaiserFilter, WelshFilter, ParzenFilter, BohmanFilter,
BartlettFilter, LagrangeFilter, LanczosFilter,
LanczosSharpFilter, Lanczos2Filter, Lanczos2SharpFilter,
RobidouxFilter, RobidouxSharpFilter, CosineFilter, SplineFilter, LanczosRadiusFilter, SentinelFilter

Answer 2

The short answer would be that you can't do that using ImageIO.

ImageIO is designed to load images progressively (as in web images), build fast thumbnails (which max size will be the one of the original image), and build groups of cached or not images. Here's the manual

The easiest way to go is using UIKit, and would be using scales ratios in the range (0 < ratio < 1).

for example, to go from a 16x9 image to a 230x180 will be:

 let image = UIImage(data: yourImageData!, scale: 0.050)!

There are other ways, as depicted in NSHipster that are just a bit more complicated but may worth to take a look into, and make some performance test to pick which one is better upsampling. But that's up to you to test.

Either way, the results are going to be pretty bad, as upsampling is still under research and is a really difficult task.

There are several papers on this topic, but they are usually related to upsampling an specific kind of objects and this allows the algorithm to make some needed assumptions, for example: Hallucinating Faces: TensorPatch Super-Resolution and Coupled Residue Compensation

If I were to choose a way to achieve this and quality were my goal, I would go for OpenCV algorithms on superresolution.

But, they are iterative algorithms, so maybe don't fit your performance needs.

Here's a demostration of how it works with the popular image "Lena"

So, you may need to rethink how important is this to your app, and maybe find a workaround, as there will be no way to achieve both, quality and close to real time performance right now. ;-)

Happy Coding!