How to efficiently create a multi-row photo collage from an array of images in Swift

Question

Problem

I am building a collage of photos from an array of images that I am placing onto a tableview. I want to make the images wrap when the number of images reaches the boundary of the tableview cell's width (this would allow me to display rows of images in the collage). Currently I get a single row. Please feel free to advise if additional information is required. I am most likely not approaching this in the most efficient way since there is a delay as the number of images used in the array begins to increase. (any feedback on this would be very much appreciated).

Nota Bene

I am creating a collage image. It is actually one image. I want to arrange the collage by creating an efficent matrix of columns and rows in memory. I then fill these rects with images. Finally I snapshot the resulting image and use it when needed. The algorithm is not efficient as written and produces only a single row of images. I need a lightweight alternative to the algorithm used below. I do not believe UICollectionView will be a useful alternative in this case.

Pseudo Code

1. Given an array of images and a target rectangle (representing the target view)
2. Get the number of images in the array compared to max number allowed per row
3. Define a smaller rectangle of appropriate size to hold the image (so that each row fills the target rectangle, i.e. - if one image then that should fill the row; if 9 images then that should fill the row completely; if 10 images with a max of 9 images per row then the 10th begins the second row)
4. Iterate over the collection
5. Place each rectangle at the correct location from left to right until either last image or a max number per row is reached; continue on next row until all images fit within the target rectangle
6. When reaching a max number of images per row, place the image and setup the next rectangle to appear on the successive row

Using: Swift 2.0

class func collageImage (rect:CGRect, images:[UIImage]) -> UIImage {

        let maxSide = max(rect.width / CGFloat(images.count), rect.height / CGFloat(images.count))

        UIGraphicsBeginImageContextWithOptions(rect.size, false, UIScreen.mainScreen().scale)

        var xtransform:CGFloat = 0.0

        for img in images {
            let smallRect:CGRect = CGRectMake(xtransform, 0.0,maxSide, maxSide)
            let rnd = arc4random_uniform(270) + 15
            //draw in rect
            img.drawInRect(smallRect)
            //rotate img using random angle.
            UIImage.rotateImage(img, radian: CGFloat(rnd))
            xtransform += CGFloat(maxSide * 0.8)
        }

        let outputImage = UIGraphicsGetImageFromCurrentImageContext();

        UIGraphicsEndImageContext();

        return outputImage
    }

    class func rotateImage(src: UIImage, radian:CGFloat) -> UIImage
    {
        //  Calculate the size of the rotated view's containing box for our drawing space
        let rotatedViewBox = UIView(frame: CGRectMake(0,0, src.size.width, src.size.height))

        let t: CGAffineTransform  = CGAffineTransformMakeRotation(radian)

        rotatedViewBox.transform = t
        let rotatedSize = rotatedViewBox.frame.size

        //  Create the bitmap context
        UIGraphicsBeginImageContext(rotatedSize)

        let bitmap:CGContextRef = UIGraphicsGetCurrentContext()

        //  Move the origin to the middle of the image so we will rotate and scale around the center.
        CGContextTranslateCTM(bitmap, rotatedSize.width/2, rotatedSize.height/2);

        //  Rotate the image context
        CGContextRotateCTM(bitmap, radian);

        //  Now, draw the rotated/scaled image into the context
        CGContextScaleCTM(bitmap, 1.0, -1.0);
        CGContextDrawImage(bitmap, CGRectMake(-src.size.width / 2, -src.size.height / 2, src.size.width, src.size.height), src.CGImage)

        let newImage = UIGraphicsGetImageFromCurrentImageContext()
        UIGraphicsEndImageContext()

        return newImage
    }

Alternative 1

I've refined my solution to this a bit. This one does stack the images in columns and rows however, as stated; my interest is in making this as efficient as possible. What's presented is my attempt at producing the simplest possible thing that works.

Caveat

The image produced using this is skewed rather than evenly distributed across the entire tableview cell. Efficient, even distribution across the tableview cell would be optimal.

class func collageImage (rect:CGRect, images:[UIImage]) -> UIImage {

        let maxSide = max(rect.width / CGFloat(images.count), rect.height / CGFloat(images.count)) //* 0.80
        //let rowHeight = rect.height / CGFloat(images.count) * 0.8
        let maxImagesPerRow = 9
        var index = 0
        var currentRow = 1
        var xtransform:CGFloat = 0.0
        var ytransform:CGFloat = 0.0
        var smallRect:CGRect = CGRectZero

        UIGraphicsBeginImageContextWithOptions(rect.size, false, UIScreen.mainScreen().scale)

        for img in images {

            let x = ++index % maxImagesPerRow //row should change when modulus is 0

            //row changes when modulus of counter returns zero @ maxImagesPerRow
            if x == 0 {
                //last column of current row
                //xtransform += CGFloat(maxSide)
                smallRect = CGRectMake(xtransform, ytransform, maxSide, maxSide)

                //reset for new row
                ++currentRow
                xtransform = 0.0
                ytransform = (maxSide * CGFloat(currentRow - 1))

            } else {
                //not a new row
                if xtransform == 0 {
                    //this is first column
                    //draw rect at 0,ytransform
                    smallRect = CGRectMake(xtransform, ytransform, maxSide, maxSide)
                    xtransform += CGFloat(maxSide)
                } else {
                    //not the first column so translate x, ytransform to be reset for new rows only
                    smallRect = CGRectMake(xtransform, ytransform, maxSide, maxSide)
                    xtransform += CGFloat(maxSide)
                }

            }

            //draw in rect
            img.drawInRect(smallRect)

        }

        let outputImage = UIGraphicsGetImageFromCurrentImageContext();

        UIGraphicsEndImageContext();

        return outputImage
    }

Alternative 2

The alternative presented below scales the images so that they always fill the rectangle (in my case the tableview cell). As more images are added they are scaled to fit the width of the rectangle. When the images meet the maximum number of images per row, they wrap. This is the desired behavior, happens in memory, is relatively fast, and is contained in a simple class function that I extend on the UIImage class. I am still interested in any algorithm that can deliver the same functionality only faster.

Nota Bene: I do not believe adding more UI is useful to achieve the effects as noted above. Therefore a more efficient coding algorithm is what I am seeking.

class func collageImage (rect:CGRect, images:[UIImage]) -> UIImage {

        let maxImagesPerRow = 9
        var maxSide : CGFloat = 0.0

        if images.count >= maxImagesPerRow {
            maxSide = max(rect.width / CGFloat(maxImagesPerRow), rect.height / CGFloat(maxImagesPerRow))
        } else {
            maxSide = max(rect.width / CGFloat(images.count), rect.height / CGFloat(images.count))
        }

        var index = 0
        var currentRow = 1
        var xtransform:CGFloat = 0.0
        var ytransform:CGFloat = 0.0
        var smallRect:CGRect = CGRectZero

        UIGraphicsBeginImageContextWithOptions(rect.size, false,  UIScreen.mainScreen().scale)

        for img in images {

            let x = ++index % maxImagesPerRow //row should change when modulus is 0

            //row changes when modulus of counter returns zero @ maxImagesPerRow
            if x == 0 {
                //last column of current row
                smallRect = CGRectMake(xtransform, ytransform, maxSide, maxSide)

                //reset for new row
                ++currentRow
                xtransform = 0.0
                ytransform = (maxSide * CGFloat(currentRow - 1))

            } else {
                //not a new row
                smallRect = CGRectMake(xtransform, ytransform, maxSide, maxSide)
                xtransform += CGFloat(maxSide)  
            }

            //draw in rect
            img.drawInRect(smallRect)

        }

        let outputImage = UIGraphicsGetImageFromCurrentImageContext();

        UIGraphicsEndImageContext();

        return outputImage
    }

Efficiency Testing

Reda Lemeden gives some procedural insight into how to test these CG calls within Instruments on this blog post. He also points out some interesting notes from Andy Matuschak (of the UIKit team) about some of the peculiarities of off-screen rendering. I am probably still not leveraging the CIImage solution properly because initial results show the solution getting slower when attempting to force GPU utilization.

Answer 1

Building on alternative 2 provided by Tommie C above I created a function that

• Always fills the total rectangle, without spaces in the collage
• determines the number of rows and columns automatically (maximum 1 more nrOfColumns than nrOfRows)
• To prevent the spaces mentioned above all individual pics are drawn with "Aspect Fill" (so for some pics this means that parts will be cropped)

Here's the function:

func collageImage(rect: CGRect, images: [UIImage]) -> UIImage {
    if images.count == 1 {
        return images[0]
    }

    UIGraphicsBeginImageContextWithOptions(rect.size, false,  UIScreen.mainScreen().scale)

    let nrofColumns: Int = max(2, Int(sqrt(Double(images.count-1)))+1)
    let nrOfRows: Int = (images.count)/nrofColumns
    let remaindingPics: Int = (images.count) % nrofColumns
    print("columns: \(nrofColumns) rows: \(nrOfRows) first \(remaindingPics) columns will have 1 pic extra")

    let w: CGFloat = rect.width/CGFloat(nrofColumns)
    var hForColumn = [CGFloat]()
    for var c=1;c<=nrofColumns;++c {
        if remaindingPics >= c {
            hForColumn.append(rect.height/CGFloat(nrOfRows+1))
        }
        else {
            hForColumn.append(rect.height/CGFloat(nrOfRows))
        }
    }
    var colNr = 0
    var rowNr = 0
    for var i=1; i<images.count; ++i {
        images[i].drawInRectAspectFill(CGRectMake(CGFloat(colNr)*w,CGFloat(rowNr)*hForColumn[colNr],w,hForColumn[colNr]))
        if i == nrofColumns || ((i % nrofColumns) == 0 && i > nrofColumns) {
            ++rowNr
            colNr = 0
        }
        else {
            ++colNr
        }
    }

    let outputImage = UIGraphicsGetImageFromCurrentImageContext();
    UIGraphicsEndImageContext();
    return outputImage
}

This uses the UIImage extension drawInRectAspectFill:

extension UIImage {
    func drawInRectAspectFill(rect: CGRect, opacity: CGFloat = 1.0) {
        let targetSize = rect.size
        let scaledImage: UIImage
        if targetSize == CGSizeZero {
            scaledImage = self
        } else {
            let scalingFactor = targetSize.width / self.size.width > targetSize.height / self.size.height ? targetSize.width / self.size.width : targetSize.height / self.size.height
            let newSize = CGSize(width: self.size.width * scalingFactor, height: self.size.height * scalingFactor)
            UIGraphicsBeginImageContext(targetSize)
            self.drawInRect(CGRect(origin: CGPoint(x: (targetSize.width - newSize.width) / 2, y: (targetSize.height - newSize.height) / 2), size: newSize), blendMode: CGBlendMode.Normal, alpha: opacity)
            scaledImage = UIGraphicsGetImageFromCurrentImageContext()
            UIGraphicsEndImageContext()
        }
        scaledImage.drawInRect(rect)
    }
}

Answer 2

To build the collage in memory, and to be as efficient as possible, I'd suggest looking into Core Image. You can combine multiple CIFilters to create your output image.

You could apply CIAffineTransform filters to each of your images to line them up (cropping them to size with CICrop if necessary), then combine them using CISourceOverCompositing filters. Core Image doesn't process anything until you ask for the output; and because it's all happening in the GPU, it's fast and efficient.

Here's a bit of code. I tried to keep it as close to your example as possible for the sake of understanding. It's not necessarily how I'd structure the code were I to use core image from scratch.

class func collageImage (rect: CGRect, images: [UIImage]) -> UIImage {

    let maxImagesPerRow = 3
    var maxSide : CGFloat = 0.0

    if images.count >= maxImagesPerRow {
        maxSide = max(rect.width / CGFloat(maxImagesPerRow), rect.height / CGFloat(maxImagesPerRow))
    } else {
        maxSide = max(rect.width / CGFloat(images.count), rect.height / CGFloat(images.count))
    }

    var index = 0
    var currentRow = 1
    var xtransform:CGFloat = 0.0
    var ytransform:CGFloat = 0.0
    var smallRect:CGRect = CGRectZero

    var composite: CIImage? // used to hold the composite of the images

    for img in images {

        let x = ++index % maxImagesPerRow //row should change when modulus is 0

        //row changes when modulus of counter returns zero @ maxImagesPerRow
        if x == 0 {

            //last column of current row
            smallRect = CGRectMake(xtransform, ytransform, maxSide, maxSide)

            //reset for new row
            ++currentRow
            xtransform = 0.0
            ytransform = (maxSide * CGFloat(currentRow - 1))

        } else {

            //not a new row
            smallRect = CGRectMake(xtransform, ytransform, maxSide, maxSide)
            xtransform += CGFloat(maxSide)
        }

        // Note, this section could be done with a single transform and perhaps increase the
        // efficiency a bit, but I wanted it to be explicit.
        //
        // It will also use the CI coordinate system which is bottom up, so you can translate
        // if the order of your collage matters.
        //
        // Also, note that this happens on the GPU, and these translation steps don't happen
        // as they are called... they happen at once when the image is rendered. CIImage can 
        // be thought of as a recipe for the final image.
        //
        // Finally, you an use core image filters for this and perhaps make it more efficient.
        // This version relies on the convenience methods for applying transforms, etc, but 
        // under the hood they use CIFilters
        var ci = CIImage(image: img)!

        ci = ci.imageByApplyingTransform(CGAffineTransformMakeScale(maxSide / img.size.width, maxSide / img.size.height))
        ci = ci.imageByApplyingTransform(CGAffineTransformMakeTranslation(smallRect.origin.x, smallRect.origin.y))!

        if composite == nil {

            composite = ci

        } else {

            composite = ci.imageByCompositingOverImage(composite!)
        }
    }

    let cgIntermediate = CIContext(options: nil).createCGImage(composite!, fromRect: composite!.extent())
    let finalRenderedComposite = UIImage(CGImage: cgIntermediate)!

    return finalRenderedComposite
}

You may find that your CIImage is rotated incorrectly. You can correct it with code like the following:

var transform = CGAffineTransformIdentity

switch ci.imageOrientation {

case UIImageOrientation.Up:
    fallthrough
case UIImageOrientation.UpMirrored:
    println("no translation necessary. I am ignoring the mirrored cases because in my code that is ok.")
case UIImageOrientation.Down:
    fallthrough
case UIImageOrientation.DownMirrored:
    transform = CGAffineTransformTranslate(transform, ci.size.width, ci.size.height)
    transform = CGAffineTransformRotate(transform, CGFloat(M_PI))
case UIImageOrientation.Left:
    fallthrough
case UIImageOrientation.LeftMirrored:
    transform = CGAffineTransformTranslate(transform, ci.size.width, 0)
    transform = CGAffineTransformRotate(transform, CGFloat(M_PI_2))
case UIImageOrientation.Right:
    fallthrough
case UIImageOrientation.RightMirrored:
    transform = CGAffineTransformTranslate(transform, 0, ci.size.height)
    transform = CGAffineTransformRotate(transform, CGFloat(-M_PI_2))
}

ci = ci.imageByApplyingTransform(transform)

Note that this code ignores fixing several mirrored cases. I'll leave that as an exercise up to you, but the gist of it is here.

If you've optimized your Core Image processing, then at this point any slowdown you see is probably due to transforming your CIImage into a UIImage; that's because your image has to make the transition from the GPU to the CPU. If you want to skip this step in order to display the results to the user, you can. Simply render your results to a GLKView directly. You can always transition to a UIImage or CGImage at the point the user wants to save the collage.

// this would happen during setup
let eaglContext = EAGLContext(API: .OpenGLES2)
glView.context = eaglContext

let ciContext = CIContext(EAGLContext: glView.context)

// this would happen whenever you want to put your CIImage on screen
if glView.context != EAGLContext.currentContext() {
    EAGLContext.setCurrentContext(glView.context)
}

let result = ViewController.collageImage(glView.bounds, images: images)

glView.bindDrawable()
ciContext.drawImage(result, inRect: glView.bounds, fromRect: result.extent())
glView.display()

Answer 3

Since you're using Swift 2.0: UIStackView does exactly what you're trying to do manually, and is significantly easier to use than UICollectionView. Assuming you're using Storyboards, creating a prototype TableViewCell with multiple nested UIStackViews should do exactly what you want. You'll just need to make sure that the UIImages you're inserting are all the same aspect ratio if that's what you want.

Your algorithm is highly inefficient because it has to re-draw, with multiple Core Animation transforms, every image any time you add or remove an image from your array. UIStackView supports dynamically adding and removing objects.

If you still, for some reason, need to snapshot the resulting collage as a UIImage, you can still do this on the UIStackView.