How to compare the "quality" of two image scaling algorithms?

Question

Suppose I want to include an image upscaling/downscaling algorithm in my program. Execution time is not important, the result "quality" is. How do you define "quality" in this case and how do you then choose the best algorithm from the available choices?

---

On a side note, if this is too general, the underlying problem for which I'm trying to find a solution is this: suppose I have a lot of images that I will need to upscale at runtime (a video, actually). I can pre-process them and upscale them somewhat with a slow and high-quality algorithm, but I don't know the final resolution (well, people have different monitors after all), so I can't resize to that immediately. Would it be beneficial if I upscaled it somewhat with my high-quality algorithm, and then let the player upscale it further to the necessary resolution at runtime (with a fast but low quality algorithm)? Or should I leave the video as-is and leave all the upscaling to be done in one pass at runtime?

Answer 1

The only way to really objectively judge the quality is to do some (semi-)scientific research. Recruit several participants. Show them the upscaled images in a random order, and have them rank the subjective quality (bonus points for doing it double-blind). Then you average out the scores and choose the algorithm with the highest average score (and perhaps test for statistical significance).

You'll want to make sure the images you test give a representative sampling of the actual images you're using. If you're doing it for video, it would probably be a good idea to use short video clips as the test images, instead of stills, as I would suspect that people would perceive the upscaling quality differently for those two.

If you don't care about rigorousness, you could just perform the tests with yourself as the only test subject.

As for doing an initial prescaling, my guess is that it would not be worth it. Scaling up from a larger image shouldn't be any less expensive than scaling up from the smaller original, and I would expect it to me much more expensive than scaling up by a convinient factor, such as 2x. However, don't take my word for it... test!