How can I rank images by only comparing them to each other? [closed]

Question

I have a large collection of images which I'm trying to sort according to quality by crowd-sourcing. Images can be assigned 1, 2, 3, 4, or 5 stars according to how much the user likes them. A 5-star image would be very visually appealing, a 1-star image might be blurry and out of focus.

At first I created a page showing an image with the option to rate it directly by choosing 1-5 stars. But it was too time-consuming to to do this. I'd like to try to create an interface where the user is presented with 2 images side by side and asked to click the image s/he likes more. Using this comparison data of one image compared to another, is there then some way to convert it over to a score of 1-5?

What kind of algorithm would allow me to globally rank images by comparing them only to each other, and how could I implement it in python?

Answer 1

Sounds like you need a ranking algorithm similar to what is used in sport to rank players. Think of the comparison of two images as a match and the one the user selects as the better one is the winner of the match. After some time, many players have played many matches and sometimes against the same person. They win some they lose some eh? How do you rank which is the best overall?

You can look at the Elo Rating System. which is used in chess to rank chess players. There is an algorithm specified so it should be a matter of implementing in your language of choice.

Answer 2

Let each image start with a ranking of 3 (the mean of 1 … 5), then for each comparison (which wasn't equal) lower the rank of the loser image and increase the rank of the winner image. I propose to simply count the +1s and the -1s, so that you have a number of wins and a number of losses for each image.

Then the value 1 … 5 could be calculated as:

import math

def rank(wins, losses):
  return 3 + 4 * math.atan(wins - losses) / math.pi

This will rank images higher and higher with each win, but it will lead to the silly situation that (+1010 / -1000) will be ranked alike with (+10 / -0) which is not desirable.

One can remedy this flaw by using a mean of the values:

def rank(wins, losses):
  return (3 + 4 * math.atan((wins - losses) / (wins + losses) * 10) / math.pi
          if wins + losses > 0 else 3)

Both curves will never quite reach 1 or 5, but they will come ever closer if an image always wins or always loses.