Algorithm for Finding Good, Reliable Players

Question

I've the following players, each value corresponds to a result in percentage of right answers in a given game.

$players = array
(
    'A' => array(0, 0, 0, 0),
    'B' => array(50, 50, 0, 0),
    'C' => array(50, 50, 50, 50),
    'D' => array(75, 90, 100, 25),
    'E' => array(50, 50, 50, 50),
    'F' => array(100, 100, 0, 0),
    'G' => array(100, 100, 100, 100),
);

I want to be able to pick up the best players but I also want to take into account how reliable a player is (less entropy = more reliable), so far I've come up with the following formula:

average - standard_deviation / 2

However I'm not sure if this is a optimal formula and I would like to hear your thoughts on this. I've been thinking some more on this problem and I've come up with a slightly different formula, here it is the revised version:

average - standard_deviation / # of bets

This result would then be weighted for the next upcoming vote, so for instance a new bet from player C would only count as half a bet.

I can't go into specifics here but this is a project related with the Wisdom of Crowds theory and the Delphi method and my goal is to predict as best as possible the next results weighting past bets from several players.

I appreciate all input, thanks.

Answer 1

You can't get an optimal formula if you haven't quantified what is better. You need to figure out how do you want to weigh consistency against average. For example one option would be to estimate the score that the player will hit a given percentage of games. This requires some kind of model of the probability distribution of the players score. For instance, if we assume that the players scores follow the normal distribution, then your given formula calculates what score the player will surpass about 70% of the time.

Answer 2

Would a Bayesian Probablity Formula fit the bill?

I think it would. Here is a link to another site that is a little less mathematical about it: http://www.experiment-resources.com/bayesian-probability.html

Essentially you are predicting the probability that each player will score the highest in the next round. This is what bayesian probabilities eat for breakfast.

Bayesian probabilities are already in use in video games (warning: .doc file) to determine stuff just like this.