Do any algorithms exist to weight various factors?

Question

Sorry in advance if I'm not explaining this correctly. I want to know if there are algorithms to weight various factors in a decision process.

I was reading programming collective intelligence and there's a chapter where you build a search engine and use various factors to rank webpages(pagerank, frequency of words, distance of words, words in title, etc..). Basically in their example code they have all the factors as functions in a class and then use this command to score them:

weights=[(1.0,self.locationscore(rows)), 
                 (1.0,self.frequencyscore(rows)), 
                 (1.0,self.distancescore(rows)),
                 (1.0,self.pagerankscore(rows)),
                 (1.0,self.linktextscore(rows, wordids))]  

Each factor is ranked equally(1.0), but I was wondering if there was a way to dynamically make different factors weight differently without manually setting the weights? In the book, they go on to use neural networks to study users clicks but the above weights remain the same.

I get a feeling there's a non-static way to do this, but not sure what. Any suggestions of how to approach this would be great.

Thanks in advance

Note: if you would like the example code from the book, its http://examples.oreilly.com/9780596529321/ and the chapter I'm referring to is chapter 4. Also if I'm not explaining anything correctly please let me know and I'll update my question.

Answer 1

First define your utility function: How to estimate if one solution in better then the other. One common utility for the problem you described is recall and precision and the F measure.

Also, create an evaluation set: Build a set of queries, and a set of expected answers for these queries.

Now, you can tune your weighting functions with any AI optimizing algorithm, such as hill climbing or genetic algorithms. In these optimization algorithms, your variables are the weights for each algorithm, and you try to optimize your utility functions.

Note: If your search engine API allows some explicit feedback, you might also use the same method to keep tuning on the fly: after enough users had indicated what is the "correct" answers for queries, you can add it to your evaluation set, and run your tuning algorithm again.