What should be considered when building a Recommendation Engine?

Question

I've read the book Programming Collective Intelligence and found it fascinating. I'd recently heard about a challenge amazon had posted to the world to come up with a better recommendation engine for their system.

The winner apparently produced the best algorithm by limiting the amount of information that was being fed to it.

As a first rule of thumb I guess... "More information is not necessarily better when it comes to fuzzy algorithms."

I know's it's subjective, but ultimately it's a measurable thing (clicks in response to recommendations).

Since most of us are dealing with the web these days and search can be considered a form of recommendation... I suspect I'm not the only one who'd appreciate other peoples ideas on this.

In a nutshell, "What is the best way to build a recommendation ?"

Answer 1

You don't want to use "overall popularity" unless you have no information about the user. Instead, you want to align this user with similar users and weight accordingly.

This is exactly what Bayesian Inference does. In English, it means adjusting the overall probability you'll like something (the average rating) with ratings from other people who generally vote your way as well.

Another piece of advice, but this time ad hoc: I find that there are people where if they like something I will almost assuredly not like it. I don't know if this effect is real or imagined, but it might be fun to build in a kind of "negative effect" instead of just clumping people by similarity.

Finally there's a company specializing in exactly this called SenseArray. The owner (Ian Clarke of freenet fame) is very approachable. You can use my name if you call him up.

Answer 2

There is an entire research area in computer science devoted to this subject. I'd suggest reading some articles.

Answer 3

Agree with @Ricardo. This question is too broad, like asking "What's the best way to optimize a system?"

One common feature to nearly all existing recommendation engines is that making the final recommendation boils down to multiplying some number of matrices and vectors. For example multiply a matrix containing proximity weights between users by a vector of item ratings.

(Of course you have to be ready for most of your vectors to be super sparse!)

My answer is surely too late for @Allain but for other users finding this question through search -- send me a PM and ask a more specific question and I will be sure to respond.

(I design recommendation engines professionally.)

Answer 4

@Lao Tzu, I agree with you.

According to me, recommendation engines are made up of:

• Context Input fed from context aware systems (logging all your data)
• Logical reasoning to filter the most obvious
• Expert systems that improve your subjective data over the period of time based on context inputs, and
• Probabilistic reasoning to do decision-making close-to-proximity based on weighted sum of previous actions(beliefs, desires, & intentions).

P.S. I made such recommendation engine.