Algorithm (or C# library) for identifying 'keywords' in a set of messages? [closed]

Question

I want to build a list of ~6 keywords (or even better: couple word keyphrases) for each message in a message forum.

• The primary use of keywords is to replace subject lines in some instances. For example: Message from Terry sent Dec 5, keywords: norweigan blue, plumage, not dead
• In a super ideal world keywords would identify both unique phases, and phrases that cluster the discussion into "topics", i.e. words that are highly relevant to the message in question, and a few other messages in the forum, but not found frequently in the forum as a whole.
• I expect junk phrases to show up, no big deal.
• Can't be too computationally expensive: I need something that can handle several hundred messages in several seconds, as I'll need to re-run this every time a new message comes in.

Anyone know a good C# library for accomplishing this? Maybe there's a way to bend Lucene.NET into providing this sort of info?

Or, failing that, can anyone suggest an algorithm (or set of algos) to read up on? If I'm implementing myself I need something not terribly complex, I can only tackle this if its tractable in about a week. Right now, the best I've found in terms of simple-but-effective is TF-IDF.

UPDATE: I've uploaded the results of using TF-IDF to select the top 5 keywords from a real dataset here: http://jsbin.com/oxanoc/2/edit#preview

The results are mediocre, but not totally useless... maybe with the addition of detecting multi-word phrases, this would be good enough.

Answer 1

I've implemented a keywords extraction algorithm in Java a few weeks ago for uni. project, and used the tf-idf model.

Algorithm:
First, we looked for all bigrams in the paragraph, and extracted the meaningful ones. (*)
Next, we took the set of unigrams and bigrams, and evaluated each with is respective tf-idf score. The idf score of each term was the "documents count" retrieved by Bing API.

(*) Deciding which bi-gram is meaningful:
We used a various heuristics to find which bi-gram can be considered meaningful. At the end, the best results were achieved by "asking" wikipedia: we searched for the bi-gram. If there is an article containing this bi-gram, we considered it meaningful.

Evaluation:
We evaluated the algorithm on a set of 50 abstracts from random articles, and extracted the precision and recall of these algorithms.
The result was ~40% recall and ~35% precision, which is not too bad.