Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Algorithm for text classification


I have millions of short (up to 30 words) documents which I need to split into several known categories. It's possible, that a document matches several of the categories (seldom, but possible). It's also possible that a document doesn't match any of the categories (also seldom). I also have millions of documents which have already been categorized. What algorithm should I use to do the job. I don't need to do it fast. I need to be sure that the algorithm categorizes correctly (as far as possible).
What algorithm should I use? Is there an implementation of in in C#?
Thank you for your help!

like image 689
StuffHappens Avatar asked Jan 22 '23 07:01

StuffHappens


2 Answers

Take a look at term frequency and inverse document frequency also cosine similarity to find important words to create categories and assign documents to categories based on similarity

EDIT:

Found an example here

like image 130
Jimmy Avatar answered Jan 31 '23 11:01

Jimmy


Interesting articles :

  • A self-organizing semantic map for information retrieval
  • WEBSOM - self-organizing maps of document collections
like image 41
Romain Meresse Avatar answered Jan 31 '23 10:01

Romain Meresse