Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to represent text for classification in weka?

Can you please let me know how to represent attribute or class for text classification in weka. By using what attribute can I do classification? word frequency or just word? What would be possible structure of ARFF format? Can you give me several lines of example of that structure?

Thank you very much in advance.

like image 520
Warren Avatar asked Nov 29 '11 15:11

Warren


People also ask

What is the best method for text classification?

Linear Support Vector Machine is widely regarded as one of the best text classification algorithms.

How does text classification work?

Text classification also known as text tagging or text categorization is the process of categorizing text into organized groups. By using Natural Language Processing (NLP), text classifiers can automatically analyze text and then assign a set of pre-defined tags or categories based on its content.


1 Answers

One of the easiest alternatives is to start with an ARFF file for a two class problem like:

@relation corpus 

@attribute text string
@attribute class {pos,neg}

@data
'long text with words ... ',pos

The text is represented as a String type and the class is a nominal with two values.

Then you could apply two filters:

  1. StringToWordVector that transforms the texts into a word vector representation. The filter uses an attribute for each word. You can tweak parameters to choose binary/frequency representation, stemming or stopwords. The best representation depends on the problem. If text are not long, usually binary representation is enough.
  2. Reorder to move the class atribute to the last position, Weka assumes it is there.

You may find more info and other approaches to transform your data in this Weka wiki page: http://weka.wikispaces.com/Text+categorization+with+WEKA

like image 77
zdepablo Avatar answered Oct 13 '22 01:10

zdepablo