Artificial Intelligence, Text Classifier [closed]

Question

I am new to AI. I am working an application that text classification via machine learning. The application needs to classify different parts of an HTML document. For example, most webpages have head, menu, sidebar, footer, main content, etc. I want to use a text classifier to classify these parts of an HTML document, and to identify different type of forms on the page.

1. It would be very helpful if anyone could provide detailed guidance on this subject.
2. Examples of similar application, would also be very helpful.

I am looking for more technical suggestions, relating to code & implementation.

I can assign labels to html tag attributes, like class or id

<div class="menu-1">
<div id="entry">
<div id="content">
<div id="footer">
<div id="comment-12">
<div id="comment-title">

like for first item:

TrainClassifier(label: "Menu", value: "menu-1", attribute: "class", position-in-string: "21%", tag: "div");

Inputs:

1. "menu-1" (attribute value)
2. List item
3. "class" (attribute name)
4. "21" (tag position in string)
5. "div" (tag name)

Output

1. "Menu" (classified as label)

What neural network library, can take the above inputs, and classify them in to labels (i.e. Menu).

All users cannot create regex, or xpath, they need more easy approach, so it is important, to make the software intelligent, user can highlight the part of html document he/she needs, using webbrowser control, and train the software till it can work on its own.

but I dont know how to make the software train using AI,

the AI I am looking for is, like it should be able to accept various inputs, and classify on the basis of that, as I have already said new to AI, don't know much about it.

It would be helpful to me if I get answer to the question I have asked, like what library I should use, and how to implement, answers suggesting Xpath or Regex or other methods pls don't answer, it often happens that you get all suggestions but the one you need.

Answer 1

I suggest you to look into simpler algorithms first which are easy to understand, I can give pointers to some.

1. Naive Bayes (you will find many implementations but you can do it yourself, the algo is simple to implement yet quite powerful).
2. Maximum Entropy (Eg. SharpMaxEnt - open source).

3. SVM (Eg. LibSVM for C# port).

   If you want to get a taste of how these work, download the WEKA toolkit:

   http://sourceforge.net/projects/weka/
   

   The commonly followed steps are usually the following:

   1. Identify as many attributes/features as you can get (and a set of labels).
   2. Collect data which is a set { Label, Attribute1, A2, A3, ... }
   3. Select a minimal set of important attributes using feature selection algorithms (also available in the WEKA toolkit)
   4. Train the classifier using standard algorithm
   5. Test the system, until you receive the desired accuracy,recall, or other params.

   Good Luck!

Answer 2

This is a very broad topic. There are a few neural network libraries out there for C#, just search for them on Stack Overflow.

You will need to perform supervised training before you can do any type of classification. In order for the ANN to understand what you are throwing at it, you will need to figure out how you will parse the HTML to get the results you are looking for.

As an example, most websites will use CSS to render content on a browser. Other sites may use tables. You will need to train for both.

Your problem is not an easy one.