algorithm: analyzing web pages for tags

Question

I've been working on a project in the last few days and there is a task in this project that I actually don't know how to do, the project includes analyzing web pages to find tags that Characterizes the page.

hey buddy , what you mean by tags? by saying tags I mean keywords that summarize what the web page about. For example here on SO you write you're own tags so people can find you're question better. What I am talking about is building an algorithm to analyze the web pages to find it's tags by the text within the page.

I started with getting the text from the page -> accomplished

generally im looking for a way to find the keywords that Concludes what the webpage about

However, I don't really know what to do next. Does anyone have a suggestion?

Answer 1

For a really basic approach, you could use the TF-IDF algorithm to find the most important word in your page

Quick overlook from wikipedia:

The tf–idf weight (term frequency–inverse document frequency) is a weight often used in information retrieval and text mining. This weight is a statistical measure used to evaluate how important a word is to a document in a collection or corpus. The importance increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus. Variations of the tf–idf weighting scheme are often used by search engines as a central tool in scoring and ranking a document's relevance given a user query. tf–idf can be successfully used for stop-words filtering in various subject fields including text summarization and classification

Once you find the most important word in your page you can use them as tags.

---

If you want to improve your tags and make them more relevant.

There are a lot of way to proceed, but you can proceed as below:

• Extract a bunch of text from which you know the main tags.
• For all this text run a TF-IDF algorithm and create a vector with the ones with the highest score.
• Try to find a main direction will all these vectors. (running an ACP for example, or any machine learning tool)
• And use this tag to represent the set of words from the main direction. (the largest vector of the ACP)

Hope it's understandable and it helps

Answer 2

Typically you look for certain words surrounded by certain html. For example, titles are typically in an H tag such as <h1>.

If you parse a page for all of it's H1 tags then it stands to reason that the content following that tag is related. An example is this very page. It has an H1 tag surrounding the question title. This gives google a hint that the page is about "algorithm", "analyzing", "web pages", etc.

The hard part is to determine context.

In our example here, the term "pages" is very generic and can relate to anything. However "web pages" are a bit more specific. You can do this with an internal dictionary that is built up over time based on term frequency after analyizing a number of documents to find commonality. The frequency should provide a weighted value in determining the top X "tags" for a given page.