Algorithm to compare similarity of ideas (as strings)

Question

Consider an arbitrary text box that records the answer to the question, what do you want to do before you die?

Using a collection of response strings (max length 240), I'd like to somehow sort and group them and count them by idea (which may be just string similarity as described in this question).

1. Is there another or better way to do something like this?
2. Is this any different than string similarity?
3. Is this the right question to be asking?

The idea here is to have people write in a text box over and over again, and me to provide a number that describes, generally speaking, that 802 people wrote approximately the same thing

Answer 1

It is much more difficult than string similarity. This is what you need to do at a minimum:

• Perform some text formatting/cleaning tasks like removing punctuations characters and common "stop words"
• Construct a corpus (collection of words with their usage statistics) from the terms that occur answers.
• Calculate a weight for every term.
• Construct a document vector from every answer (each term corresponds to a dimension in a very high dimensional Euclidian space)
• Run a clustering algorithm on document vectors.

Read a good statistical natural language processing book, or search google for good introductions / tutorials (likely terms: statistical nlp, text categorization, clustering) You can probably find some libraries (weka or nltk comes to mind) depending on the language of your choice but you need to understand the concepts to use the library anyway.

Answer 2

The Latent Semantic Analysis (LSA) might interest you. Here is a nice introduction.

Latent semantic analysis (LSA) is a technique in natural language processing, in particular in vectorial semantics, of analyzing relationships between a set of documents and the terms they contain by producing a set of concepts related to the documents and terms. [...]