How do I get started with a project on Text Summarization using NLP?

Question

My final year engineering project requires me to build an application using Java or Python which summarizes a text document using Natural Language Processing. How do I even begin with the programming of such an application?

Based on some research, I've just noted down that extraction-based summarization will be the best bet for me since it isn't so complex as abstraction based algorithms. Even then, it'd be really helpful if someone would guide me in the right direction to go about this.

Answer 1

Text summarization is still an open problem in NLP.

I guess that you might start by asking yourself what is the purpose of the summary:

• A summary that discriminates a document from other documents
• A summary that mines only the frequent patterns
• A summary that covers all the topics in the document
• etc

Because this will influence the way you generate the summary.

But as a start you could use in python the NLTK framework to extract basic elements from a text. For example you can extract the most frequent words, or the most frequent N-grams( N adjacent words) from the text.

Also a simple way to extract the most relevant sentences is using TF-IDF that stands for term frequency, Inverse document frequency. Basically this function gives higher scores to sentences that tend to appear frequently in one document compared to other document.

Some python libraries that you can use :

• sickitlearn that has more advanced features.
• Also gensim library has a text summarization tutorial (also in python)
• You can also use Dato that has as well a text analysis module.

Some helpful resources:

• This book: Foundations of Statistical Natural Language Processing
• There is also a coursera course that you can enroll in, in order to understand the basics in text mining: https://www.coursera.org/learn/text-mining
• Also this coursera course from stanford university (TF-IDF is explained in one of the videos) https://class.coursera.org/nlp/lecture/preview

Hope this helps.

Answer 2

These days, using Neural Net to summarize the corpus is considered state of the art.

Here is an article worth reading for you: A Neural Attention Model for Sentence Summarization http://www.aclweb.org/anthology/D15-1044