How do I remove verbs, prepositions, conjunctions etc from my text? [closed]

Question

Basically in my text I just want to keep nouns and remove other parts of speech.

I do not think there is any automated way for this. If there is please suggest.

If there is no automated way, I can also do it manually, but for that I would require lists of all possible say, verbs or prepositions or conjunctions or adjectives etc. Can somebody please suggest a possible source where I can get these specific lists.

Answer 1

You can use the NLTK part-of-speech tagger to tag each word, then only keep the nouns. Here's an example of the NLTK tagger, taken from the NLTK homepage:

>>> import nltk
>>> sentence = """At eight o'clock on Thursday morning
... Arthur didn't feel very good."""
>>> tokens = nltk.word_tokenize(sentence)
>>> tokens
['At', 'eight', "o'clock", 'on', 'Thursday', 'morning',
'Arthur', 'did', "n't", 'feel', 'very', 'good', '.']
>>> tagged = nltk.pos_tag(tokens)
>>> tagged[0:6]
[('At', 'IN'), ('eight', 'CD'), ("o'clock", 'JJ'), ('on', 'IN'),
('Thursday', 'NNP'), ('morning', 'NN')]

In your case, you'd keep every element of the tagged list that have a tag starting with N, i.e. all the nouns, and throw the rest away. Check out the complete list of tags; you might also want to include foreign words (FW), for example.

NLTK is free to use, and it comes with its own data sets that are also free. You won't have to build lists of prepositions and so on yourself.

Answer 2

On the manual end.

The wiktionary dump.

https://dumps.wikimedia.org/enwiktionary/20140609/

I would just skip the full-articles dump in any flavor, and just go with the abstracts. It contains the word class. Good luck, the formatting is a beast.

to get started in python:

import xml.etree.ElementTree as ET
wiktionary = file('/path/to/wiktionary.xml')
tree = ET.iterparse(wiktionary.xml)
for event, elem in tree:
    if elem.tag == your_target_tag:
        do magic

Should get you started.

It's more work than a lot of other lists, but it is far richer than anything else I've used for NLP. Best of luck to you, and watch out for the unicode!