Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

NLTK - how to find out what corpora are installed from within python?

I'm trying to load some corpora I installed with the NLTK installer but I got a:

>>> from nltk.corpus import machado
      Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      ImportError: cannot import name machado

But in the download manager (nltk.download()) the package machado is marked as installed and I have a nltk_data/corpus/machado folder.

How can I see from inside the python intepreter what are the installed corpora?

Also, what package should I install to work with this how-to? http://nltk.googlecode.com/svn/trunk/doc/howto/portuguese_en.html

I can't find the module nltk.examples refered to in the how-to.

like image 339
Rafael S. Calsaverini Avatar asked Dec 14 '09 19:12

Rafael S. Calsaverini


2 Answers

try

import nltk.corpus
dir(nltk.corpus)

at which point, it probably told you something about __LazyModule__... so do dir(nltk.corpus) again.

If that doesn't work, try tab-completion in iPython.

like image 118
Hank Gay Avatar answered Nov 15 '22 20:11

Hank Gay


The NLTK includes a package, nltk.corpus, that contains definitions of corpus readers (such as PlainTextCorpusReader). This package also includes a large list of pre-defined access points for corpora that can be downloaded with nltk.downloader(). These access points (e.g., nltk.corpus.brown) are defined whether or not the corresponding corpus has been downloaded.

  1. To see which access points are defined in the NLTK, use dir(nltk.corpus) (after import nltk).

  2. To see which corpora you have in your nltk_data area, try this:

    import os
    import nltk
    print( os.listdir( nltk.data.find("corpora") ) )
    

    This just dumps a list with the contents of the folder nltk_data/corpora. You can take it from there.

  3. If you have installed your own corpus in the nltk_data/corpora area and the NLTK doesn't know about it, you need to fire up the appropriate reader yourself. E.g., if it's a plaintext corpus in corpora/mycorpus and all the files end in .txt, you'd do it like this:

    import nltk
    from nltk.corpus import PlaintextCorpusReader
    
    mypath = nltk.data.find("corpora/mycorpus")
    mycorpus = PlaintextCorpusReader(mypath, r".*\.txt$")
    

    But in that case you could put your own corpus anywhere, and point mypath to it directly instead of asking the NLTK to find it.

like image 32
alexis Avatar answered Nov 15 '22 20:11

alexis