sklearn : TFIDF Transformer : How to get tf-idf values of given words in document

Question

I used sklearn for calculating TFIDF (Term frequency inverse document frequency) values for documents using command as :

from sklearn.feature_extraction.text import CountVectorizer count_vect = CountVectorizer() X_train_counts = count_vect.fit_transform(documents) from sklearn.feature_extraction.text import TfidfTransformer tf_transformer = TfidfTransformer(use_idf=False).fit(X_train_counts) X_train_tf = tf_transformer.transform(X_train_counts) 

X_train_tf is a scipy.sparse matrix of shape (2257, 35788).

How can I get TF-IDF for words in a particular document? More specific, how to get words with maximum TF-IDF values in a given document?

Answer 1

You can use TfidfVectorizer from sklean

from sklearn.feature_extraction.text import TfidfVectorizer import numpy as np from scipy.sparse.csr import csr_matrix #need this if you want to save tfidf_matrix  tf = TfidfVectorizer(input='filename', analyzer='word', ngram_range=(1,6),                      min_df = 0, stop_words = 'english', sublinear_tf=True) tfidf_matrix =  tf.fit_transform(corpus) 

The above tfidf_matix has the TF-IDF values of all the documents in the corpus. This is a big sparse matrix. Now,

feature_names = tf.get_feature_names() 

this gives you the list of all the tokens or n-grams or words. For the first document in your corpus,

doc = 0 feature_index = tfidf_matrix[doc,:].nonzero()[1] tfidf_scores = zip(feature_index, [tfidf_matrix[doc, x] for x in feature_index]) 

Lets print them,

for w, s in [(feature_names[i], s) for (i, s) in tfidf_scores]:   print w, s