What does this error mean "idf vector not fitted"

Question

This is how I am making a call to the TFIDFVectorizer:

vectorizer = TfidfVectorizer(
                vocabulary=selected_vocabulary,
                stop_words='english',
                use_idf=True,
                norm=norm,
                tokenizer=self.tokenize,
                lowercase=True,
                smooth_idf=True) 

and I get this error when I call

vectorizer.transform(data_to_vectorize)

Error:

  File "/root/anaconda/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 1305, in transform
    return self._tfidf.transform(X, copy=False)

  File "/root/anaconda/lib/python2.7/site-packages/sklearn/feature_extraction/text.py", line 1024, in transform
    raise ValueError("idf vector not fitted")

ValueError: idf vector not fitted

What does this error mean here?

Answer 1

You need to fit the model first (for example build the vocabulary from the data), before you can transform arbitrary text:

vectorizer.fit(data_to_vectorize)
X = vectorizer.transform(data_to_vectorize)

or

X = vectorizer.fit_transform(data_to_vectorize)