Word Cloud built out of TF-IDF Vectorizer function

Question

I have a list called corpus that I am attempting TF-IDF on, using the sklearn in-built function. The list has 5 items. Each of these items comes from text files. I have generated a toy list called corpus for this example.

corpus = ['Hi what are you accepting here do you accept me',
'What are you thinking about getting today',
'Give me your password to get accepted into this school',
'The man went to the tree to get his sword back',
'go away to a far away place in a foreign land']

vectorizer = TfidfVectorizer(stop_words='english')
vecs = vectorizer.fit_transform(corpus)
feature_names = vectorizer.get_feature_names()
dense = vecs.todense()
lst1 = dense.tolist()
df = pd.DataFrame(lst1, columns=feature_names)
df

Using the above code, I was able to get a dataframe with 5 rows (for each item in the list) and n-columns with the tf-idf for each term in this corpus.

As a next step, I want to build the word cloud with largest tf-idf terms across the 5 items in the corpus getting the highest weight.

I tried the following:

x = vectorizer.vocabulary_
Cloud = WordCloud(background_color="white", max_words=50).generate_from_frequencies(x)

This clearly does not work. The dictionary is a list of words with an index attached to it, not a word scoring.

Hence, I need a dictionary that assigns the TF-IDF score to each word across the corpus. Then, the word cloud generated has the highest scored words as the largest size.

Answer 1

You're almost there. You need to transpose to get the frequencies per term rather than term frequencies per document, then sum hem, then pass that series directly to your wordcloud

df.T.sum(axis=1)

accept       0.577350
accepted     0.577350
accepting    0.577350
away         0.707107
far          0.353553
foreign      0.353553
getting      0.577350
hi           0.577350
land         0.353553
man          0.500000
password     0.577350
place        0.353553
school       0.577350
sword        0.500000
thinking     0.577350
today        0.577350
tree         0.500000
went         0.500000

Cloud = WordCloud(background_color="white", max_words=50).generate_from_frequencies(df.T.sum(axis=1))