Use tf-idf with FastText vectors

Question

I'm interested in using tf-idf with FastText library, but have found a logical way to handle the ngrams. I have used tf-idf with SpaCy vectors already for what I have found several examples like these ones:

• http://dsgeek.com/2018/02/19/tfidf_vectors.html

• https://www.aclweb.org/anthology/P16-1089

• http://nadbordrozd.github.io/blog/2016/05/20/text-classification-with-word2vec/

But for FastText library is not that clear to me, since it has a granularity that isn't that intuitive, E.G.

For a general word2vec aproach I will have one vector for each word, I can count the term frequency of that vector and divide its value accordingly.

But for fastText same word will have several n-grams,

"Listen to the latest news summary" will have n-grams generated by a sliding windows like:

lis ist ste ten tot het...

These n-grams are handled internally by the model so when I try:

model["Listen to the latest news summary"] 

I get the final vector directly, hence what I have though is to split the text into n-grams before feeding the model like:

model['lis']
model['ist']
model['ten']

And make the tf-idf from there, but that seems like an inefficient approach both, is there a standar way to apply tf-idf to vector n-grams like these.

Answer 1

I would leave FastText deal with trigrams, but keep building the tfidf-weighted embeddings at the word level.

That is, you send

model["Listen"]
model["to"]
model["the"]
...

to FastText, and then use your old code to get the tf-idf weights.

In any case, it would be good to know whether FastText itself considers the word construct when processing a sentence, or it truly only works it as a sequence of trigrams (blending consecutive words). If the latter is true, then for FastText you would lose information by breaking the sentence into separate words.