cosine similarity between two words in a list

Question

I am defining a function which takes a list of words and returns information about the words in the list that have non-zero, cosine similarity between each other (along with the similarity value).

Can anyone help me out with this. I was thinking if I can get a precomputed word2vec vector file then it would be very helpful,but there is none on the internet.

Answer 1

You could define these two functions

def word2vec(word):
    from collections import Counter
    from math import sqrt

    # count the characters in word
    cw = Counter(word)
    # precomputes a set of the different characters
    sw = set(cw)
    # precomputes the "length" of the word vector
    lw = sqrt(sum(c*c for c in cw.values()))

    # return a tuple
    return cw, sw, lw

def cosdis(v1, v2):
    # which characters are common to the two words?
    common = v1[1].intersection(v2[1])
    # by definition of cosine distance we have
    return sum(v1[0][ch]*v2[0][ch] for ch in common)/v1[2]/v2[2]

and use them as in this example

>>> a = 'safasfeqefscwaeeafweeaeawaw'
>>> b = 'tsafdstrdfadsdfdswdfafdwaed'
>>> c = 'optykop;lvhopijresokpghwji7'
>>> 
>>> va = word2vec(a)
>>> vb = word2vec(b)
>>> vc = word2vec(c)
>>> 
>>> print cosdis(va,vb)
0.551843662321
>>> print cosdis(vb,vc)
0.113746579656
>>> print cosdis(vc,va)
0.153494378078

BTW, the word2vec that you mention in a tag is quite a different business, that requires that one of us take a great deal of time and commitment for studying it and guess what, I'm not that one...

Answer 2

What about this?

scipy.spatial.distance.cosine(word2vec(a),word2vec(b))

You can use word2vec library for that.