Latent Semantic Analysis (LSA) Tutorial

Question

I am trying to work with a tutorial in LSA in this link (edit: July 2017. Remove dead link)

Here is the code of the tutorial:

titles = [doc1,doc2]
stopwords = ['and','edition','for','in','little','of','the','to']
ignorechars = ''',:'!'''

class LSA(object):
    def __init__(self, stopwords, ignorechars):
        self.stopwords = open('stop words.txt', 'r').read()
        self.ignorechars = ignorechars
        self.wdict = {}
        self.dcount = 0        
    def parse(self, doc):
        words = doc.split();
        for w in words:
            w = w.lower()
            if w in self.stopwords:
                continue
            elif w in self.wdict:
                self.wdict[w].append(self.dcount)
            else:
                self.wdict[w] = [self.dcount]
        self.dcount += 1      
    def build(self):
        self.keys = [k for k in self.wdict.keys() if len(self.wdict[k]) > 1]
        self.keys.sort()
        self.A = zeros([len(self.keys), self.dcount])
        for i, k in enumerate(self.keys):
            for d in self.wdict[k]:
                self.A[i,d] += 1
    def calc(self):
        self.U, self.S, self.Vt = svd(self.A)
    def TFIDF(self):
        WordsPerDoc = sum(self.A, axis=0)        
        DocsPerWord = sum(asarray(self.A > 0, 'i'), axis=1)
        rows, cols = self.A.shape
        for i in range(rows):
            for j in range(cols):
                self.A[i,j] = (self.A[i,j] / WordsPerDoc[j]) * log(float(cols) / DocsPerWord[i])
    def printA(self):
        print 'Here is the count matrix'
        print self.A
    def printSVD(self):
        print 'Here are the singular values'
        print self.S
        print 'Here are the first 3 columns of the U matrix'
        print -1*self.U[:, 0:3]
        print 'Here are the first 3 rows of the Vt matrix'
        print -1*self.Vt[0:3, :]

mylsa = LSA(stopwords, ignorechars)
for t in titles:
    mylsa.parse(t)
mylsa.build()
mylsa.printA()
mylsa.calc()
mylsa.printSVD()

I read it and read it again, but I cannot figure something. If I execute the code, the results will be the following

Here are the singular values
[  4.28485706e+01   3.36652135e-14]
Here are the first 3 columns of the U matrix
[[  3.30049181e-02  -9.99311821e-01   7.14336493e-04]
 [  6.60098362e-02   1.43697129e-03   6.53394384e-02]
 [  6.60098362e-02   1.43697129e-03  -9.95952378e-01]
 ..., 
 [  3.30049181e-02   7.18485644e-04   2.02381089e-03]
 [  9.90147543e-02   6.81929920e-03   6.35728804e-03]
 [  3.30049181e-02   7.18485644e-04   2.02381089e-03]]
Here are the first 3 rows of the Vt matrix
array([[ 0.5015178 ,  0.86514732],
   [-0.86514732,  0.5015178 ]])

How can I figure the similarity of doc1 and doc2 from those matrices? In a tfidf algorithm I wrote myself, I have as a result a simple float number and here 3 matrices. Any advice?

Answer 1

One option is to run Cosine Similarity between the two matrices. I think you will find good information in question that I posted sometime ago. I also posted the answer for the question and I see that others have also given great answers.

Python: tf-idf-cosine: to find document similarity