Using Contrastive Divergence for Conditional Restricted Boltzmann Machines

Question

I would like to use a Restricted Boltzmann Machine for pattern recognition. It has come to my attention that they are actually used for finding distributions in patterns rather than pattern recognition. I looked at the following paper: http://www.cs.toronto.edu/~hinton/absps/uai_crbms.pdf which seems to use an extension of RBM, called ConditionalRBM. I would like to implement that. I already used Contrastive Divergence to implement RBM, and I would like to stick to that for CRBM, for simplicity. The paper focuses on replacing contrastive divergence, with more accurate algorithms.

From what I see in the paper, I now need to create three weight matrices (as now I also have to include the classification vectors)(see Figure1 in the paper), and I am not sure how to update each of them (ie how to create the vectors which will influence the change of the matrix.)

Could someone please clarify this for me or suggest an algorithm for classification using simple RBM, which I already implemented?

Thanks.

Answer 1

I found the following paper which clarifies the issue: http://uai.sis.pitt.edu/papers/11/p463-louradour.pdf . The poster here is also very helpful, especially for implementation: http://www.dmi.usherb.ca/~larocheh/publications/drbm-mitacs-poster.pdf . Instead of using 3 weight matrices it is enough to use 2, one for classification vectors and one for the actual patterns.

The formulas for the activation probabilities change, but the idea is the same.