The Free energy approximation Equation in Restriction Boltzmann Machines

Question

According a deeplearning tutorial:

The free energy in python is

def free_energy(self, v_sample):
    ''' Function to compute the free energy '''
    wx_b = T.dot(v_sample, self.W) + self.hbias
    vbias_term = T.dot(v_sample, self.vbias)
    hidden_term = T.sum(T.log(1 + T.exp(wx_b)), axis=1)
    return -hidden_term - vbias_term

I am not very good at python, basically it get product expert of each visible unit as vector wx_b, calculate exp and plus 1 , calculate log and sum it for the hidden term.

Which I believe is a little different than free energy equation in the Learning Deep Architectures:

FreeEnergy(x) = −b′x − ∑log∑e^hi(ci+Wix).

Where:

• hi is the unit i hidden layer,
• ci is the i hidden bias in vector c.

It calculates exp and sum, calculate log respect to the sum value. after all sum all the product expert based on the number of visible unit.

The above equation is eq.5.21 from Learning Deep Architectures for AI (Yoshua Bengio)

Below is my draft of java implementation vis_v is the visible layer sample, hid_v is the hidden layer unit sample.

private double freeEnergy(RealVector vis_v, RealVector hid_v){
 RealVector wx_hb= W.preMultiply(vis_v).add(hBias);
 double vbias_term= vis_v.dotProduct(vBias);
 double sum_hidden_term = 0;
 for(int i=0;i< wx_hb.getDimension();i++){
     RealVector vis_expert = hid_v.mapMultiply(wx_hb.getEntry(i));
     double hidden_term= StatUtils.sum(vis_expert.map(new Exp()).toArray());
     sum_hidden_term+=Math.log(hidden_term);
 }
 return -sum_hidden_term-vbias_term;
}

Is this some kind of approximation? I am trying to implement the same thing in java, but am getting confused over it. Thanks in advance for any help!

Answer 1

I gather your confusion is over the definition of the free energy function in the reference python code. If this isn't what your asking I apologize.

First off, this is not an approximation. It looks like they're assuming the hidden units are binary valued. Remember, the free energy is just the (log of) the energy with hidden variables marginalized out. So, the inner sum in the free energy equation you listed above is just a sum over the values the i^th hidden unit can take on which, in this case, are {0,1}. Since exp(0) = 1 that inner sum just becomes 1+exp(...). See the "RBMs With Binary Units" section in the link you provided.

I'm not familiar with the apache commons math library in java so I can't be a huge amount of help there, but the implementation should be a straightforward translation from that python function.