Custom dropout in tensorflow

Question

I'm training a DNN model on some data, and am hoping to analyze the learned weights to learn something about the true system I am studying (signaling cascades in biology). I guess one could say I am using Artificial NNs to learn about Biological NNs.

For each of my training examples, I have removed a single gene, that is responsible for signaling at the top layer.

As I am modeling this signaling cascade as a NN, and removing one of the nodes in the first hidden layer, I realized that I'm doing a real life version of dropout.

I would therefore like to use dropout to train my model, however the implementations of dropout that I have seen online seem to randomly drop out a node. What I need is a way to specify which node to dropout for each training example.

Any advice on how to implement this? I'm open to any package, but right now everything i have already done is in Tensorflow so I'd appreciate a solution that uses that framework.

For those that prefer the details explained:

I have 10 input variables, that are fully connected to 32 relu nodes in the first layer, which are fully connected to a second layer (relu), which is fully connected to the output (linear because I am doing regression).

In addition to the 10 input variables, I also happen to know which of the 28 nodes should be dropped out.

Is there a way I can specify this when training?

Here is the code I currently use:

num_stresses = 10
num_kinase = 32
num_transcription_factors = 200
num_genes = 6692

# Build neural network
# Input variables (10)
# Which Node to dropout (32)
stress = tflearn.input_data(shape=[None, num_stresses])
kinase_deletion = tflearn.input_data(shape=[None, num_kinase])

# This is the layer that I want to perform selective dropout on,
# I should be able to specify which of the 32 nodes should output zero
# based on a 1X32 vector of ones and zeros. 
kinase = tflearn.fully_connected(stress, num_kinase, activation='relu')

transcription_factor = tflearn.fully_connected(kinase, num_transcription_factors, activation='relu')

gene = tflearn.fully_connected(transcription_factor, num_genes, activation='linear')

adam = tflearn.Adam(learning_rate=0.00001, beta1=0.99)

regression = tflearn.regression(gene, optimizer=adam, loss='mean_square', metric='R2')

# Define model
model = tflearn.DNN(regression, tensorboard_verbose=1)

Answer 1

I would supply your input variables and an equal sized vector of all 1's except for the one you want to drop, that one is a 0.

The very first operation should then be multiplication to zero out the gene you want to drop. From there on out, it should be the exact same as what you have now.

You can either multiply (zero out your gene) before handing it to tensorflow or add another place holder and feed it into the graph in the feed_dict like you do your variables. The latter one would probably be better.

If you need to drop a hidden node (in layer 2), it's just another vector of 1s and a 0.

Let me know if that works or if you need more help.

---

Edit: Ok, so I haven't really worked with tflearn very much (I just did regular tensorflow), but I think you can combine tensorflow and tflearn. Basically, I added tf.multiply. You might have to add in another tflearn.input_data(shape =[num_stresses]) and tflearn.input_data(shape =[num_kinase]) to give you placeholders for the stresses_dropout_vector and kinase_dropout_vector. And of course, you can change the number and positions of zeros in those two vectors.

import tensorflow as tf ###### New ######
import tflearn

num_stresses = 10
num_kinase = 32
num_transcription_factors = 200
num_genes = 6692

stresses_dropout_vector = [1] * num_stresses ###### NEW ######
stresses_dropout_vector[desired_node_to_drop] = 0 ###### NEW ######

kinase_dropout_vector = [1] * num_kinase ###### NEW ######
kinase_dropout_vector[desired_hidden_node_to_drop] = 0 ###### NEW ######

# Build neural network
# Input variables (10)
# Which Node to dropout (32)
stress = tflearn.input_data(shape=[None, num_stresses])
kinase_deletion = tflearn.input_data(shape=[None, num_kinase])

# This is the layer that I want to perform selective dropout on,
# I should be able to specify which of the 32 nodes should output zero
# based on a 1X32 vector of ones and zeros. 

stress_dropout = tf.multiply(stress, stresses_dropout_vector) ###### NEW ###### Drops out an input
kinase = tflearn.fully_connected(stress_dropout, num_kinase, activation='relu') ### changed stress to stress_dropout
kinase_dropout = tf.multiply(kinase, kinase_dropout_vector) ###### NEW ###### Drops out a hidden node

transcription_factor = tflearn.fully_connected(kinase_dropout, num_transcription_factors, activation='relu') ### changed kinase to kinase_dropout

gene = tflearn.fully_connected(transcription_factor, num_genes, activation='linear')

adam = tflearn.Adam(learning_rate=0.00001, beta1=0.99)

regression = tflearn.regression(gene, optimizer=adam, loss='mean_square', metric='R2')

# Define model
model = tflearn.DNN(regression, tensorboard_verbose=1)

If adding in tensorflow doesn't work, you just have to find a regular old tflearn.multiply function that does an element wise multiplication of two given tensors/vectors.

Hope that helps.