How to expand a Tensorflow Variable

Question

Is there any way to make a Tensorflow Variable larger? Like, let's say I wanted to add a neuron to a layer of a neural network in the middle of training. How would I go about doing that? An answer in This question told me how to change the shape of the variable, to expand it to fit another row of weights, but I don't know how to initialize those new weights.

I figure another way of going about this might involve combining variables, as in initializing the weights first in a second variable and then adding that in as a new row or column of the first variable, but I can't find anything that lets me do that either.

Answer 1

Figured it out. It's kind of a roundabout process, but it's the only one I can tell that actually functions. You need to first unpack the variables, then append the new variable to the end, then pack them back together.

If you're expanding along the first dimension, it's rather short: only 7 lines of actual code.

#the first variable is 5x3
v1 = tf.Variable(tf.zeros([5, 3], dtype=tf.float32), "1")

#the second variable is 1x3
v2 = tf.Variable(tf.zeros([1, 3], dtype=tf.float32), "2")

#unpack the first variable into a list of size 3 tensors
#there should be 5 tensors in the list
change_shape = tf.unpack(v1)

#unpack the second variable into a list of size 3 tensors
#there should be 1 tensor in this list
change_shape_2 = tf.unpack(v2)

#for each tensor in the second list, append it to the first list
for i in range(len(change_shape_2)):
  change_shape.append(change_shape_2[i])

#repack the list of tensors into a single tensor
#the shape of this resultant tensor should be [6, 3]
final = tf.pack(change_shape)

If you want to expand along the second dimension, it gets somewhat longer.

#First variable, 5x3
v3 = tf.Variable(tf.zeros([5, 3], dtype=tf.float32))

#second variable, 5x1
v4 = tf.Variable(tf.zeros([5, 1], dtype=tf.float32))

#unpack tensors into lists of size 3 tensors and size 1 tensors, respectively
#both lists will hold 5 tensors
change = tf.unpack(v3)
change2 = tf.unpack(v4)

#for each tensor in the first list, unpack it into its own list
#this should make a 2d array of size 1 tensors, array will be 5x3
changestep2 = []
for i in range(len(change)):
  changestep2.append(tf.unpack(change[i]))

#do the same thing for the second tensor
#2d array of size 1 tensors, array will be 5x1
change2step2 = []
for i in range(len(change2)):
  change2step2.append(tf.unpack(change2[i]))

  #for each tensor in the array, append it onto the corresponding array in the first list
  for j in range(len(change2step2[i])):
    changestep2[i].append(change2step2[i][j])

  #pack the lists in the array back into tensors
  changestep2[i] = tf.pack(changestep2[i])

#pack the list of tensors into a single tensor
#the shape of this resultant tensor should be [5, 4]
final2 = tf.pack(changestep2)

I don't know if there's a more efficient way of doing this, but this works, as far as it goes. Changing further dimensions would require more layers of lists, as necessary.

Answer 2

There are various ways you could accomplish this.

1) The second answer in that post (https://stackoverflow.com/a/33662680/5548115) explains how you can change the shape of a variable by calling 'assign' with validate_shape=False. For example, you could do something like

# Assume var is [m, n] 
# Add the new 'data' of shape [1, n] with new values
new_neuron = tf.constant(...)  

# If concatenating to add a row, concat on the first dimension.
# If new_neuron was [m, 1], you would concat on the second dimension.
new_variable_data = tf.concat(0, [var, new_neuron])  # [m+1, n]

resize_var = tf.assign(var, new_variable_data, validate_shape=False)

Then when you run resize_var, the data pointed to by 'var' will now have the updated data.

2) You could also create a large initial variable, and call tf.slice on different regions of the variable as training progresses, since you can dynamically change the 'begin' and 'size' attributes of slice.

Answer 3

Simply using tf.concat for expand a Tensorflow Variable,you can see the api_docs for detail.

    v1 = tf.Variable(tf.zeros([5,3]),dtype=tf.float32)
    v2 = tf.Variable(tf.zeros([1,3]),dtype=tf.float32)
    v3 = tf.concat(0,[v1, v2])