'tensorflow.python.framework.ops.EagerTensor' object has no attribute '_in_graph_mode'

Question

I am trying to visualize CNN filters by optimizing a random 'image' so that it produces a high mean activation on that filter which is somehow similar to the neural style transfer algorithm.

For that purpose, I am using TensorFlow==2.2.0-rc. But during the optimization process, an error occurs saying 'tensorflow.python.framework.ops.EagerTensor' object has no attribute '_in_graph_mode'. I tried debugging it and it somehow works when I don't use the opt.apply_gradients() and instead, apply its gradient manually like img = img - lr * grads but I want to use the "Adam" optimizer rather than the simple SGD.

Here is my code for the optimization part

opt = tf.optimizers.Adam(learning_rate=lr, decay = 1e-6)
for _ in range(epoch):
    with tf.GradientTape() as tape:
        tape.watch(img)
        y = model(img)[:, :, :, filter]
        loss = -tf.math.reduce_mean(y)
        grads = tape.gradient(loss, img)
        opt.apply_gradients(zip([grads], [img]))

Answer 1

The reason for the bug is that the tf.keras optimizers apply gradients to variable objects (of type tf.Variable), while you are trying to apply gradients to tensors (of type tf.Tensor). Tensor objects are not mutable in TensorFlow, thus the optimizer cannot apply gradients to it.

You should initialize the variable img as a tf.Variable. This is how your code should be:

# NOTE: The original image is lost here. If this is not desired, then you can
# rename the variable to something like img_var.
img = tf.Variable(img)
opt = tf.optimizers.Adam(learning_rate=lr, decay = 1e-6)

for _ in range(epoch):
    with tf.GradientTape() as tape:
        tape.watch(img)
        y = model(img.value())[:, :, :, filter]
        loss = -tf.math.reduce_mean(y)

    grads = tape.gradient(loss, img)
    opt.apply_gradients(zip([grads], [img]))

Also, it is recommended to calculate the gradients outside the tape's context. This is because keeping it in will lead to the tape tracking the gradient calculation itself, leading to higher memory usage. This is only desirable if you want to calculate higher-order gradients. Since you don't need those, I have kept them outside.

Note I have changed the line y = model(img)[:, :, :, filter] to y = model(img.value())[:, :, :, filter]. This is because tf.keras models need tensors as input, not variables (bug, or feature?).

Answer 2

Well although not directly related, but can be somewhat useful to understand what causes this type of errors at the first place.

This type of error occurs whenever we try to modify a constant tensor.

Simple example, which raise similar error below--

unchangeable_tensors = tf.constant([1,2,3])
unchangeable_tensors[0].assign(7)

A way to bypass the error is using tf.Variable() as shown below

changeable_tensors = tf.Variable([1,2,3])
changeable_tensors[0].assign(7)