Edit image as tensorflow tensor python

Question

I'll do my best to provide a reproducible example here.

I have an image:

This image of Aaron Eckhart is (150, 150)

My goal is to perturb a ROI of this image by doing mathematical operations on the pixels, however, the issue is that the math must be done as a tensorflow tensor because the mathematical operation to be done is to multiply the tensor by it's scaled gradient (which is also a tensor of size (row_pixels, column_pixels, 3))

So here's the process I imagine:

1. Read in image as numpy array RGB size: (1, 150, 150, 3) (1 is batch size)

   w, h = img.shape

   ret = np.empty((w, h, 3), dtype=np.uint8)

   ret[:, :, 0] = ret[:, :, 1] = ret[:, :, 2] = img

2. Make pixel values between 0 and 1

   img = (faces1 - min_pixel) / (max_pixel - min_pixel)

3. for i in range(steps):

(a) extract ROI of image this is the part I don't understand how to do

(b) calculate gradient of smaller img ROI tensor's loss

loss = utils_tf.model_loss(y, preds, mean=False)
grad, = tf.gradients(loss, x)

(c) multiply img ROI tensor by gradient of loss

scaled_grad = eps * normalized_grad
adv_img = img + scaled_grad

(d) place this newly perturbed ROI tensor back into the same positions it was in the original tensor this is the other part I don't understand how to do

This will result in an image where only some of the pixel values have been perturbed and the rest remain the same

Answer 1

Given an image:

(a) Get a region of interest ((440, 240), (535, 380)) from the image:

roi_slice = tf.slice(
  image_in,
  [top_left_x, top_left_y, top_left_z],
  [roi_len_x, roi_len_y, bottom_right_z]
)

Get a boolean mask of the ROI that's the same size as the image

roi_mask = tf.ones_like(roi_slice)
mask_canvas = tf.image.pad_to_bounding_box(
  [roi_mask],
  top_left_x,
  top_left_y,
  np_image.shape[0],
  np_image.shape[1]
)
bool_mask = tf.cast(mask_canvas, tf.bool)

(b) For the purposes of this example, I'm using fake gradients, but you can replace with your real gradients.

fake_gradients = tf.ones_like(image_in) * 0.2

(c) Mask the gradients, to get the gradient where the ROI is, and 0 otherwise.

masked_gradients = tf.where(bool_mask[0], fake_gradients, mask_canvas[0])

(d) Make an editable copy of the image and update it with the masked gradients

# Make an editable copy of the image
editable_image = tf.get_variable(
    name='editable_image', shape=image_in.shape, dtype=tf.float32)
init_op = tf.assign(editable_image, image_in)

# Make sure we don't update the image before we've set its initial value.
with tf.control_dependencies([init_op]):
  update_roi_op = tf.assign_add(editable_image, masked_gradients)

You can find a fully working Colab example on GitHub.