cv2.remap or scipy.interpolate.map_coordinates equivalent/implementation in Tensorflow?

Question

edit: Tensorflow 1.3 now includes the tf.contrib.resampler for this operation. PyTorch also supports this operation as of v0.2 with the affine_grid function.

I am wondering whether there is an official or custom implementation of a function equivalent to cv2.remap(or scipy.ndimage.interpolate.map_coordinates, which is basically the same thing) in Tensorflow.

This question is similar but the answer is not what I am looking for since tf.contrib.image.transform function performs projective mapping and cv2.remap and scipy...map_coordinates perform pixel-wise mapping.

Answer 1

I just browsed through the GitHub repository and it does not seem to be implemented, tf.contrib.image.transform does not use any subroutines and purely returns the interpolated values. However, transformations of pixel locations themselves are just simple matrix multiplications that you can do yourself. See my answer here if you're not familiar. You will need to do the interpolation on your own though. Basically, you just put your coordinates into a new (3, N) matrix (where N is the number of points) like so (suppose you only have N=4 points for brevity):

x0 x0 x0 x0 x1 x1 x1 x1 x2 x2 x2 x2 x3 x3 x3 x3
y0 y1 y2 y3 y0 y1 y2 y3 y0 y1 y2 y3 y0 y1 y2 y3 
 1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  

and then multiply by your (3,3) transformation matrix to obtain new locations

s0*x0' s1*x0' s2*x0' ...
s0*y0' s1*y1' s2*y2' ...
s0     s1     s2  

where s0, ..., sN are scaling factors, so finally divide by the last row to remove scaling and then just take the top two rows as your points.

x0' x0' x0' ...
y0' y1' y2' ...

You'll want to be sure you're using floating-point operations, and then you can interpolate as you like.

You can do all of these operations in TensorFlow. If you're just doing this as a pre-processing step, you can of course break out the tensor into a numpy array and process with cv2.remap or scipy.ndimage.interpolate.map_coordinates and put it back into a tensor, but there's no real benefit to doing this.

Answer 2

A late answer

Upon TF 1.13 there is a new function tf.contrib.image.dense_image_warp which more or less does remap but without interpolation.