How to implement a sliding window in tensorflow?

Question

I have created a sliding window algorithm using numpy that slides over a wav audio file and feeds slices of it to my NN in tensorflow, which detects features in the audio slices. Once tensorflow does its thing, it returns its output to numpy land, where I reassemble the slices into an array of predictions that match each sample position of the original file:

import tensorflow as tf
import numpy as np
import nn

def slide_predict(layers, X, modelPath):
    output = None

    graph = tf.Graph()
    with graph.as_default():
        input_layer_size, hidden_layer_size, num_labels = layers

        X_placeholder = tf.placeholder(tf.float32, shape=(None, input_layer_size), name='X')
        Theta1 = tf.Variable(nn.randInitializeWeights(input_layer_size, hidden_layer_size), name='Theta1')
        bias1 = tf.Variable(nn.randInitializeWeights(hidden_layer_size, 1), name='bias1')
        Theta2 = tf.Variable(nn.randInitializeWeights(hidden_layer_size, num_labels), name='Theta2')
        bias2 = tf.Variable(nn.randInitializeWeights(num_labels, 1), name='bias2')
        hypothesis = nn.forward_prop(X_placeholder, Theta1, bias1, Theta2, bias2)

        sess = tf.Session(graph=graph)
        saver = tf.train.Saver()
        init = tf.global_variables_initializer()
        sess.run(init)

        saver.restore(sess, modelPath)

        window_size = layers[0]

        pad_amount = (window_size * 2) - (X.shape[0] % window_size)
        X = np.pad(X, (pad_amount, 0), 'constant')

        for w in range(window_size):
            start = w
            end = -window_size + w
            X_shifted = X[start:end]
            X_matrix = X_shifted.reshape((-1, window_size))

            prediction = sess.run(hypothesis, feed_dict={X_placeholder: X_matrix})

            output = prediction if (output is None) else np.hstack((output, prediction))

        sess.close()

    output.shape = (X.size, -1)

    return output

Unfortunately, this algorithm is quite slow. I placed some logs along the way and by far the slowest portion is the part where I actually run my tensorflow graph. This could be due to the actual tensorflow calculations being slow (if so, I'm probably just SOL), but I am wondering if a large part of the slowness isn't because I am transferring large audio files repeatedly back and forth in and out of tensorflow. So my questions are:

1) Is feeding a placeholder repeatedly like this going to be noticeably slower than feeding it once and calculating the values for X inside tensorflow?

2) If yes, whats the best way to implement a sliding window algorithm inside tensorflow to do this calculation?

Answer 1

The first issue is that your algorithm is has quadratic time complexity in window_size, because of calling np.hstack() in each iteration to build the output array, which copies both the current values of output and prediction into a new array:

for w in range(window_size):
    # ...
    output = prediction if (output is None) else np.hstack((output, prediction))

Instead of calling np.hstack() in every iteration, it would be more efficient to build a Python list of the prediction arrays, and call np.hstack() on them once, after the loop terminates:

output_list = []
for w in range(window_size):
    # ...
    prediction = sess.run(...)
    output_list.append(prediction)
output = np.hstack(output_list)

The second issue is that feeding large values to TensorFlow can be inefficient, if the amount of computation in the sess.run() call is small, because those values are (currently) copied into C++ (and the results are copied out. One useful strategy for this is to try and move the sliding window loop into the TensorFlow graph, using the tf.map_fn() construct. For example, you could restructure your program as follows:

# NOTE: If you call this function often, you may want to (i) move the `np.pad()`
# into the graph as `tf.pad()`, and (ii) replace `X_t` with a placeholder.
X = np.pad(X, (pad_amount, 0), 'constant')
X_t  = tf.convert_to_tensor(X)

def window_func(w):
    start = w
    end = w - window_size
    X_matrix = tf.reshape(X_t[start:end], (-1, window_size))
    return nn.forward_prop(X_matrix, Theta1, bias1, Theta2, bias2)

output_t = tf.map_fn(window_func, tf.range(window_size))
# ...
output = sess.run(output_t)