What is the difference between CuDNNLSTM and LSTM in Keras?

Question

In Keras, the high-level deep learning library, there are multiple types of recurrent layers; these include LSTM (Long short term memory) and CuDNNLSTM. According to the Keras documentation, a CuDNNLSTM is a:

Fast LSTM implementation backed by CuDNN. Can only be run on GPU, with the TensorFlow backend.

It is my belief that Keras automatically uses the GPU wherever possible. According to the TensorFlow build instructions, to have a working TensorFlow GPU backend, you will need CuDNN:

The following NVIDIA software must be installed on your system:

• NVIDIA's Cuda Toolkit (>= 7.0). We recommend version 9.0. For details, see NVIDIA's documentation. Ensure that you append the relevant Cuda pathnames to the LD_LIBRARY_PATH environment variable as described in the NVIDIA documentation.
• The NVIDIA drivers associated with NVIDIA's Cuda Toolkit.
• cuDNN (>= v3). We recommend version 6.0. For details, see NVIDIA's documentation, particularly the description of appending the appropriate pathname to your LD_LIBRARY_PATH environment variable.

Therefore, how would a CuDNNLSTM differ in any way from a normal LSTM using a TensorFlow GPU backend? Will CuDNNLSTM be automatically selected and replace the normal LSTM when an available TensorFlow GPU backend is found?

Answer 1

Why don't you try it out for yourself and see? In my case, training a model with LSTM took 10mins 30seconds. Simply switching the call from LSTM() to CuDNNLSTM() took less than a minute.

I also noticed that switching to CuDNNLSTM() speeds up model.evaluate() and model.predict() substantially as well.