Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is GEMM or BLAS used in Tensorflow, Theano, Pytorch

I know that Caffe uses GEneral Matrix to Matrix Multiplication (GEMM) which is part of Basic Linear Algebra Subprograms (BLAS) library for performing convolution operations. Where a convolution is converted to matrix multiplication operation. I have referred below article. https://petewarden.com/2015/04/20/why-gemm-is-at-the-heart-of-deep-learning/

I want to understand how other deep learning frameworks like Theano, Tensorflow, Pytorch perform convolution operations. Do they use similar libraries in the backend. There might be some articles present on this topic. If someone can point me to those or can explain with an answer.

PS: I posted the same question on datascience.stackexchange.com. As I didn't get a reply there, I am posting it here as well. If there is a better forum to post this question please let me know.

like image 831
Gaurav Srivastava Avatar asked Aug 13 '18 01:08

Gaurav Srivastava


People also ask

Does TensorFlow use Blas?

Most of the operations are implemented with cuDNN, some use cuBLAS, and others use cuda. You can also use openCL instead of cuda, but you should compile tensorflow by yourself. for CPU, intel mkl is used as the blas library.

What is GEMM in neural network?

GEMMs (General Matrix Multiplications) are a fundamental building block for many operations in neural networks, for example fully-connected layers, recurrent layers such as RNNs, LSTMs or GRUs, and convolutional layers.

What is better than TensorFlow?

TensorFlow is an open-sourced end-to-end platform, a library for multiple machine learning tasks, while Keras is a high-level neural network library that runs on top of TensorFlow. Both provide high-level APIs used for easily building and training models, but Keras is more user-friendly because it's built-in Python.

What is GEMV?

Computes a matrix-vector product using a general matrix. Description. The gemv routines compute a scalar-matrix-vector product and add the result to a scalar-vector product, with a general matrix.


1 Answers

tensorflow has multiple alternatives for the operations.

for GPU, cuda support is used. Most of the operations are implemented with cuDNN, some use cuBLAS, and others use cuda.

You can also use openCL instead of cuda, but you should compile tensorflow by yourself.

for CPU, intel mkl is used as the blas library.

I'm not familiar with pytorch and theano, but some commonly used blas libraries are listed below:

  • cuDNN, cuBLAS, and cuda: nvidia GPU support, most popular library
  • openCL: common GPU support, I don't know about it at all.
  • MKL: CPU blas library provided by intel
  • openBLAS: CPU library
like image 189
Kaihong Zhang Avatar answered Sep 25 '22 19:09

Kaihong Zhang