what is Intel MKL FATAL ERROR: Cannot load libmkl_core.dylib. while running pyspark in MacOs?

Question

Pyspark is installed in my Mac. This error I am getting while I try to run Pyspark from the command line.

I am installing Pyspark using homebrew and following instructions in this blog(https://medium.com/@roshinijohri/spark-with-jupyter-notebook-on-macos-2-0-0-and-higher-c61b971b5007). When I start running Pyspark in the shell I am getting the following error.

    Python 3.7.1 (default, Dec 14 2018, 13:28:58) 
    [Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    Intel MKL FATAL ERROR: Cannot load libmkl_core.dylib.

Answer 1

MKL is Intel's "Math Kernel Library", and it basically handles calculations. The new version of Numpy uses MKL by default, and that is likely why this error is showing up.

Try to update numpy,

conda update numpy

or disable MKL by

conda install nomkl

Hope that helped! Reference: https://github.com/BVLC/caffe/issues/3884

Answer 2

Update your mkl package. That should fix this.