Error while importing Tensorflow in Python 2.7 in Ubuntu 12.04. 'GLIBC_2.17 not found'

Question

I have installed the Tensorflow bindings with python successfully. But when I try to import Tensorflow, I get the follwoing error.

ImportError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.17' not found (required by /usr/local/lib/python2.7/dist-packages/tensorflow/python/_pywrap_tensorflow.so)

I have tried to update GLIBC_2.15 to 2.17, but no luck.

Answer 1

I've just managed to install tensorflow 0.12rc0 on CentOS 6.5 with glibc 2.12, without having root privileges. Simply installing tensorflow binary via pip was giving me an error, related to GLIBC version as well.

Basically, you have 4 options how to deal with this (each with some advantages and disadvantages):

Option 1 - Upgrade your system GLIBC globally.

This is, probably, the best option, if your system supports this, you have root privileges, and you are confident that this upgrade won't break anything for some weird reason. Ultimately, this goes up to upgrading the whole Linux distribution. Here's a nice short list of default GLIBC versions on popular distributions.

Option 2 - Add second GLIBC to your system

Compile or download binary. The most simple&straightforward option. Especially if you only need to run few simple scripts.

• It is possible to have multiple versions of glibc on the same system, but one should do this with a great care.
• You won't destroy your system, if all your changes would be limited to a virtual environment.
• Many programs, installed/compiled before might be relying on old GLIBC, would just crash in your new environment (e.g. your python IDE). Including most basic bash commands, like "lc", "cd", etc.
• Other side-effects like significant memory leaks are also possible.
• Thus, it's a very bad idea to add new GLIBC to your normal environment, e.g. via .bashrc.
• On the other hand, if you need some specific tool in your new virtual environment, you may recompile it, linking against new GLIBC. So, that it would work OK in your new enviroment.
• However, personally, I quickly gave up recompiling everything I need in a new environment (without root and a package manager).
• A slightly different approach is officially offered by GLIBC developers, for testing new GLIBC builds.

Option 3 - Patch tensorflow

This may work for TF 0.6.0, but you would probably have to start again from scratch, when each new tensorflow version is released. E.g. here's a fix for 0.9.0.

Option 4 - Compile tensorflow from source

If you re-compile it from source and link against your existing GLIBC, newer GLIBC would be no longer needed. Somehow, this option was not mentioned in any answer here yet. Imho, this is the best option, both "in general", and "specifically for tensorflow".

• This works OK with r0.11 and would probably work for years, but theoretically, it might break in some newer tensorflow version, if they would decide to actually use some new GLIBC functionality, not present in older versions.
• To be honest, building tensorflow from source is not straightforward, especially on outdated systems.

A quick summary of "building tensorflow on outdated system":

Although the official guide provides a "installing from sources" section, there are few tricks you need to do to build it on an outdated system. Here I assume, that you do not have root privileges (if you do - you probably would be able to install the same pre-requestities with a package manager, rather them manually building them from source).

I found two well-documented success stories: #1, #2 and a number of useful posts on the official github (mostly about a set of libraries to link inside the binary): #1, #2, #3, #4. I had to combine tricks, described there to successfully compile TF in my case.

1. First of all, check your gcc --version, and verify that it supports c++11. Mine was 4.4.7, so it won't work. I've downloaded gcc-4.9.4 source code, and compiled it. This step is pretty straightforward, but the compilation itself may take few hours. As a workaround for an issue in bazel, I've compiled gcc with hardcoded paths to as,ld and nm. However, you may try another workarounds: (1, 2).

   #!/bin/sh
   
   unset LIBRARY_PATH CPATH C_INCLUDE_PATH 
   unset PKG_CONFIG_PATH CPLUS_INCLUDE_PATH INCLUDE LD_LIBRARY_PATH
   
   cd gcc-4.9.4
   ./contrib/download_prerequisites
   
   mkdir objdir
   cd objdir
   
   
   # I've added --disable-multilib to fix the following error:
   # /usr/bin/ld: crt1.o: No such file: No such file or directory
   # collect2: ld returned 1 exit status
   # configure: error: I suspect your system does not have 32-bit 
   # developement libraries (libc and headers). If you have them,
   # rerun configure with --enable-multilib. If you do not have them, 
   # and want to build a 64-bit-only compiler, rerun configure 
   # with --disable-multilib.           
   
   ../configure --prefix=$HOME/opt/gcc-4.9.4 \
                --disable-multilib \
                --disable-nls \
                --enable-languages=c,c++ \
                --with-ld=/usr/bin/ld \
                --with-nm=/usr/bin/nm \
                --with-as=/usr/bin/as
   
   make        
   make install
   

2. Check your java --version. Bazel requires JDK 8, install it if necessary. (They still provide some jdk7 related downloads, for bazel-0.4.1 but it looks like they consider it deprecated)

3. I've created a separate use_gcc_4.9.4.sh file, with necessary environment variables. I use source ./use_gcc_4.9.4.sh when I need to so something related to this newer compiler.

   #!/bin/sh
   this=$HOME/opt/gcc-4.9.4
   export PATH=$this/bin:$PATH
   export CPATH=$this/include:$CPATH
   export LIBRARY_PATH=$this/lib:$LIBRARY_PATH
   export LIBRARY_PATH=$this/lib64:$LIBRARY_PATH
   export LD_LIBRARY_PATH=$this/lib:$LD_LIBRARY_PATH
   export LD_LIBRARY_PATH=$this/lib64:$LD_LIBRARY_PATH
   

4. The current bazel binary (0.4.1) requires GLIBC 2.14, so we have to compile bazel from source as well (with our new gcc). Works OK, unless you are only allowed to run a very limited number of threads on the target machine. (This post describes some additional workarounds, but in my case they were not needed, maybe due to recent updates in bazel code.)

5. Obtain tensorflow source code git clone https://github.com/tensorflow/tensorflow, and install prerequisites you need (CUDA,cuDNN,python, etc). See official guide.

6. If you're not using default system gcc (e.g. if you had to compile newer gcc, like discussed above), add the following linker flags to tensorflow/third_party/gpus/crosstool/CROSSTOOL.tpl, line 59:

   linker_flag: "-L/home/username/localinst/opt/gcc-4.9.4/lib64"
   linker_flag: "-Wl,-rpath,/home/username/localinst/opt/gcc-4.9.4/lib64"
   

   Without this step, you would likely run into error messages like this:

   # ERROR: /home/username/localdistr/src/tensorflow/tensorflow/tensorflow/core/debug/BUILD:33:1: null failed: protoc failed: error executing command bazel-out/host/bin/external/protobuf/protoc '--cpp_out=bazel-out/local_linux-py3-opt/genfiles/' '--plugin=protoc-gen-grpc=bazel-out/host/bin/external/grpc/grpc_cpp_plugin' ... (remaining 8 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
   # bazel-out/host/bin/external/protobuf/protoc: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by bazel-out/host/bin/external/protobuf/protoc)
   # bazel-out/host/bin/external/protobuf/protoc: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by bazel-out/host/bin/external/protobuf/protoc)
   # bazel-out/host/bin/external/protobuf/protoc: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.18' not found (required by bazel-out/host/bin/external/protobuf/protoc)
   

7. Finally, to avoid GLIBC dependencies, we have to statically link some libraries, by adding the -lrt linker flag (maybe -lm as well). I found multiple posts, suggesting to add this in a different manner:

   • via bazel command line (may sound reasonable, but not working for me on current tensorflow version, somehow),
   • via "bazel-tensorflow/external/protobuf/BUILD"(not sure if it's working, but this does not look convenient - this file is only created during the build attempt itself)

   • via "third_party/gpus/crosstool/CROSSTOOL.tpl" (the same file we've just edited in the previous step, just below the lines we've already added).

     linker_flag: "-lrt"
     linker_flag: "-lm"
     

   • via "tensorflow/tensorflow.bzl" (works for me, but less convenient just because you have to edit one more file. I'm not sure it's 100% equivalent to the previous point)

   Without -lrt I ran into GLIBC-version-specific error again, trying to import tensorflow:

   # ImportError: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /home/username/anaconda3/envs/myenvname/lib/python3.5/site-packages/tensorflow/python/_pywrap_tensorflow.so)
   

   Without -lm you may run into this (for me, it turned out to be not necessary).

8. Run the build process.

    source ./use_gcc_4.9.4.sh
    ./configure
    bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
    bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
    pip install --upgrade /tmp/tensorflow_pkg/tensorflow-0.12.0rc0-cp35-cp35m-linux_x86_64.whl

9. Try to run the following simple python script to test if the most basic stuff is functioning:

   import tensorflow as tf
   hello = tf.constant('Hello, TensorFlow!')
   sess = tf.Session()
   print(sess.run(hello))
   
   a = tf.constant(10)
   b = tf.constant(32)
   print(sess.run(a + b))
   

Answer 2

Okay so here is the other solution I mentionned in my previous answer, it's more tricky, but should always work on systems with GLIBC>=2.12 and GLIBCXX>=3.4.13. In my case it was on a CentOS 6.7, but it's also fine for Ubuntu 12.04.

We're going to need a version of gcc that supports c++11, either on another machine or an isolated install; but not for the moment.

What we're gonna do here is edit the _pywrap_tensorflow.so binary in order to 'weakify' its libc and libstdc++ dependencies, so that ld accepts to link the stubs we're gonna make. Then we'll make those stubs for the missing symbols, and finally we're gonna pre-load all of this when running python.

First of all, I want to thank James for his great article ( http://www.lightofdawn.org/wiki/wiki.cgi/NewAppsOnOldGlibc ) and precious advices, I couldn't have made it without him.

So let's start by weakifying the dependencies, it's just about replacing the right bytes in _pywrap_tensorflow.so. Please note that this step only works for the current version of tensorflow (0.6.0). So if its not done already create and activate your virtualenv if you have one (if you're not admin virtualenv is a solution, another is to add --user flag to pip command), and install tensorflow 0.6.0 (replace cpu by gpu in the url if you want the gpu version) :

pip install --upgrade https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.6.0-cp27-none-linux_x86_64.whl

And let's weakify all the annoying dependencies, here is the command for the cpu version of tensorflow:

TENSORFLOW_DIR=`python -c "import imp; print(imp.find_module('tensorflow')[1])"`
for addr in 0xC6A93C 0xC6A99C 0xC6A9EC 0xC6AA0C 0xC6AA1C 0xC6AA3C; do printf '\x02' | dd conv=notrunc of=${TENSORFLOW_DIR}/python/_pywrap_tensorflow.so bs=1 seek=$((addr)) ; done

And here is the gpu one (run only the correct one or you'll corrupt the binary):

TENSORFLOW_DIR=`python -c "import imp; print(imp.find_module('tensorflow')[1])"`
for addr in 0xDC5EA4 0xDC5F04 0xDC5F54 0xDC5F74 0xDC5F84 0xDC5FA4; do printf '\x02' | dd conv=notrunc of=${TENSORFLOW_DIR}/python/_pywrap_tensorflow.so bs=1 seek=$((addr)) ; done

You can check it with:

readelf -V ${TENSORFLOW_DIR}/python/_pywrap_tensorflow.so

Have a look at the article if you want to understand what's going on here.

Now we're gonna make the stubs for the missing libc symbols:

mkdir ~/my_stubs
cd ~/my_stubs
MYSTUBS=~/my_stubs
printf "#include <time.h>\n#include <string.h>\nvoid* memcpy(void *dest, const void *src, size_t n) {\nreturn memmove(dest, src, n);\n}\nint clock_gettime(clockid_t clk_id, struct timespec *tp) {\nreturn clock_gettime(clk_id, tp);\n}" > mylibc.c
gcc -s -shared -o mylibc.so -fPIC -fno-builtin mylibc.c

You need to perform that step on the machine with the missing dependencies (or machine with similar versions of standard libraries (in a cluster for example)).

Now we're gonna probably change of machine since we need a gcc that supports c++11, and it is probably not on the machine that lacks all the dependencies (or you can use an isolated install of a recent gcc). In the following I assume we're still in ~/my_stubs and somehow you share your home accross the machines, otherwise you'll just have to copy the .so files we're gonna generate when it's done.

So, one stub that we can do for libstdc++, and for the remaining missing ones we're going to compile them from gcc source (it might take some time to clone the repository):

printf "#include <functional>\nvoid std::__throw_bad_function_call(void) {\nexit(1);\n}" > bad_function.cc
gcc -std=c++11 -s -shared -o bad_function.so -fPIC -fno-builtin bad_function.cc
git clone https://github.com/gcc-mirror/gcc.git
cd gcc
mkdir my_include
mkdir my_include/ext
cp libstdc++-v3/include/ext/aligned_buffer.h my_include/ext
gcc -I$PWD/my_include -std=c++11 -fpermissive -s -shared -o $MYSTUBS/hashtable.so -fPIC -fno-builtin libstdc++-v3/src/c++11/hashtable_c++0x.cc
gcc -std=c++11 -fpermissive -s -shared -o $MYSTUBS/chrono.so -fPIC -fno-builtin libstdc++-v3/src/c++11/chrono.cc
gcc -std=c++11 -fpermissive -s -shared -o $MYSTUBS/random.so -fPIC -fno-builtin libstdc++-v3/src/c++11/random.cc
gcc -std=c++11 -fpermissive -s -shared -o $MYSTUBS/hash_bytes.so -fPIC -fno-builtin ./libstdc++-v3/libsupc++/hash_bytes.cc

And that's it! You can now run a tensorflow python script by preloading all our shared libraries (and your local libstdc++):

LIBSTDCPP=`ldconfig -p | grep libstdc++.so.6 | grep 64 | cut -d' ' -f4` #For 64bit machines
LD_PRELOAD="$MYSTUBS/mylibc.so:$MYSTUBS/random.so:$MYSTUBS/hash_bytes.so:$MYSTUBS/chrono.so:$MYSTUBS/hashtable.so:$MYSTUBS/bad_function.so:$LIBSTDCPP" python ${TENSORFLOW_DIR}/models/image/mnist/convolutional.py

:)

Answer 3

I had the same problem, so googling I made these steps:

$ sudo pip install --upgrade virtualenv
$ virtualenv --system-site-packages ~/tensorflow
$ cd ~/tensorflow
$ source bin/activate
$ pip install --upgrade https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.5.0-cp27-none-linux_x86_64.whl
$ cd /tmp
$ wget http://launchpadlibrarian.net/137699828/libc6_2.17-0ubuntu5_amd64.deb
$ wget http://launchpadlibrarian.net/137699829/libc6-dev_2.17-0ubuntu5_amd64.deb
$ mkdir libc6_2.17
$ cd libc6_2.17
$ ar p ../libc6_2.17-0ubuntu5_amd64.deb data.tar.gz | tar zx
$ ar p ../libc6-dev_2.17-0ubuntu5_amd64.deb data.tar.gz | tar zx
$ cd -
$ LD_LIBRARY_PATH=/tmp/libc6_2.17/lib/x86_64-linux-gnu/ /tmp/libc6_2.17/lib/x86_64-linux-gnu/ld-2.17.so bin/python local/lib/python2.7/site-packages/tensorflow/models/image/mnist/convolutional.py

And to exit:

$ deactivate 

That works for me.