debugging Tensorflow's C++ code behind the SWIG interface

Question

I'm not sure how to debug (presumably with GDB) the Python code behind a SWIG interface.

I can use ipdb to watch the execution of Tensorflow's Python code all the way to the SWIG wrapper (e.g. tf_session.TF_Run in session.py), but I would like to debug the C++ code behind the SWIG interface.

Presumably I build Tensorflow with bazel build --config debug, but how do I attach gdb to the resulting code when called from the Python interface?

Answer 1

TensorFlow's C++ code executes in the same process as the Python code that calls it (or, if you are using the distributed version, in the same process as one of the Python programs that created a tf.GrpcServer).

The simplest interface between Python and C++ is the pure-C API in tensor_c_api.h. To intercept one of these calls, you can attach gdb to the process ID of the Python interpreter that is running TensorFlow, and create a breakpoint on one of these functions.

For example, using an interactive Python session, in the first terminal enter:

$ python
>>> import tensorflow
>>> import os
>>> os.getpid()
14680

Then, in another terminal, start gdb:

$ gdb -p 14680
[...]
(gdb) break TF_NewSession
Breakpoint 1 at 0x7f15f450a4d0
(gdb) continue
Continuing.

Back in the Python interpreter, create a new session:

>>> sess = tf.Session()

The interpreter will pause, and your debugger will print something like the following:

Breakpoint 1, 0x00007f15f450a4d0 in TF_NewSession () from [...]/tensorflow/python/_pywrap_tensorflow.so
(gdb) backtrace
#0  0x00007f15f450a4d0 in TF_NewSession () from [...]/tensorflow/python/_pywrap_tensorflow.so
#1  0x00007f15f3ac5cdb in _wrap_TF_NewSession () from [...]/tensorflow/python/_pywrap_tensorflow.so
#2  0x000000000049968d in PyEval_EvalFrameEx ()
#3  0x00000000004a090c in PyEval_EvalCodeEx ()
#4  0x0000000000499a52 in PyEval_EvalFrameEx ()
[...]

You can now use the full power of gdb to debug TensorFlow.