Where does tensorflow log error?

Question

I use TF v0.12.1 with GPU support on Ubuntu-16.04 64 bits, but have an internal error in my pipeline:

  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1034, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Failed to run py callback pyfunc_4: see error log.

Where does TF log its error messages?

Answer 1

The log corresponding to that exception should be written to standard error in the process that run the op. The error message is produced by this code, which calls PyErr_Print(), which renders the current Python exception (in the C API) to standard error.

(Note that if you are using distributed TensorFlow and the tf.py_func() op is placed in a different task, you should look in the standard error for that task for log messages. However, please note that tf.py_func() does not work if the op is placed in a different process from the process that created the graph, because it relies on the code for the function being registered in a process-wide registry.)