How to find where does program crashed when Cuda API error detected: cudaMemcpy returned (0xb)

Question

I am debugging a cuda program and got the following warning:

warning: Cuda API error detected: cudaMemcpy returned (0xb)

warning: Cuda API error detected: cudaMemcpy returned (0xb)

warning: Cuda API error detected: cudaGetLastError returned (0xb)

Error in kernel
GPUassert: invalid argument

when I type "where" in cuda-gdb, it says "no stack."

(cuda-gdb) where
No stack.

How should find where my program crashed ?

Answer 1

Find the answer here : http://on-demand.gputechconf.com/gtc/2012/presentations/S0027A-Monday-Debugging-Experience-CUDA.pdf @ page 27.

You need to first:

(cuda-gdb) set cuda api_failures stop

Then when error happens, it will stop:

Cuda API error detected: cudaMemcpy returned (0xb)
(cuda-gdb) where
#0  0x00007fffea6a06d0 in cudbgReportDriverApiError () from       /usr/lib64/nvidia/libcuda.so.1
#1  0x00007fffea6a2c36 in cudbgReportDriverInternalError () from /usr/lib64/nvidia/libcuda.so.1
#2  0x00007fffea6eed93 in cudbgGetAPIVersion () from /usr/lib64/nvidia/libcuda.so.1
...