What are the latencies of GPU?

Question

I can find the latencies in terms of either ns or CPU cylces between CPU core and its cache, main memory, etc.

But it seems so hard to find similiar information about modern GPU.

Does anyone know about the latencies of GPU, esepecially the latencies between modern nvidia GPU (GF110 or later) and their memory, thanks.

GPU memory do have a much larger bandwidth, but what about their latencies?

I heard that the latencies for GPU are just as high as these for CPU, so basically make the larger bandwidth largely pointless for many general purpose computing tasks, I just need to confirm this.

Answer 1

Since vendors do not reveal all the architectural details, researchers have used reverse engineering to demystify GPU architecture. See this paper Demystifying GPU microarchitecture through microbenchmarking and other papers that cite this (note that it is not my paper). I have copied their findings in the image below.