What is the meaning of GPU performance counters and driver counters?

Question

https://developer.nvidia.com/sites/default/files/akamai/tools/files/PerfKit_4_5_User_Guide.pdf

NVIDIA PerfKit SDK allows graphics developers access to low-level NVIDIA GPU
performance counters and NVIDIA driver counters.

I want to understand the meaning of these counters? Are they some kind of hardware or software? What do they do?

How they are helpful to me? Please give examples of making use of them.

I have to use Nvidia perfkit to determine the performance of certain softwares dealing with robotics.

Answer 1

There are many different kinds of performance issues that may occur in GPUs, each requires using different performance counters to identify and analyze. You can consider Peter's answer as a starting point. I'll discuss here the different profiling tools and libraries that are available to you and how to get started with each of them.

Libraries

NVIDIA PerfKit is a library that gives you access to performance counters. Which means that you have to write to code to call the APIs provided the by the library to measure performance events of interest. So using PerfKit may not be the best approach to diagnose performance issues. Anyway, you can find the user guide of PerfKit at the bottom of this page. You can download it from here.

The CUDA Toolkit comes with its own similar library, called the CUDA Profiling Tools Interface (CUPTI). The user guide can be found here. These slides also discuss how to use some performance counters through CUPTI.

Tools

The CUDA Toolkit comes with two profiling tools, one is a command-line tool, called nvprof and the other is a GUI tool, called the Visual Profiler. These tools are built on top of CUPTI and they are much easier to use since you don't have to write any code (or just a little code), so I think you should use one of these tools. The user guides of both of these tools can be found here. Here is a video on the performance analysis with the Visual Profiler. Here is an introductory article on nvprof. Here are also a bunch of videos on how to analyze all kinds of issues using different tools.

It's to difficult for anyone to give you a specific answer because you are not asking about a specific problem. But I think my answer and Peter's answer should help you to get started.