Inconsistency of IDs between 'nvidia-smi -L' and cuDeviceGetName()

Question

I'm running this command into a shell and get:

C:\Users\me>nvidia-smi -L    
GPU 0: Quadro K2000 (UUID: GPU-b1ac50d1-019c-58e1-3598-4877fddd3f17)    
GPU 1: Quadro 2000 (UUID: GPU-1f22a253-c329-dfb7-0db4-e005efb6a4c7)

But in my code, when I run cuDeviceGetName(.., ID) where ID is the ID given by the nvidia-smi output, the devices have been inverted: GPU 0 becomes Quadro 2000 and GPU 1 becomes Quadro K2000.

Is this an expected behavior or a bug ? Does anyone know a workaround to make nvidia-smi get the 'real' ID of GPUs ? I could use the UUID to get the proper device with nvmlDeviceGetUUID() but using nvml API seems a bit too complicated for what I'm trying to achieve.

This question discuss how CUDA assign IDs to devices without clear conclusion.

I am using CUDA 6.5.

EDIT: I've had a look at nvidia-smi manpage (should have done that earlier...). It states:

"It is recommended that users desiring consistencyuse either UUDI or PCI bus ID, since device enumeration ordering is not guaranteed to be consistent"

Still looking for a kludge...

Answer 1

You can set the device order for CUDA environment in your shell to follow the bus ID instead of the default of fastest card. Requires CUDA 7 and up.

export CUDA_DEVICE_ORDER=PCI_BUS_ID

Answer 2

It's expected behavior.

nvidia-smi enumerates in PCI order.

By default, the CUDA driver and runtime APIs do not.

The question you linked clearly shows how to associate the two numbering/ordering schemes.

There is no way to cause nvidia-smi to modify its ordering scheme to match whatever will be generated by the CUDA runtime or driver APIs. However you can modify the CUDA runtime enumeration order through the use of an environment variable in CUDA 8.