CUDA device stack and synchronization; SSY instruction

Question

Edit: this question is a re-done version of the original, so the first several responses may no longer be relevant.

I'm curious about what impact a device function call with forced no-inlining has on synchronization within a device function. I have a simple test kernel that illustrates the behavior in question.

The kernel takes a buffer and passes it to a device function, along with a shared buffer and an indicator variable which identifies a single thread as the "boss" thread. The device function has divergent code: the boss thread first spends time doing trivial operations on the shared buffer, then writes to the global buffer. After a synchronization call, all threads write to the global buffer. After the kernel call, the host prints the contents of the global buffer. Here is the code:

CUDA CODE:

test_main.cu

#include<cutil_inline.h>
#include "test_kernel.cu"

int main()
{
  int scratchBufferLength = 100;
  int *scratchBuffer;
  int *d_scratchBuffer;

  int b = 1;
  int t = 64;

  // copy scratch buffer to device
  scratchBuffer = (int *)calloc(scratchBufferLength,sizeof(int));
  cutilSafeCall( cudaMalloc(&d_scratchBuffer,
        sizeof(int) * scratchBufferLength) );
  cutilSafeCall( cudaMemcpy(d_scratchBuffer, scratchBuffer,
        sizeof(int)*scratchBufferLength, cudaMemcpyHostToDevice) );

  // kernel call
  testKernel<<<b, t>>>(d_scratchBuffer);

  cudaThreadSynchronize();

  // copy data back to host
  cutilSafeCall( cudaMemcpy(scratchBuffer, d_scratchBuffer,
        sizeof(int) * scratchBufferLength, cudaMemcpyDeviceToHost) );

  // print results
  printf("Scratch buffer contents: \t");
  for(int i=0; i < scratchBufferLength; ++i)
  {
    if(i % 25 == 0)
      printf("\n");
    printf("%d ", scratchBuffer[i]);
  }
  printf("\n");

  //cleanup
  cudaFree(d_scratchBuffer);
  free(scratchBuffer);

  return 0;
}

test_kernel.cu

#ifndef __TEST_KERNEL_CU
#define __TEST_KERNEL_CU


#define IS_BOSS() (threadIdx.x == blockDim.x - 1)

__device__
__noinline__
void testFunc(int *sA, int *scratchBuffer, bool isBoss) {

  if(isBoss)  {   // produces unexpected output-- "broken" code
//if(IS_BOSS())  {    // produces expected output-- "working" code

    for (int c = 0; c < 10000; c++)  {
      sA[0] = 1;
    }
  }

  if(isBoss) {
    scratchBuffer[0] = 1;
  }

  __syncthreads();

  scratchBuffer[threadIdx.x ] = threadIdx.x;

  return;

}

__global__
void testKernel(int *scratchBuffer)
{
  __shared__ int sA[4];

  bool isBoss = IS_BOSS();

  testFunc(sA, scratchBuffer, isBoss);
  return;
}
#endif

I compiled this code from within the CUDA SDK to take advantage of the "cutilsafecall()" functions in test_main.cu, but of course these could be taken out if you'd like to compile outside the SDK. I compiled with CUDA Driver/Toolkit version 4.0, compute capability 2.0, and the code was run on a GeForce GTX 480, which has the Fermi architecture.

The expected output is

0 1 2 3 ... blockDim.x-1

However, the output I get is

1 1 2 3 ... blockDim.x-1

This seems to indicate that the boss thread executed the conditional "scratchBuffer[0] = 1;" statement AFTER all threads execute the "scratchBuffer[threadIdx.x] = threadIdx.x;" statement, even though they are separated by a __syncthreads() barrier.

This occurs even if the boss thread is instructed to write a sentinel value into the buffer position of a thread in its same warp; the sentinel is the final value present in the buffer, rather than the appropriate threadIdx.x .

One modification that causes the code to produce expected output is to change the conditional statement

if(isBoss) {

to

if(IS_BOSS()) {

; i.e., to change the divergence-controlling variable from being stored in a parameter register to being computed in a macro function. (Note the comments on the appropriate lines in the source code.) It's this particular change I've been focusing on to try and track down the problem. In looking at the disassembled .cubins of the kernel with the 'isBoss' conditional (i.e., broken code) and the 'IS_BOSS()' conditional (i.e., working code), the most conspicuous difference in the instructions seems to be the absence of an SSY instruction in the disassembled broken code.

Here are the disassembled kernels generated by disassembling the .cubin files with "cuobjdump -sass test_kernel.cubin" . everything up to the first 'EXIT' is the kernel, and everything after that is the device function. The only differences are in the device function.

DISASSEMBLED OBJECT CODE:

"broken" code

code for sm_20

    Function : _Z10testKernelPi
/*0000*/     /*0x00005de428004404*/     MOV R1, c [0x1] [0x100];
/*0008*/     /*0x20009de428004000*/     MOV R2, c [0x0] [0x8];
/*0010*/     /*0x84001c042c000000*/     S2R R0, SR_Tid_X;
/*0018*/     /*0xfc015de428000000*/     MOV R5, RZ;
/*0020*/     /*0x00011de428004000*/     MOV R4, c [0x0] [0x0];
/*0028*/     /*0xfc209c034800ffff*/     IADD R2, R2, 0xfffff;
/*0030*/     /*0x9001dde428004000*/     MOV R7, c [0x0] [0x24];
/*0038*/     /*0x80019de428004000*/     MOV R6, c [0x0] [0x20];
/*0040*/     /*0x08001c03110e0000*/     ISET.EQ.U32.AND R0, R0, R2, pt;
/*0048*/     /*0x01221f841c000000*/     I2I.S32.S32 R8, -R0;
/*0050*/     /*0x2001000750000000*/     CAL 0x60;
/*0058*/     /*0x00001de780000000*/     EXIT;
/*0060*/     /*0x20201e841c000000*/     I2I.S32.S8 R0, R8;
/*0068*/     /*0xfc01dc231a8e0000*/     ISETP.NE.AND P0, pt, R0, RZ, pt;
/*0070*/     /*0xc00021e740000000*/     @!P0 BRA 0xa8;
/*0078*/     /*0xfc001de428000000*/     MOV R0, RZ;
/*0080*/     /*0x04001c034800c000*/     IADD R0, R0, 0x1;
/*0088*/     /*0x04009de218000000*/     MOV32I R2, 0x1;
/*0090*/     /*0x4003dc231a8ec09c*/     ISETP.NE.AND P1, pt, R0, 0x2710, pt;
/*0098*/     /*0x00409c8594000000*/     ST.E [R4], R2;
/*00a0*/     /*0x600005e74003ffff*/     @P1 BRA 0x80;
/*00a8*/     /*0x040001e218000000*/     @P0 MOV32I R0, 0x1;
/*00b0*/     /*0x0060008594000000*/     @P0 ST.E [R6], R0;
/*00b8*/     /*0xffffdc0450ee0000*/     BAR.RED.POPC RZ, RZ;
/*00c0*/     /*0x84001c042c000000*/     S2R R0, SR_Tid_X;
/*00c8*/     /*0x10011c03200dc000*/     IMAD.U32.U32 R4.CC, R0, 0x4, R6;
/*00d0*/     /*0x10009c435000c000*/     IMUL.U32.U32.HI R2, R0, 0x4;
/*00d8*/     /*0x08715c4348000000*/     IADD.X R5, R7, R2;
/*00e0*/     /*0x00401c8594000000*/     ST.E [R4], R0;
/*00e8*/     /*0x00001de790000000*/     RET;
    .................................

"working" code

code for sm_20

    Function : _Z10testKernelPi
/*0000*/     /*0x00005de428004404*/     MOV R1, c [0x1] [0x100];
/*0008*/     /*0x20009de428004000*/     MOV R2, c [0x0] [0x8];
/*0010*/     /*0x84001c042c000000*/     S2R R0, SR_Tid_X;
/*0018*/     /*0xfc015de428000000*/     MOV R5, RZ;
/*0020*/     /*0x00011de428004000*/     MOV R4, c [0x0] [0x0];
/*0028*/     /*0xfc209c034800ffff*/     IADD R2, R2, 0xfffff;
/*0030*/     /*0x9001dde428004000*/     MOV R7, c [0x0] [0x24];
/*0038*/     /*0x80019de428004000*/     MOV R6, c [0x0] [0x20];
/*0040*/     /*0x08001c03110e0000*/     ISET.EQ.U32.AND R0, R0, R2, pt;
/*0048*/     /*0x01221f841c000000*/     I2I.S32.S32 R8, -R0;
/*0050*/     /*0x2001000750000000*/     CAL 0x60;
/*0058*/     /*0x00001de780000000*/     EXIT;
/*0060*/     /*0x20009de428004000*/     MOV R2, c [0x0] [0x8];
/*0068*/     /*0x8400dc042c000000*/     S2R R3, SR_Tid_X;
/*0070*/     /*0x20201e841c000000*/     I2I.S32.S8 R0, R8;
/*0078*/     /*0x4000000760000001*/     SSY 0xd0;
/*0080*/     /*0xfc209c034800ffff*/     IADD R2, R2, 0xfffff;
/*0088*/     /*0x0831dc031a8e0000*/     ISETP.NE.U32.AND P0, pt, R3, R2, pt;
/*0090*/     /*0xc00001e740000000*/     @P0 BRA 0xc8;
/*0098*/     /*0xfc009de428000000*/     MOV R2, RZ;
/*00a0*/     /*0x04209c034800c000*/     IADD R2, R2, 0x1;
/*00a8*/     /*0x04021de218000000*/     MOV32I R8, 0x1;
/*00b0*/     /*0x4021dc231a8ec09c*/     ISETP.NE.AND P0, pt, R2, 0x2710, pt;
/*00b8*/     /*0x00421c8594000000*/     ST.E [R4], R8;
/*00c0*/     /*0x600001e74003ffff*/     @P0 BRA 0xa0;
/*00c8*/     /*0xfc01dc33190e0000*/     ISETP.EQ.AND.S P0, pt, R0, RZ, pt;
/*00d0*/     /*0x040021e218000000*/     @!P0 MOV32I R0, 0x1;
/*00d8*/     /*0x0060208594000000*/     @!P0 ST.E [R6], R0;
/*00e0*/     /*0xffffdc0450ee0000*/     BAR.RED.POPC RZ, RZ;
/*00e8*/     /*0x10311c03200dc000*/     IMAD.U32.U32 R4.CC, R3, 0x4, R6;
/*00f0*/     /*0x10309c435000c000*/     IMUL.U32.U32.HI R2, R3, 0x4;
/*00f8*/     /*0x84001c042c000000*/     S2R R0, SR_Tid_X;
/*0100*/     /*0x08715c4348000000*/     IADD.X R5, R7, R2;
/*0108*/     /*0x00401c8594000000*/     ST.E [R4], R0;
/*0110*/     /*0x00001de790000000*/     RET;
    .................................

The "SSY" instruction is present in the working code but not the broken code. The cuobjdump manual describes the instruction with, "Set synchronization point; used before potentially divergent instructions." This makes me think that for some reason the compiler does not recognize the possibility of divergence in the broken code.

I also found that if I comment out the __noinline__ directive, then the code produces the expected output, and indeed the assembly produced by the otherwise "broken" and "working" versions is exactly identical. So, this makes me think that when a variable is passed via the call stack, that variable cannot be used to control divergence and a subsequent synchronization call; the compiler does not seem to recognize the possibility of divergence in that case, and therefore doesn't insert an "SSY" instruction. Does anyone know if this is indeed a legitimate limitation of CUDA, and if so, if this is documented anywhere?

Thanks in advance.

Answer 1

This appears to have simply been a compiler bug fixed in CUDA 4.1/4.2. Does not reproduce for the asker on CUDA 4.2.