Performance difference between Windows and Linux using Intel compiler: looking at the assembly

Question

I am running a program on both Windows and Linux (x86-64). It has been compiled with the same compiler (Intel Parallel Studio XE 2017) with the same options, and the Windows version is 3 times faster than the Linux one. The culprit is a call to std::erf which is resolved in the Intel math library for both cases (by default, it is linked dynamically on Windows and statically on Linux but using dynamic linking on Linux gives the same performance).

Here is a simple program to reproduce the problem.

#include <cmath>
#include <cstdio>

int main() {
  int n = 100000000;
  float sum = 1.0f;

  for (int k = 0; k < n; k++) {
    sum += std::erf(sum);
  }

  std::printf("%7.2f\n", sum);
}

When I profile this program using vTune, I find that the assembly is a bit different in between the Windows and the Linux version. Here is the call site (the loop) on Windows

Block 3:
"vmovaps xmm0, xmm6"
call 0x1400023e0 <erff>
Block 4:
inc ebx
"vaddss xmm6, xmm6, xmm0"
"cmp ebx, 0x5f5e100"
jl 0x14000103f <Block 3>

And the beginning of the erf function called on Windows

Block 1:
push rbp
"sub rsp, 0x40"
"lea rbp, ptr [rsp+0x20]"
"lea rcx, ptr [rip-0xa6c81]"
"movd edx, xmm0"
"movups xmmword ptr [rbp+0x10], xmm6"
"movss dword ptr [rbp+0x30], xmm0"
"mov eax, edx"
"and edx, 0x7fffffff"
"and eax, 0x80000000"
"add eax, 0x3f800000"
"mov dword ptr [rbp], eax"
"movss xmm6, dword ptr [rbp]"
"cmp edx, 0x7f800000"
...

On Linux, the code is a bit different. The call site is:

Block 3
"vmovaps %xmm1, %xmm0"
"vmovssl  %xmm1, (%rsp)"
callq  0x400bc0 <erff>
Block 4
inc %r12d
"vmovssl  (%rsp), %xmm1"
"vaddss %xmm0, %xmm1, %xmm1"   <-------- hotspot here
"cmp $0x5f5e100, %r12d"
jl 0x400b6b <Block 3>

and the beginning of the called function (erf) is:

"movd %xmm0, %edx"
"movssl  %xmm0, -0x10(%rsp)"   <-------- hotspot here
"mov %edx, %eax"
"and $0x7fffffff, %edx"
"and $0x80000000, %eax"
"add $0x3f800000, %eax"
"movl  %eax, -0x18(%rsp)"
"movssl  -0x18(%rsp), %xmm0"
"cmp $0x7f800000, %edx"
jnl 0x400dac <Block 8>
...

I have shown the 2 points where the time is lost on Linux.

Does anyone understand assembly enough to explain me the difference of the 2 codes and why the Linux version is 3 times slower?

Answer 1

In both cases the arguments and results are passed only in registers, as per the respective calling conventions on Windows and GNU/Linux.

In the GNU/Linux variant, the xmm1 is used for accumulating the sum. Since it's a call-clobbered register (a.k.a caller-saved) it's stored (and restored) in the stack frame of the caller on each call.

In the Windows variant, the xmm6 is used for accumulating the sum. This register is callee-saved in the Windows calling convention (but not in the GNU/Linux one).

So, in summary, the GNU/Linux version saves/restores both xmm0 (in the callee[1]) and xmm1 (in the caller), whereas the Windows version saves/restores only xmm6 (in the callee).

[1] need to look at std::errf to figure out why.

Answer 2

Using Visual Studio 2015, Win 7 64 bit mode, I find the following code for some of the paths used in erf() (not all paths shown). Each path involves up to 8 (maybe more for other paths) constants read from memory, so a single store / load to save a register seems unlikely to result in a 3x speed differential between Linux and Windows. As far for save / restores, this example saves and restores xmm6 and xmm7. As for the time, the program in the original post takes about 0.86 seconds on an Intel 3770K (3.5ghz cpu) (VS2015 / Win 7 64 bit). Update - I later determined the overhead for a save and restore of a xmm register is about 0.03 seconds in the case of the programs 10^8 loops (about 3 nanoseconds per loop).

000007FEEE25CF90  mov         rax,rsp  
000007FEEE25CF93  movss       dword ptr [rax+8],xmm0  
000007FEEE25CF98  sub         rsp,48h  
000007FEEE25CF9C  movaps      xmmword ptr [rax-18h],xmm6  
000007FEEE25CFA0  lea         rcx,[rax+8]  
000007FEEE25CFA4  movaps      xmmword ptr [rax-28h],xmm7  
000007FEEE25CFA8  movaps      xmm6,xmm0  
000007FEEE25CFAB  call        000007FEEE266370  
000007FEEE25CFB0  movsx       ecx,ax  
000007FEEE25CFB3  test        ecx,ecx  
000007FEEE25CFB5  je          000007FEEE25D0AF  
000007FEEE25CFBB  sub         ecx,1  
000007FEEE25CFBE  je          000007FEEE25D08F  
000007FEEE25CFC4  cmp         ecx,1  
000007FEEE25CFC7  je          000007FEEE25D0AF  
000007FEEE25CFCD  xorps       xmm7,xmm7  
000007FEEE25CFD0  movaps      xmm2,xmm6  
000007FEEE25CFD3  comiss      xmm7,xmm6  
000007FEEE25CFD6  jbe         000007FEEE25CFDF  
000007FEEE25CFD8  xorps       xmm2,xmmword ptr [7FEEE2991E0h]  
000007FEEE25CFDF  movss       xmm0,dword ptr [7FEEE298E50h]  
000007FEEE25CFE7  comiss      xmm0,xmm2  
000007FEEE25CFEA  jbe         000007FEEE25D053  
000007FEEE25CFEC  movaps      xmm2,xmm6  
000007FEEE25CFEF  mulss       xmm2,xmm6  
000007FEEE25CFF3  movaps      xmm0,xmm2  
000007FEEE25CFF6  movaps      xmm1,xmm2  
000007FEEE25CFF9  mulss       xmm0,dword ptr [7FEEE298B34h]  
000007FEEE25D001  mulss       xmm1,dword ptr [7FEEE298B5Ch]  
000007FEEE25D009  addss       xmm0,dword ptr [7FEEE298B8Ch]  
000007FEEE25D011  addss       xmm1,dword ptr [7FEEE298B9Ch]  
000007FEEE25D019  mulss       xmm0,xmm2  
000007FEEE25D01D  mulss       xmm1,xmm2  
000007FEEE25D021  addss       xmm0,dword ptr [7FEEE298BB8h]  
000007FEEE25D029  addss       xmm1,dword ptr [7FEEE298C88h]  
000007FEEE25D031  mulss       xmm0,xmm2  
000007FEEE25D035  mulss       xmm1,xmm2  
000007FEEE25D039  addss       xmm0,dword ptr [7FEEE298DC8h]  
000007FEEE25D041  addss       xmm1,dword ptr [7FEEE298D8Ch]  
000007FEEE25D049  divss       xmm0,xmm1  
000007FEEE25D04D  mulss       xmm0,xmm6  
000007FEEE25D051  jmp         000007FEEE25D0B2  
000007FEEE25D053  movss       xmm1,dword ptr [7FEEE299028h]  
000007FEEE25D05B  comiss      xmm1,xmm2  
000007FEEE25D05E  jbe         000007FEEE25D076  
000007FEEE25D060  movaps      xmm0,xmm2  
000007FEEE25D063  call        000007FEEE25CF04  
000007FEEE25D068  movss       xmm1,dword ptr [7FEEE298D8Ch]  
000007FEEE25D070  subss       xmm1,xmm0  
000007FEEE25D074  jmp         000007FEEE25D07E  
000007FEEE25D076  movss       xmm1,dword ptr [7FEEE298D8Ch]  
000007FEEE25D07E  comiss      xmm7,xmm6  
000007FEEE25D081  jbe         000007FEEE25D08A  
000007FEEE25D083  xorps       xmm1,xmmword ptr [7FEEE2991E0h]  
000007FEEE25D08A  movaps      xmm0,xmm1  
000007FEEE25D08D  jmp         000007FEEE25D0B2  
000007FEEE25D08F  mov         eax,8000h  
000007FEEE25D094  test        word ptr [rsp+52h],ax  
000007FEEE25D099  je          000007FEEE25D0A5  
000007FEEE25D09B  movss       xmm0,dword ptr [7FEEE2990DCh]  
000007FEEE25D0A3  jmp         000007FEEE25D0B2  
000007FEEE25D0A5  movss       xmm0,dword ptr [7FEEE298D8Ch]  
000007FEEE25D0AD  jmp         000007FEEE25D0B2  
000007FEEE25D0AF  movaps      xmm0,xmm6  
000007FEEE25D0B2  movaps      xmm6,xmmword ptr [rsp+30h]  
000007FEEE25D0B7  movaps      xmm7,xmmword ptr [rsp+20h]  
000007FEEE25D0BC  add         rsp,48h  
000007FEEE25D0C0  ret