What is the performance penalty of C++11 thread_local variables in GCC 4.8?

Question

From the GCC 4.8 draft changelog:

G++ now implements the C++11 thread_local keyword; this differs from the GNU __thread keyword primarily in that it allows dynamic initialization and destruction semantics. Unfortunately, this support requires a run-time penalty for references to non-function-local thread_local variables even if they don't need dynamic initialization, so users may want to continue to use __thread for TLS variables with static initialization semantics.

What is precisely the nature and origin of this run-time penalty?

Obviously to support non-function-local thread_local variables there needs to be a thread initialization phase before the entry to every thread main (just as there is a static initialization phase for global variables), but are they referring to some run-time penalty beyond that?

Roughly speaking what is the architecture of gcc's new implementation of thread_local?

Answer 1

(Disclaimer: I don't know much about the internals of GCC, so this is also an educated guess.)

The dynamic thread_local initialization is added in commit 462819c. One of the change is:

* semantics.c (finish_id_expression): Replace use of thread_local
variable with a call to its wrapper.

So the run-time penalty is that, every reference of the thread_local variable will become a function call. Let's check with a simple test case:

// 3.cpp extern thread_local int tls;     int main() {     tls += 37;   // line 6     tls &= 11;   // line 7     tls ^= 3;    // line 8     return 0; }  // 4.cpp  thread_local int tls = 42; 

When compiled*, we see that every use of the tls reference becomes a function call to _ZTW3tls, which lazily initialize the the variable once:

00000000004005b0 <main>: main():   4005b0:   55                          push   rbp   4005b1:   48 89 e5                    mov    rbp,rsp   4005b4:   e8 26 00 00 00              call   4005df <_ZTW3tls>    // line 6   4005b9:   8b 10                       mov    edx,DWORD PTR [rax]   4005bb:   83 c2 25                    add    edx,0x25   4005be:   89 10                       mov    DWORD PTR [rax],edx   4005c0:   e8 1a 00 00 00              call   4005df <_ZTW3tls>    // line 7   4005c5:   8b 10                       mov    edx,DWORD PTR [rax]   4005c7:   83 e2 0b                    and    edx,0xb   4005ca:   89 10                       mov    DWORD PTR [rax],edx   4005cc:   e8 0e 00 00 00              call   4005df <_ZTW3tls>    // line 8   4005d1:   8b 10                       mov    edx,DWORD PTR [rax]   4005d3:   83 f2 03                    xor    edx,0x3   4005d6:   89 10                       mov    DWORD PTR [rax],edx   4005d8:   b8 00 00 00 00              mov    eax,0x0              // line 9   4005dd:   5d                          pop    rbp   4005de:   c3                          ret  00000000004005df <_ZTW3tls>: _ZTW3tls():   4005df:   55                          push   rbp   4005e0:   48 89 e5                    mov    rbp,rsp   4005e3:   b8 00 00 00 00              mov    eax,0x0   4005e8:   48 85 c0                    test   rax,rax   4005eb:   74 05                       je     4005f2 <_ZTW3tls+0x13>   4005ed:   e8 0e fa bf ff              call   0 <tls> // initialize the TLS   4005f2:   64 48 8b 14 25 00 00 00 00  mov    rdx,QWORD PTR fs:0x0   4005fb:   48 c7 c0 fc ff ff ff        mov    rax,0xfffffffffffffffc   400602:   48 01 d0                    add    rax,rdx   400605:   5d                          pop    rbp   400606:   c3                          ret 

Compare it with the __thread version, which won't have this extra wrapper:

00000000004005b0 <main>: main():   4005b0:   55                          push   rbp   4005b1:   48 89 e5                    mov    rbp,rsp   4005b4:   48 c7 c0 fc ff ff ff        mov    rax,0xfffffffffffffffc // line 6   4005bb:   64 8b 00                    mov    eax,DWORD PTR fs:[rax]   4005be:   8d 50 25                    lea    edx,[rax+0x25]   4005c1:   48 c7 c0 fc ff ff ff        mov    rax,0xfffffffffffffffc   4005c8:   64 89 10                    mov    DWORD PTR fs:[rax],edx   4005cb:   48 c7 c0 fc ff ff ff        mov    rax,0xfffffffffffffffc // line 7   4005d2:   64 8b 00                    mov    eax,DWORD PTR fs:[rax]   4005d5:   89 c2                       mov    edx,eax   4005d7:   83 e2 0b                    and    edx,0xb   4005da:   48 c7 c0 fc ff ff ff        mov    rax,0xfffffffffffffffc   4005e1:   64 89 10                    mov    DWORD PTR fs:[rax],edx   4005e4:   48 c7 c0 fc ff ff ff        mov    rax,0xfffffffffffffffc // line 8   4005eb:   64 8b 00                    mov    eax,DWORD PTR fs:[rax]   4005ee:   89 c2                       mov    edx,eax   4005f0:   83 f2 03                    xor    edx,0x3   4005f3:   48 c7 c0 fc ff ff ff        mov    rax,0xfffffffffffffffc   4005fa:   64 89 10                    mov    DWORD PTR fs:[rax],edx   4005fd:   b8 00 00 00 00              mov    eax,0x0                // line 9   400602:   5d                          pop    rbp   400603:   c3                          ret 

This wrapper is not needed for in every use case of thread_local though. This can be revealed from decl2.c. The wrapper is generated only when:

• It is not function-local, and,

  1. It is extern (the example shown above), or
  2. The type has a non-trivial destructor (which is not allowed for __thread variables), or
  3. The type variable is initialized by a non-constant-expression (which is also not allowed for __thread variables).

In all other use cases, it behaves the same as __thread. That means, unless you have some extern __thread variables, you could replace all __thread by thread_local without any loss of performance.

---

*: I compiled with -O0 because the inliner will make the function boundary less visible. Even if we turn up to -O3 those initialization checks still remain.