Static Variables and Thread-Local Storage

Question

Background:

I have discovered something of an interesting edge case relating to static memory initialization across multiple threads. Specifically, I am using Howard Hinnant's TZ library which has been working fine for the rest of my code across many different threads.

Now, I am developing a logging class which relies on yet another thread and condition variable. Unfortunately, when I attempt to format a chrono time_point using date::make_zoned(data::locate_zone("UTC"), tp) the library crashes. Upon digging through tz.cpp, I find that the time zone database returned internally is evaluating to NULL. This all comes from the following snippet:

tzdb_list&
get_tzdb_list()
{
    static tzdb_list tz_db = create_tzdb();
    return tz_db;
}

As can be seen, the database list is stored statically. With a few printf()s and some time with GDB I can see that the same db is returned for multiple calls from the main thread but returns NULL when called from my logger thread.

If, however, I change the declaration of tzdb_list to:

static thread_local tzdb_list tz_db = create_tzdb();

Everything works as expected. This is not surprising as thread_local will cause each thread to do the heavy-lifting of creating a standalone instance of tzdb_list. Obviously this is wasteful of memory and can easily cause problems later. As such, I really don't see this as a viable solution.

Questions:

• What about the invocation of one thread versus another would cause static memory to behave differently? If anything, I would expect the opposite of what is happening (eg. for the threads to 'fight' over initialized memory; not have one receive a NULL pointer).

• How is it possible for a returned static reference to have multiple different values in the first place (in my case, valid memory versus NULL)?

• With thread_local built into the library I get wildly different memory locations on opposite ends of the addressable region; why? I suspect that this has to do with where thread memory is allocated versus the main process memory but do not know the exact details of thread allocation regions.

Reference:

My logging thread is created with:

outputThread = std::thread(Logger::outputHandler, &outputQueue);

And the actual output handler / invocation of the library (LogMessage is just a typedef for std::tuple):

void Logger::outputHandler(LogQueue *queue)
{
    LogMessage entry;
    std::stringstream ss;

    while (1)
    {
        queue->pop(entry);           // Blocks on a condition variable

        ss << date::make_zoned(date::locate_zone("UTC"), std::get<0>(entry))
           << ":" << levelId[std::get<1>(entry)
           << ":" << std::get<3>(entry) << std::endl;

        // Printing stuff

        ss.str("");
        ss.clear();
    }
}

Additional code and output samples available on request.

---

EDIT 1

This is definitely a problem in my code. When I strip everything out my logger works as expected. What is strange to me is that my test case in the full application is just two prints in main and a call to the logger before manually exiting. None of the rest of the app initialization is run but I am linking in all support libraries at that point (Microsoft CPP REST SDK, MySQL Connector for C++ and Howard's date library (static)).

It is easy for me to see how something could be stomping this memory but, even in the "full" case in my application, I don't know why the prints on the main thread would work but the next line calling into the logger would fail. If something were going sideways at init I would expect all calls to break.

I also noticed that if I make my logger static the problem goes away. Of course, this changes the memory layout so it doesn't rule out heap / stack smashing. What I do find interesting is that I can declare the logger globally or on the stack at the start of main() and both will segfault in the same way. If I declare the logger as static, however, both global and stack-based declaration work.

Still trying to create a minimal test case which reproduces this.

I am already linking with -lpthread; have been pretty much since the inception of this application.

OS is Fedora 27 x86_64 running on an Intel Xeon. Compiler:

$ g++ --version
g++ (GCC) 7.3.1 20180130 (Red Hat 7.3.1-2)
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Answer 1

It appears that this problem was caused by a bug in tz.cpp which has since been fixed.

The bug was that there was a namespace scope variable whose initialization was not guaranteed in the proper order. This was fixed by turning that variable into a function-local-static to ensure the proper initialization order.

My apologies to all who might have been impacted by this bug. And my thanks to all those who have reported it.