std::wcstok in VS 2015

Question

I have this toned down used case of code which when compiled with VS 2015 C++ compiler produces a warning.

#include <cwchar>
#include <iostream>

int main()
{
    wchar_t input[100] = L"A bird came down the walk";
    wchar_t* token = std::wcstok(input, L" ");
    while (token) {
        std::wcout << token << '\n';
        token = std::wcstok(nullptr, L" ");
    }
}

This produced following warnings.

warning C4996: 'wcstok': wcstok has been changed to conform with the ISO C standard, adding an extra context parameter. To use the legacy Microsoft wcstok, define _CRT_NON_CONFORMING_WCSTOK.
1>  c:\program files (x86)\windows kits\10\include\10.0.10240.0\ucrt\corecrt_wstring.h(254): note: see declaration of 'wcstok'

warning C4996: 'wcstok': wcstok has been changed to conform with the ISO C standard, adding an extra context parameter. To use the legacy Microsoft wcstok, define _CRT_NON_CONFORMING_WCSTOK.
1>  c:\program files (x86)\windows kits\10\include\10.0.10240.0\ucrt\corecrt_wstring.h(254): note: see declaration of 'wcstok'

Looking up online, I read about std::wcstok and breaking changes in VS 2015 which mentions that C standard has introduced a third parameter and that

It used an internal, per-thread context to track state across calls, as is done for strtok. The function now has the signature wchar_t* wcstok(wchar_t*, wchar_t const*, wchar_t**), and requires the caller to pass the context as a third argument to the function.

At the cost of sounding inherently stupid, I will still go ahead and ask, Can anybody please explain the purpose of this third parameter in simple terms and how it has changed std::wcstok from its earlier version?

Answer 1

Older version was similar to strtok and used global thread local storage to store position past the end of the last token.

The problem with used approach is that it did not allow nesting functions like strtok/wcstok.

Imagine we have a string like "r0c0;r0c1\nr1c0;r1c1" (a table with 2 rows and 2 columns) and we want to split it into rows first, then split each row into columns.

To do this we need 2 loops. With old approach this is impossible, as nested loop would overwrite state of the outer loop. With new approach every loop can have a separate state stored in a separate variables:

#include <cwchar>
#include <iostream>

int main()
{
    wchar_t input[] = L"r0c0;r0c1\n"
                      L"r1c0;r1c1";
    wchar_t *rowstate;
    wchar_t *row = std::wcstok(input, L"\n", &rowstate);

    while (row != nullptr) {
        std::wcout << L"Row: " << row << std::endl;

        wchar_t *colstate;
        wchar_t *col = std::wcstok(row, L";", &colstate);

        while (col != nullptr) {
            std::wcout << "  Col: " << col << std::endl;
            col = std::wcstok(nullptr, L" ", &colstate);
        }

        row = std::wcstok(nullptr, L" ", &rowstate);
    }
}

Output is:

Row: r0c0;r0c1
  Col: r0c0
  Col: r0c1
Row: r1c0;r1c1
  Col: r1c0
  Col: r1c1