Is using strtok() in two functions safe?

Question

I'm using strtok() to parse a line into individual words and check them for something. If that thing is found, another function is called which must also use strtok().

I know strtok() is not re-entrant. Does that mean that if I call it in the second function, my position in the string in the first function will be lost? If so, would using strtok() in the first function and strtok_r() in the second solve the problem? Is there another solution?

edit: thanks. it is indeed not possible to use strtok in two functions but apparently strtok_r is not standard. redesign it is...

Answer 1

Since strtok internally uses a global variable to store how far it had advanced in the string, intermingling calls to strtok will fail, just like you suspect. Your options are:

• switch to strtok_r, which has a similar API, but is not standard C (it is in POSIX, though);
• avoid strtok altogether in favor of some other function that doesn't carry hidden global state, such as strsep (also non-standard);
• make sure your first function fully exhausts strtok before calling another function that can call strtok.

All in all, strtok is a function best avoided.

Answer 2

The library function strtok uses an internal static state for the current parsing position:

• when called with strings, it starts a new parse,
• when called with NULL as the first argument, it uses its internal state.

If you directly or indirectly call strtok from your parse loop, the internal state will be updated and the call with NULL from the outer scope will not continue from the previous state, possibly invoking undefined behavior.

Posix function strtok_r takes an explicit state argument, so it can be used in nested contexts. If this function is available on your system, use it in all places where you use strtok. Alternatiely, you could a different method with strchr() or strcspn().

strtok_r is standardized in Posix. Depending on your target system, it may or may not be available. MacOS and most Unix systems are Posix compliant. Windows might have it under a different name. If it is not available, you can redefine it in your program and conditionally compile it.

Here is a simple implementation you ca use:

char *strtok_r(char *s, const char *delim, char **context) {
    char *token = NULL;

    if (s == NULL)
        s = *context;

    /* skip initial delimiters */
    s += strspn(s, delim);
    if (*s != '\0') {
        /* we have a token */
        token = s;
        /* skip the token */
        s += strcspn(s, delim);
        if (*s != '\0') {
            /* cut the string to terminate the token */
            *s++ = '\0';
        }
    }
    *context = s;
    return token;
}